Coverage metrics and source-data quality

Specify exactly what must be addressed

We choose a question with its material conditions as the unit. ‘What is the nominal volume?’ and ‘How much was filled for the test?’ concern different quantities. Two differently worded versions of the nominal-volume question must not automatically become two plan items.

The boundary of a question affects the assessment. A broad ‘what is known about temperature?’ does not define which information is mandatory for credit; ‘what conditions and outcome does the protocol specify?’ requires a particular set of information. The wording must not be changed after reading a convenient answer while presenting the metric as unchanged.

The list receives a version and a product association. Each item records its purpose, complete-answer criterion, weight and reason for inclusion. Inapplicable questions are excluded under a defined rule, with the reason retained. Missing data alone does not make a question inapplicable.

A weight represents an agreed priority within this plan. It is neither a purchase probability nor a count of people. Overlapping query frequencies cannot be added as independent buyer weights. If demand data informs selection, record how it is used separately.

One illustrative plan and one state of the text

Take five T500 questions. The weights 5, 4, 4, 3 and 4 are chosen solely for this calculation; they do not recommend priorities for every insulated-mug seller. They total 20. The source dossier specifies a nominal 500 ml, an empty-mug mass of 280 g, and permission to hand-wash the removable lid.

The protocol specifies 450 ml of beverage initially at 95 °C, a mug kept closed throughout the six-hour interval, and ambient air at +20 °C; the beverage ends at no less than 60 °C. No dishwasher information is available.

Define the illustrative passage to be assessed: ‘T500 has a nominal volume of 500 ml. The empty mug weighs 280 g. The temperature-retention test was conducted at an ambient temperature of +20 °C. The available materials contain no information about washing the lid in a dishwasher.’ It reports some conditions without presenting them as the full protocol.

The assessment concerns this passage, not the whole library or a finished listing from another article. Every state has a verifiable basis in the explicitly defined text.

Define zero and one before assigning them

For data sufficiency, let d = 1 when supported information is enough to answer the selected question fully, and d = 0 otherwise. Conflicts, partial evidence and unknowns are also retained in words: zero alone does not explain the cause.

For the text, choose a strict binary rubric: a = 1 only for a complete, supported answer with the necessary conditions; a = 0 for all other states. In our plan, Q1 and Q2 receive one. Q3 remains partial, Q4 is absent, and Q5 lacks sufficient evidence for either permission or prohibition.

Zero for Q5 does not mean the editor should invent an answer. The uncertainty has been honestly identified, which is useful and recorded separately. The chosen index measures whether there is a supported resolution of ‘may the lid be washed in a dishwasher?’ If the question were ‘does the dossier contain information about washing the lid in a dishwasher?’, the criterion would differ. That would be a different assessment plan.

A fractional scale is also possible. But ‘0.5 for a partial answer’ needs a predefined meaning: exactly which part is answered and why it receives that contribution. A fraction must not first be assigned from a model's impression and later presented as objectively measured completeness.

Blocking violations are checked separately. If the text adds unsupported compatibility, a high average coverage score does not override publication eligibility conditions. A question's weight and its ability to block publication are different attributes.

Three metrics with different denominators

Let w be a question's weight and W the sum of all weights in the chosen plan. Here, W = 20. Denote data readiness by D, complete coverage by C, and implementation among questions with sufficient data by R.

D = Σ(w × d) / W = 16 / 20 = 80 %.

There is enough data for Q1–Q4: 5 + 4 + 4 + 3 = 16. The denominator also includes Q5 with weight 4, although its answer is unknown.

C = Σ(w × a) / W = 9 / 20 = 45 %.

Complete answers to Q1 and Q2 contribute 5 + 4 = 9. Under the chosen binary rubric, the partial protocol adds no points, but remains visible as a separate status.

R = Σ(w × d × a) / Σ(w × d) = 9 / 16 = 56.25%.

The last fraction answers a different question: how much has been implemented among items for which the data is sufficient? It must not be labelled as complete coverage of all needs.

How to improve the number without improving the text

Remove Q5, for which no data is available. The remaining total weight is 16. The same complete answers to Q1 and Q2 now yield 9 / 16 = 56.25% instead of 45%. Data readiness becomes 100%. Not a sentence has been added: the basis of the calculation has changed.

A conditional metric for data-ready items is useful when explicitly named and shown alongside the full plan. The problem is a hidden substitution of the denominator. The same issue arises when difficult product variants or unanswered questions disappear from a report.

The list can legitimately change: a new task is discovered, applicability is clarified, or a grouping error is corrected. Retain the reason and version. To compare two revisions, either recalculate both on an agreed set or explicitly show that the metrics refer to different plans.

Weights must not be adjusted to favour a preferred outcome either. Increasing the weight of the well-answered nominal-volume question after assessment raises the fraction without changing the content. The value of the new priority needs separate justification, and the change history must be retained.

When the calculation becomes undefined

Weights must be non-negative and their total positive. For an empty set or a zero total, D and C cannot be calculated. If the weight of questions with sufficient data is zero, R is undefined too. Arbitrary values of 0% or 100% would conceal the absence of a basis for the fraction.

Both indicators are assessed against the same dossier version and the same answer-sufficiency criteria. Under this strict rubric, a complete supported answer implies sufficient data. C therefore cannot exceed D. With non-zero data readiness, the arithmetic check C = D × R also holds: 0.8 × 0.5625 = 0.45. This is a property of this scheme, not a rule for every partial-credit scale.

The annotation itself can be wrong. One evaluator may count +20 °C as a full answer about the protocol; another may require fill volume, initial temperature, closure conditions, duration and final outcome. Agree on the required answer content and resolve such disagreements before calculating.

The index does not measure whether an answer can be found. All the information may sit in a long, inaccessible block. It also does not measure search visibility, the proportion of satisfied buyers or effects on purchases. Those tasks require different observations.

A metric should identify the next action

The original T500 plan reveals two different work queues. Data already exists for Q3 and Q4: the protocol and hand-washing instructions for the lid can be added. Q5 requires clarification from the data owner. Regenerating from the same dossier will not create permission or a prohibition on washing the lid in a dishwasher.

Adding a complete care answer raises the contribution from 9 to 12 with W still 20: C becomes 60%, R becomes 75%, and D remains 80%. Correctly describing the full protocol next brings C to 80% and R to 100%. These changes come from an illustrative calculation, not a test of the text with buyers.

Receiving a new document for Q5 requires a separate review of the data and dependent statements, including the earlier ‘no information’ message. It changes the editor's working conditions. Dependency history helps distinguish that change from the quality of a new revision.

For reproducibility, retain the question table, weights, rubric, data states, assessed text and scoring evidence. Disputed decisions remain visible. The number can then be reproduced and explained row by row.

The final question concerns whoever applies the rubric: do they detect incomplete answers and avoid giving credit to unsupported promises? This requires testing the evaluator itself on cases with decisions verified in advance.