Word forms, phrases and attribute extraction

Keep the excerpt and its context together

We will examine three lines from the fictional dossier for the Alfa set. The document context identifies the product and the purpose of each section: the first line concerns paint applications, the second the contents of the set, and the third brush care.

Without its context, the first line does not even name the product or explicitly say ‘suitable for’. It cannot supply a product identifier or reliably assign the stated application to any nearby item. Its connection to the heading, section and document version must remain part of the evidence.

A record locator can include the document, revision, line and boundaries of the source span. For volume, retaining ‘20’ alone is insufficient: the span «6 туб по 20 мл» (‘6 tubes of 20 ml each’) is needed to show how the quantity is distributed. In the original Russian string, meaning ‘The set contains 6 tubes of 20 ml each’, В комплекте 6 туб по 20 мл. with no line number, «20 мл» (‘20 ml’) occupies the range [21, 26), and the context «6 туб по 20 мл» occupies [12, 26). Characters are counted from zero and the right boundary is excluded; this is a locator within the stored string, not a byte offset in the file. For care instructions, the whole span with both timing conditions is needed.

When the file is updated, the previous coordinates may change. Both the version used and the quotation must therefore be retained. This extends the work on data provenance: the locator helps verify a decision, but does not establish that the source document is true.

Tokens and lemmas have a limited role

For the second line, a manual tokenization can identify «В», «комплекте», «6», «туб», «по», «20», «мл» and the full stop. These Russian tokens correspond to ‘in’, ‘the set’, ‘6’, ‘tubes’, ‘each’, ‘20’ and ‘ml’. This is one chosen way to address the elements. Another tool might handle the unspaced ‘20мл’ or a special space differently; its specific rules must be understood before comparing results.

Lemmas help connect inflected forms. In this manual illustration, Russian «бумаги» maps to «бумага» (‘paper’), «водой» to «вода» (‘water’), and «туб» to «туба» (‘tube’). The forms in the excerpt are inflected; the second form in each pair is the dictionary form. This operation neither connects water with care nor determines the volume of one tube. Relations between elements are needed for that.

The spaCy documentation describes tokenization, lemmatization and dependency parsing as separate operations; tokenization in that library preserves the original string. The result depends on the language, components and model selected. This supports separating the processing layers, rather than guaranteeing correct extraction of product fields.

Lemmatization must also be distinguished from mapping actions to a shared domain operation. Russian «промыть» and «промывать» (‘rinse’, perfective and imperfective respectively) are different aspectual lemmas. Mapping them to a shared care operation requires a separate rule that retains the original form. The UD description of aspect specifically notes that Slavic aspectual pairs belong to different lemmas.

Removing function words is particularly risky in this line: Russian «по» (‘each’) helps distinguish the volume of one item from that of the whole set, while «до» (‘before’) retains the time boundary. One representation may be acceptable for a search index; checking an instruction requires a more complete one.

Attach a quantity to its entity before calculating

In line 2, the number six refers to tubes. The construction «по 20 мл» (‘20 ml each’) specifies the volume of the contents of each tube. The domain record must retain the two quantities separately, even if the next stage needs only the total volume.

In UD, the relation nummod describes a numerical modification of a noun. Assigning a syntactic relation does not itself calculate the contents of a set or determine the role of every number in a document. A number in a model code may be a designation rather than a quantity.

«6 туб общим объёмом 20 мл» (‘6 tubes with a total volume of 20 ml’) specifies a different distribution. «20 туб по 6 мл» (‘20 tubes of 6 ml each’) again totals 120 ml, but describes a different set. Comparing only total volume would therefore miss the difference in the number of tubes and the volume of the contents of each.

The calculation is stored as a separate inference: the inputs are 6 tubes and 20 ml per tube, the operation is multiplication, and the result is 120 ml. It cannot establish consumption, the number of paintings or a dilution ratio. Those questions require other parameters.

The full dossier in the first article does specify six colours. Here, that is a different record with its own evidence. Extraction from the selected excerpt must not silently borrow it from memory and then present it as information contained in line 2.

Move from grammar to domain roles

In the chosen interpretation of the first line, Russian «грунтованного» (‘primed’) modifies canvas. Expanding the list requires three records: paper; primed canvas; wood. Copying the modifier into every record would add conditions for paper and wood; removing it would broaden the claim about canvas.

Syntactic agreement and the structure of the list help with analysis, but do not resolve every ambiguity. The UD description of coordination notes a limitation of the basic representation: it does not always distinguish a dependent of the first element from a shared dependent of the whole group. Domain-level scope may require additional annotation.

In line 3, the action is rinsing. Brushes are the object being cleaned; water is the cleaning medium. «После работы» (‘after use’) and «до высыхания краски» (‘before the paint dries’) specify the timing of the action; «сразу» (‘immediately’) qualifies the first condition. The state of the paint does not become a property of the water or brushes.

The labels case and obl describe linguistic relations. Not every obl can be relabelled as a ‘medium’, nor every preposition as a ‘condition’. Assigning a domain role depends on the construction, context and extraction task.

Replacing brushes with clothing creates a new instruction. Replacing «до» (‘before’) with «после» (‘after’) changes the timing recommendation. The identified lemmas «вода» (‘water’) and «краска» (‘paint’) cannot reconstruct a lost relation. Acceptance checks must examine the complete action record.

What a verifiable record looks like

The following records can represent our excerpt. This is an author-defined format for explaining the method; it is not proposed as a universal schema for every category.

The record also retains the document version, product scope and derivation method: extracted, normalized or calculated. The original quotation remains available. If another document contradicts this record, the new relation is recorded separately; it does not erase the earlier evidence without a decision.

An unknown field must remain unknown. The permitted proportion of water for dilution does not follow from water appearing in the care instructions. A dilution record needs another sentence from the source dossier; no dosage is given for the fictional Alfa set.

Normalize a record while retaining the original

If the source specifies 0.02 l for the same tube, a unit-conversion rule gives 20 ml. The normalized value helps with comparison, but the original 0.02 l and the entity measured must remain alongside it. Unit conversion does not turn the volume of one tube into the volume of the set.

Recognizing text in an image is a different task. In ‘2O мл’ (‘2O ml’), the last character in the apparent number may be the letter O rather than zero. Correcting it to match the expected appearance of the string must not be presented as direct reading of a reliable number. When in doubt, inspect the image or use another source, and retain the disputed recognition separately.

The code A6-20 must not be parsed as subtraction, nor «до 20 мл» (‘up to 20 ml’) as an exact value of 20 ml. Classify the span before calculating: an identifier, equality, range and limit require different records. «Не более» (‘no more than’) and «не менее» (‘no less than’) must stay with the boundary.

Linguistic features help process negation, but do not replace scope analysis. In UD v2 the former universal relation neg was removed; negation is represented by the feature Polarity=Neg with an ordinary syntactic relation. Domain analysis must still establish whether the negation concerns suitability, the need for an action or another property.

A formatting requirement may still be literal. Russian «двадцать миллилитров» (‘twenty millilitres’) and «20 мл» (‘20 ml’) can express the same quantity but fail an exact-string requirement. The results of semantic and literal checks must be retained separately.

Make ambiguity an explicit result rather than a hidden guess

In a real document, a line may lose its heading during PDF extraction or appear beside an image of another variant. A grammatically well-formed record then still lacks a reliable connection to the product. The issue must return to document reconstruction and identification.

The nearest word does not resolve every ambiguity. If permission to wash something follows a reference to both a mug and its lid, a pronoun may allow two possible objects. In that case, retain the candidates and a clarification question. Assigning the property to whichever object is convenient would expand the source information.

In our excerpt, the context establishes that the brushes are tools for paint. The isolated Russian lemma «кисть» does not establish that sense: the word can also refer to the hand as a body part. Domain interpretation must be linked to the surrounding text, not merely to the dictionary form.

An analyser's confidence and the sufficiency of the evidence are different matters. Even a stable decision about a linguistic relation does not establish that a manufacturer's recommendation is true. Publication also requires applicability to the product and verification of the source itself.

Test extraction against changes that preserve or alter meaning

For acceptance testing, define the expected records before running the system. Check each attachment first, then the entire claim with its conditions. A matching number attached to the wrong entity does not count as correct attribute extraction.

These variants alter the illustrative input; they do not add to Alfa's original dossier. The principle of different expectations for different edits helps detect both insensitivity to a condition and unwarranted rejection of a valid paraphrase.

Process evaluation must account for omitted information and relations added without evidence. If the system extracts only simple numbers and does not process the instruction, high precision among the remaining records does not describe the completeness of the analysis. Define the counting unit — field, relation or whole claim — in advance.

The output consists of source-linked records about application, set contents and care, together with a separate total-volume calculation. They can now be retrieved and compared with other phrasings. Such comparison requires text representations and a similarity measure; similarity alone does not establish that two questions require the same answer.