Which hypotheses about a text stand up to testing

A hypothesis begins with a possible objection

“Make the product page better” does not define an outcome that can be refuted. A test must name the change, expected result, conditions and criterion that would require reconsidering the hypothesis. Not every test ends with a simple yes or no: the data may be insufficient.

An analytical hypothesis for T500: questions about the result at +20 °C and in freezing weather can use the same numerical result without distinguishing temperature conditions. The dossier contains only a test at +20 °C: 450 ml of a drink initially at 95 °C remained in the closed mug for six hours, with a final temperature of at least 60 °C. A common answer promising the same result in freezing weather is unsupported.

The dossier does not justify applying the measured result to both conditions. That supports keeping the questions distinct; it does not prove they cannot be discussed in one section. Nor does it establish the number of winter buyers or demand for a separate model. Question boundaries are checked against the information and conditions needed for an answer, not the product page's sales.

A different hypothesis must be formulated separately: “Under the chosen comparison plan, version B more often enables participants, without a prompt, to name +20 °C and correctly say that the freezing-weather result is unknown.” This needs observations of page reading. The mere presence of +20 °C in the markup does not answer it. One counterexample can refute a universal interpretation; one reader's failure does not yet refute an average improvement.

Three kinds of testing support different conclusions

An analytical test compares an interpretation with the original phrase and its context, retains alternatives and seeks cases where the proposed grouping breaks down. Checking search intent may require talking to a user. Frequency alone does not establish intent.

A comprehension study records the answer found, its interpretation and the route taken to find it. A behavioural study observes actions within a defined comparison. “It is clearer now” may help explain the interface. It does not replace measuring purchases, and a purchase does not prove that every condition was understood correctly.

Preserve these distinctions in reports too. “The evaluator distinguished the conditions in five teaching cases” and “conversion increased” require different data. Even without deliberate exaggeration, a broad heading such as “hypothesis confirmed” may hide which level was actually tested.

First, check what the person read

Prepare two accurate pages for the study. In A, the full protocol sits in a clearly named separate section with a prominent link. In B, the same section is placed closer to the specifications. Both versions distinguish the nominal 500 ml from the 450 ml used in testing. Neither includes an unconditional “six hours in freezing weather” promise.

Give participants this task: “At what ambient temperature was the mug tested, and what does that result tell you about a walk in freezing weather?” A successful answer must name +20 °C and the limit of the information about freezing conditions. Evaluate search time alongside correctness. Quickly finding a wrong answer does not complete the task.

Observe what the person opens, which words they rely on and where they draw an inference. A leading question such as “did you notice the restriction beside the temperature?” changes the test itself. When comparing variants, their presentation must account for learning: after seeing the first protocol, participants already know the conditions and may find them in the second from memory.

A small targeted group can uncover difficulties and inform the next edit. It does not automatically provide an accurate proportion of all buyers. Estimating error frequency or differences between variants requires appropriate selection, sample size and comparison design. “I would buy it” also remains a stated intention.

Suppose participants start finding the section but still mistake 450 ml for the nominal capacity. The test has exposed a different problem: the placement and labelling of parameters. A positive result for time cannot be extended to comprehension.

How to design a behavioural comparison

A purchase question needs a different plan. A separate behavioural hypothesis is that version B increases the share of users who buy T500. Kohavi, Henne and Sommerfield discuss the basis of randomised comparisons, the assignment unit and power planning. For our two pages, the following fictional design is possible.

At their first eligible visit to the product page, participating users are randomly assigned A or B and remain in that group. Eligibility conditions are set before exposure. Both groups are observed concurrently. Price, availability and other page elements operate under the same rules; changes and failures are recorded.

The primary metric could be the proportion of assigned users who buy T500 within a preselected observation window. This tests a change in purchase frequency. A hypothesis about a mistaken choice needs a separate measure tied to the cause of the mistake; buying or not buying does not establish that cause. Define what counts as a purchase and how repeat visits are combined. Do not switch the denominator to “people who read the section” if the new layout itself affects reading: that would select on an event influenced by the variant.

Record exposure to the variant and whether the section actually loads to check that the experiment works. If B fails to load for some assigned users, they cannot simply be removed while retaining the original interpretation. Investigate the cause and assess which conclusions remain justified.

Choose guardrail metrics in advance, such as loading errors and cancellations or returns with an appropriate observation period. Specify each denominator. Before launch, define the purchase window from group assignment and the return-observation period from order or delivery. Analyse the final result only when the required period has ended for the last enrolled participants too. The return rate among orders is useful diagnostically, but the composition of buyers may also change; it cannot automatically be read as an effect on all assigned users.

If the heading is rewritten and the photograph replaced while the section is moved, the experiment compares that whole package. The result cannot be attributed to one line of conditions without an additional comparison.

Decide in advance which effect can be distinguished

Before launch, define an effect that matters in practice and plan enough observations. The required number depends on the baseline event rate, variability, analysis unit and chosen method. There is no universal visitor count or duration for every product page; sample-size planning requires initial assumptions.

Low traffic may still reveal a major comprehension problem without distinguishing a small change in purchases. These are different tasks with different data requirements. If the necessary sample is unavailable, narrow the research question rather than declaring a small sample sufficient because it is convenient.

The final analysis needs both the estimated difference and its uncertainty, which can be expressed as a confidence interval. No statistically significant difference does not prove zero effect. A wide interval may leave both a useful improvement and substantial harm compatible with the data. “The variants are equal” would require an appropriate design and predefined bounds for a practically negligible difference.

A small but reliably distinguishable change does not necessarily justify deployment either. Compare it with costs, guardrails and the task's objectives. Set those criteria before choosing a result you like.

Monitoring a test and choosing a winner are different actions

Microsoft ExP recommends monitoring metrics during the experiment to detect failures and deterioration. Check whether the actual group ratio matches the planned one. An unexpected discrepancy is called SRM — sample ratio mismatch; investigate its cause before interpreting the effect. Ordinary random differences in group sizes do not in themselves establish SRM.

Repeated significance testing and early stopping require a statistical procedure that accounts for that decision process. Waiting for the first favourable number in an ordinary report and stopping the test is not the same as following a preselected sequential analysis.

For the fictional T500, this means two separate rules: what happens if exposure fails or substantial harm appears, and when the primary result is evaluated. Another company's historical test duration does not replace our plan. Monitoring helps manage the study, but does not authorise an arbitrary reporting time.

What to record after testing

The report must make the comparison reconstructable: page versions, participants and inclusion conditions, assignment method, primary metric, guardrails, observation window, departures from the plan and result uncertainty. Unsuccessful or inconclusive outcomes are retained alongside successful ones.

Moving the conditions may reduce purchases if some visitors previously expected an untested result in freezing weather. That scenario needs data, but it shows why higher conversion is not the only conceivable sign of quality. Behaviour must be compared with the task and the nature of selection mistakes.

“Sales increased after the update” is useful for generating hypotheses. It does not rule out changes in price, advertising, season or visitor composition. If the platform does not permit a valid randomised comparison, keep that limitation in the conclusion. Calling something A/B does not remove it.

Our fictional plan could produce three independent records: whether grouping the questions is justified; how participants found and understood the answer; and what the comparison established about behaviour. When extending the decision to a catalogue, check applicability and preserve the conditions under which the result was obtained.