Ask an AI system the same factual question several times and you may receive different sentences. Some differences change the answer. Others only change how the answer is expressed. Counting every new sentence as a new factual position mixes those two kinds of variation.

Research on semantic uncertainty makes that distinction explicit: it considers uncertainty across meanings rather than only across exact sequences of words [1]. The researchers’ later explanation of semantic entropy also separates changing the wording from changing the underlying answer [2]. This offers a useful way to inspect consistency, but it does not make consistency equivalent to truth.

We will use a fictional gallery and a complete six-reply probability model. Every probability and meaning group is supplied in advance, so the numbers can be checked without running an AI model. Then we will deliberately alter the grouping, the observed sample and the factual reference. Those changes show which question the uncertainty score answers, which assumptions it uses, and why a confidently repeated mistake can remain a mistake.

Fix a question and generation context. Let S denote a complete output string and G=g(S) its assigned meaning group. When g is a deterministic partition of the string space, uncertainty over S and uncertainty over G are different quantities. The 2023 semantic-uncertainty paper defines group probability by summing the probabilities of strings sharing a meaning [1].

The construction below specifies the complete finite distribution, rather than estimating the mass of an open-ended language space. We use base-two Shannon entropy in bits, hand-assigned groups and an explicit question. It is not an execution of the paper’s NLI system, its Monte Carlo estimator or a present-day hallucination detector.

Three additional distinctions matter: a grouping judgment can be wrong, an observed sample need not equal its generating distribution, and the distribution itself contains no external truth label. The Oxford authors describe their later method as addressing a particular class of variable, unsupported answers rather than every mechanism producing an incorrect response [2]. Our examples make those boundaries concrete.

The question is: “What is the fictional gallery’s one weekly closed day?” Our invented reply source can produce exactly six sentences. The first is “The gallery’s only closed day is Monday.” The second is “Monday is the gallery’s one weekly closure day.” We assign both to the Monday group.

The other four are: “The gallery’s only closed day is Tuesday.” “Tuesday is the gallery’s one weekly closure day.” “The gallery closes only on Tuesdays each week.” “Its single weekly day of closure is Tuesday.” We assign all four to the Tuesday group. The question’s one-day condition and the explicit wording keep the intended grouping narrow.

These are six different strings but only two assigned answers. We have not asked a classifier to discover their meanings, and we have not yet said which day is factually correct. The distinction is deliberate. A grouping describes which replies express the same answer under the stated question; a factual reference supplies information about whether that answer matches the gallery’s actual schedule in our fictional world.

Define six strings s₁ through s₆ in the displayed order. Strings s₁ and s₂ assert Monday as the gallery’s only weekly closure day; s₃ through s₆ assert Tuesday. Let g(s₁)=g(s₂)=M and g(s₃)=g(s₄)=g(s₅)=g(s₆)=T. The full wording and ordering are part of the retained example record.

The sets M and T form a disjoint, exhaustive partition of this finite space. Equivalence here is assigned for the one stated question; no classifier accuracy or universal paraphrase judgment is inferred. The mathematical definition requires every possible string to belong to exactly one group, so their probabilities can be aggregated without overlap or omission [1].

A truth value is not part of g. Identifying the proposition asserted by a reply does not establish whether the proposition is true. This separation lets the example hold the language distribution fixed while later changing the external reference. It also prevents a meaning label such as M from being silently interpreted as a correctness label.

Assign each Monday sentence probability one quarter. Assign each Tuesday sentence probability one eighth. These six probabilities add to one, and there are no other possible replies in the construction. Although Tuesday has more different phrasings, each of its phrasings is individually less likely.

The Monday group therefore has probability 1/4+1/4=1/2. The Tuesday group has probability four times 1/8, also 1/2. Simply counting the distinct sentences would instead give Monday two out of six and Tuesday four out of six. That would answer a different question because it would treat the six strings as equally likely.

The semantic-uncertainty paper defines a meaning’s probability through the sum of its member strings’ probabilities [1]. Our small inventory makes that sum exact. With a real language model, the possible strings are not limited to a short list we have completely supplied. An observed list of unique replies therefore needs its own sampling and weighting explanation before it can be treated as a complete probability distribution.

Set p=(1/4,1/4,1/8,1/8,1/8,1/8). Then Σp_i=1, P(G=M)=p₁+p₂=1/2 and P(G=T)=p₃+p₄+p₅+p₆=1/2. This implements the finite version of P(c|x)=Σ_{s∈c}P(s|x) from the paper’s meaning-probability definition [1].

A uniform distribution over the six distinct strings would instead induce group masses 1/3 and 2/3. It is a different stipulated distribution, not another way to compute the same p. Removing duplicates from a recorded batch and assigning equal weight to the survivors can likewise change the quantity being summarized.

Our p values are normalized probabilities over the entire declared output space. They are neither arbitrary similarity scores nor length-adjusted token likelihoods. The source discusses sequence probabilities, variable lengths and length-normalization choices separately [1]. Those distinctions matter in a real estimator; the exact finite calculation here does not establish that any set of displayed scores can be summed as probability mass without further justification.

Entropy is a way to summarize how spread out a probability distribution is. If only one outcome is possible, its entropy is zero. If two outcomes each have probability one half, the entropy is one bit. The word “bit” here names the information unit used by the logarithm; it is not a percentage of factual confidence.

For our six strings, the two quarter-probability replies contribute a total of one bit, and the four eighth-probability replies contribute one and a half bits. The wording distribution therefore has entropy 2.5 bits. After combining strings into their two meaning groups, the two half-probability answers contribute one bit in total.

Both numbers are correct, but they summarize different outcomes. The first includes uncertainty about the wording inside each answer. The second keeps only the uncertainty between Monday and Tuesday. The difference does not mean that a detector has removed 1.5 bits of false information. No factual reference has entered either calculation, so neither result yet says which answer is right.

For a finite distribution, define H(P)=−Σ_i p_i log₂p_i, with the zero-probability contribution defined as zero. The paper’s semantic-entropy expression has the same grouping structure [1]; we choose base two explicitly, so all results below are in bits rather than leaving the logarithm’s unit implicit.

Each p=1/4 contributes −(1/4)log₂(1/4)=1/2 bit. Each p=1/8 contributes 3/8 bit. Hence H(S)=2(1/2)+4(3/8)=5/2 bits. The meaning masses give H(G)=−2(1/2)log₂(1/2)=1 bit.

Entropy concerns the concentration of the declared distribution over its outcomes. It does not itself define a decision threshold, a probability that a claim is false or a calibrated correctness score. Changing the log base rescales the numerical entropy while leaving the grouping and probability masses unchanged. Reports must retain the quantity, unit and outcome space before comparing their values or attaching a downstream interpretation.

Once you know the answer group is Monday, the two Monday sentences are equally likely. Choosing between those phrasings contributes one bit of remaining uncertainty. Once you know the group is Tuesday, its four sentences are equally likely, contributing two bits.

The groups themselves each occur half the time. Their average remaining wording uncertainty is therefore half of one plus half of two, or 1.5 bits. Add the one bit of uncertainty between Monday and Tuesday, and the result is the original 2.5-bit wording entropy. Nothing has disappeared from the accounting; we have separated two questions that were previously combined.

This explains why a varied writing style need not imply varied factual positions. A reply source could have several ways to express one answer while being completely concentrated on that answer’s meaning. Conversely, two short, nearly identical strings can assert different days. Similar appearance and shared meaning need to be assessed at the relevant level, which is the motivation behind the semantic distinction in the research [1][2].

The conditional string distribution within M is (1/2,1/2), giving H(S|G=M)=1. Within T it is (1/4,1/4,1/4,1/4), giving H(S|G=T)=2. Thus H(S|G)=P(M)·1+P(T)·2=3/2 bits.

Since G is determined by S, H(S)=H(G)+H(S|G)=1+3/2=5/2. One can derive the identity directly by writing p(s)=P(g(s))P(s|g(s)) inside the logarithm and distributing the weighted sum. This calculation uses our complete partition and normalized distribution, not an estimated latent meaning assignment.

The identity also establishes H(G)≤H(S) for this deterministic grouping because conditional entropy is nonnegative. The reduction measures the variation discarded by the grouping map; it is not automatically a measure of removed factual error. If the map merges distinct factual claims, a smaller score can reflect a worse representation of the question rather than improved reliability. Group quality and entropy arithmetic require separate checks.

The research operationalizes shared meaning using inference in both directions, with the question included as context [1]. That context matters: a short day name can answer our gallery question, while the same word on its own may not specify a complete proposition. Comparing bare words without their question can discard information needed to understand them.

Claim boundaries matter too. “The gallery closes on Tuesday” and “The gallery closes on Tuesday and offers free entry on Wednesday” are not equivalent as complete factual statements. The second includes an extra assertion that the first does not establish. If a system keeps only the closure-day answer, it may group that part together while leaving the Wednesday assertion outside the check.

A useful record says which of those tasks it performed. Grouping an answer to one narrow question is different from checking every claim in a paragraph. The Oxford authors describe a separate approach for longer passages that works with individual claims and associated questions [2]. Our six-reply example deliberately avoids additional claims so the partition’s scope stays visible.

The 2023 method uses bidirectional entailment on question/answer combinations rather than treating surface similarity as equivalence [1]. The context supplies the proposition for an otherwise incomplete answer. In the formal construction, E must identify the same meaning under a fixed relevant context and claim scope.

Let A assert only a Tuesday closure, and B assert that closure plus free Wednesday entry. Under a complete-statement reading, B entails A, but A does not establish B. A projection that retains only closure day can map both to the same day label while discarding the extra claim. That projection may suit a narrow question; it does not verify the omitted Wednesday assertion.

Thus the group definition should record whether it covers the complete response or a selected proposition. The 2024 author explanation distinguishes a longer-passage extension using claim extraction and associated questions [2]. We do not implement that extension or assume its error rate. The finite partition used for our entropy calculation is explicitly narrower and is supplied as part of the example.

Keep our six reply probabilities unchanged. First pretend that each wording has its own meaning, even when two sentences say the same day. The resulting score is the full 2.5 bits. The grouping has counted phrasing differences as differences between answers.

Now make the opposite mistake and put all six sentences into one group. The only group has probability one, so its entropy is zero. This falsely merges Monday and Tuesday. The intended two-group partition gives one bit between those extremes. The probability list did not change in any of the three calculations; only the grouping did.

This is why a low score should not be inspected without its clusters. A system can create apparent certainty by merging claims too broadly, or apparent uncertainty by splitting equivalent wording. The researchers discuss both context-sensitive grouping and cases in which differences in answer specificity affected a result [1][2]. Our extreme partitions are constructed illustrations, not measured failure rates for those methods or for any current model.

Under the identity partition g_identity(s_i)=i, H(G_identity)=H(S)=5/2. Under the constant partition g_all(s_i)=c for every i, H(G_all)=0. The intended M/T partition has H(G)=1. These are exact values under one fixed p, demonstrating that the grouping map is part of the statistic’s definition.

Refining a deterministic partition can retain more distinctions; coarsening it can discard them. Neither direction alone establishes whether the retained distinctions correspond to the reader’s factual question. The identity map is too fine for our assigned paraphrases, while the constant map is too coarse for the two different closure days.

Inspect group membership and the underlying comparison decisions alongside the reported entropy. The source paper acknowledges imperfect equivalence judgments, and the later author account describes context and specificity issues [1][2]. Those observations support checking the map; they do not supply a universal error probability or justify assuming that a detector’s lowest attainable score corresponds to its most trustworthy factual answer.

A mathematical equivalence relation is consistent across linked comparisons. If A has the same meaning as B, and B has the same meaning as C, then A and C must also match. An imperfect automated comparison can fail that requirement. The source paper explicitly acknowledges this issue and notes that grouping can then depend on comparison order [1].

Consider an invented judge that marks A with B as matching, B with C as matching, and A with C as different. Our toy grouping rule compares each new item with the first item in each existing group. Process A, then B, then C, and the result is a group containing A and B plus a separate C. Process B first, followed by A and C, and all three join B’s group.

The pairwise outputs stayed the same while the resulting groups changed. Give each item probability one third: the first grouping has about 0.9183 bits of entropy, while the second has zero. These are fictional comparison outputs under a fully specified rule. They show why preserving only the final score can hide an unresolved grouping problem.

Define a symmetric invented relation R with R(A,B)=R(B,C)=true and R(A,C)=false, plus reflexive self-matches. It is not transitive, so it is not an equivalence relation. The source paper’s appendix identifies this possible gap between formal equivalence and approximate pairwise classification [1].

Use an explicit first-representative rule: assign an item to the first existing group whose first member matches it; otherwise create a group. Order (A,B,C) yields ({A,B},{C}), while (B,A,C) yields ({B,A,C}). With equal input probabilities, their group distributions are (2/3,1/3) and (1), giving entropies approximately 0.918295834 and zero.

This is our own small grouping algorithm, not a run or reproduction of the paper’s implementation. It establishes order dependence for these declared pairwise outputs. A record of the input order, pair decisions and grouping rule makes the discrepancy reproducible. A single score cannot identify whether a change came from the reply distribution or from how inconsistent comparisons were resolved.

Return to our original Monday/Tuesday probabilities of one half each. Imagine a constructed record of eight replies containing seven Monday answers and one Tuesday answer. Its observed group fractions are seven eighths and one eighth. The entropy of that observed record is about 0.5436 bits, even though the declared generating distribution has one bit.

The difference does not mean either calculation is arithmetically wrong. One summarizes the complete probability rule we supplied; the other summarizes one particular finite record. An extreme record could contain eight answers from the same group and therefore have observed entropy zero.

Under independent draws from our fair two-group rule, there are 256 equally likely sequences of eight group labels. Two contain only one group: all Monday or all Tuesday. Thus zero observed meaning entropy occurs with probability 2/256=1/128 in this construction, despite the underlying one-bit distribution. That number depends on the stated sampling assumptions; it is not a real detector’s error rate or a recommended number of requests.

For the declared independent meaning law P(M)=P(T)=1/2, consider the possible empirical count vector (7,1) over n=8. Its plug-in entropy is Ĥ=−(7/8)log₂(7/8)−(1/8)log₂(1/8)≈0.543564443 bits. The generating H(G) remains exactly one bit.

Enumerating the 2⁸ equiprobable meaning-label sequences gives two with a single occupied group. Therefore P(Ĥ=0)=2/256=1/128 for this specific fair independent law. The event of all Monday alone has probability 1/256. The six underlying strings are not equiprobable; the enumeration works because their aggregated meaning labels are fair and the draws are assumed independent.

This is an exact finite-sampling calculation, not the paper’s probability-weighted or Monte Carlo estimator [1]. The later author explanation also describes a discrete variant without token probabilities [2], but we do not identify our example with an implementation of that variant. Empirical frequencies, full probabilities and research estimators must keep their separate definitions and assumptions.

The earlier probability of an all-Monday record assumed independent sampling from our stated distribution. Change the procedure and that probability can change completely. Suppose a new rule always chooses the first highest-probability complete string in our six-string inventory. It always returns the first Monday sentence, so eight repetitions always agree.

That deterministic procedure is not the previous sampling experiment. Its agreement does not show that the original distribution assigned all its probability to Monday. Nor have we described how a real model’s token-by-token decoding chooses a complete sequence. The rule is deliberately defined over our small list so its effect is unambiguous.

The research discusses how sampling method and temperature affect uncertainty estimation [1]. For a real record, preserve the model, question, context, settings and method used to obtain the replies. Repeating a deterministic selection, changing the prompt between requests and drawing independent samples from one distribution are different procedures. Their outputs should not silently receive the same statistical interpretation.

Define a new complete-string selection rule d(p)=the first index attaining max_i p_i. For our p, indices 1 and 2 tie at 1/4, so the rule selects s₁. Repeating d yields an all-M record with probability one under that deterministic protocol, whereas eight independent samples from p yield all M with probability 1/256.

This selection rule is expressly not a claim about greedy token decoding, which operates at a different boundary. The contrast only shows that the output protocol participates in the probability statement. The original distribution, a deterministic choice from it and a sampled record are different mathematical objects.

The source paper discusses sampling methods, temperature and variable-length probability treatment [1]. Our comparison uses none of its empirical parameter recommendations. Record those choices and any dependence between requests before interpreting sample entropy. A changing context or sampling policy can alter the distribution being observed, so a difference in measured reply variation cannot by itself identify a change in factual knowledge.

Consider a different reply source that chooses between the two Monday sentences with equal probability and never says Tuesday. It has one bit of wording variation but zero uncertainty between meanings. Every reply agrees on Monday.

Now supply a fictional schedule stating that Tuesday is the gallery’s only closed day. Both possible replies are false under that reference. If the schedule instead stated Monday, both would be true. The reply distribution and its zero semantic entropy are identical in the two comparisons; the external fact changed.

This is the limit of using agreement as verification. Agreement tells us something about how the source responds, while the schedule supplies information about the world it is describing. The Oxford authors explicitly list consistent incorrect facts or reasoning among the error mechanisms their approach should not be relied on to catch [2]. An uncertainty signal may help decide where to investigate, but a low value does not supply the missing factual evidence.

Take a separate distribution q with q(s₁)=q(s₂)=1/2 and zero mass elsewhere. Then H_q(S)=1 bit while P_q(G=M)=1 and H_q(G)=0. Define factual correctness by an external reference r for the gallery’s sole closure day.

For r=Tuesday, every possible q output is incorrect. For r=Monday, every possible q output is correct. The identical q and identical entropy therefore coexist with opposite correctness outcomes under different stipulated references. Entropy cannot recover an external label absent from its inputs.

The 2024 author explanation limits its target and names consistent erroneous reasoning or facts, failures in a new context and other mechanisms outside a blanket guarantee [2]. We use no claim about a present model’s prevalence of those failures. The constructed counterexample is enough to separate concentration over meanings from evidence of truth, without assigning an unsupported numerical probability of correctness to the uncertainty score.

Sampling-based consistency is a useful research direction, but different methods ask different questions. The SelfCheckGPT publication abstract describes comparing sampled responses without an external database [3]. That describes where its comparison material comes from; it does not turn sampled agreement into a new independent observation of the gallery’s opening schedule.

A clear review record keeps the original question, exact replies, meaning groups, grouping decisions, weights and sampling procedure. It also names the entropy unit and any factual evidence used separately. For a longer response, record which claims were checked and which additional details remain outside the grouping. That makes it possible to challenge an overbroad merge or a factual leap rather than merely rerun an unexplained score.

Our six replies contained two equally probable meanings despite having unequal numbers of phrasings. Their wording entropy was 2.5 bits and their meaning entropy one bit. Those exact results explain the distinction. Whether Monday or Tuesday is correct remains a separate question, answered by the declared factual reference in the example and by suitable evidence in a real task.

SelfCheckGPT’s 2023 abstract presents sampled-response consistency as an approach without an external database [3]. Our article cites that publication-level description only; no full-paper score formula, current performance result or implementation has been inferred from the abstract. Its existence reinforces the need to name the particular detector and evidence boundary rather than treating all self-checking methods as one guarantee.

A reproducible record includes fixed context, complete strings or retained samples, probability/weight provenance, comparison decisions, grouping rule, entropy definition and any separate factual reference. For our exact construction, the full distribution and assigned partition make every value auditable. A real estimate needs additional sampling and model-evaluation evidence appropriate to its intended use.

The central distinctions remain operational: identify which replies express the same proposition, compute the statistic for the supplied distribution, and verify factual claims using suitable evidence. A sample count or a low entropy threshold alone completes none of the other steps. Keeping these boundaries explicit allows uncertainty to inform review while preserving the difference between consistent language and a supported answer.