A language model normally generates a continuation through repeated next-token decisions. Speculative decoding tries to make that process faster by asking a cheaper draft mechanism to propose several candidates, then using the target model to evaluate the proposed path. If enough candidates can be kept, one target evaluation round can advance the output by more than one token. The draft is useful because it supplies work to check ahead of time, not because it is trusted to replace the target's decisions. [1][2]

The interesting part is what happens when the draft proposes a token too often. Simply accepting every plausible suggestion would change the output probabilities. Simply throwing away a proposal and taking an ordinary fresh target sample is not always the right repair either. This article works through an invented three-token distribution to show why the replacement needs a carefully chosen rule. It concerns the exact sampling procedure in the cited research, not every system marketed as speculative decoding. The target still sets the desired distribution; an accepted token has passed a sampling calculation, not an independent factual review of what the resulting sentence means.

Speculative decoding uses an efficient proposal mechanism to expose candidate continuations for parallel target evaluation. The target model's conditional distribution remains the reference. The cited algorithm accepts a contiguous candidate prefix and draws an additional token using the appropriate target or corrected distribution, potentially producing several tokens per serial target round [1]. Hugging Face's versioned implementation documentation likewise distinguishes assistant proposals from main-model validation [4]. The optimization changes execution structure while preserving the specified sampling law under the algorithm's assumptions.

The key correctness question is how a proposal distribution q can differ from the desired distribution p without biasing the final sample. We examine a complete three-outcome example, including the rejected branch, rather than treating verification as a vague test of whether text looks reasonable. All probabilities and runtime numbers are invented teaching inputs. No model inference or speed benchmark was performed. The explanation applies to the exact acceptance-and-residual method described in the May 2023 paper version; approximate verification schemes and other speculative variants require their own correctness arguments.

A probability distribution gives each possible next token a chance of being selected. Suppose a rule gives token A a 60 percent chance. Preserving that rule means preserving the chance, not promising that every individual run outputs A. Two legitimate draws can differ. In the research, the comparison is between the target's sampling distribution and the distribution produced by the speculative procedure. [1][2] It is not a claim that the target's most likely sentence is necessarily true.

This distinction also explains why a matching random-seed label is not the whole argument. Different implementations can consume randomness in different ways; the mathematical statement is about the resulting probabilities. To establish reproducible behavior for a particular program, its actual implementation needs separate checks. The target distribution must also include the sampling choices you intend to use, such as a temperature adjustment or restrictions on candidate tokens. The paper explicitly treats these choices as adjustments to the probabilities before its sampling method is applied. [1] A claim of unchanged sampling becomes unclear if the baseline and accelerated system quietly use different rules.

The preservation claim concerns the conditional next-token law p(x given prefix), and by repeated valid conditional sampling, the corresponding sequence distribution [1][2]. It does not establish factual accuracy, an external quality score or sample-by-sample identity under an arbitrary coupling of random draws. Equality in distribution is a mathematical property of probabilities, whereas identical output traces additionally depend on random-number consumption and implementation choices. These are related but different verification targets.

The paper defines p and q after their respective sampling-policy adjustments, including the transformation of greedy selection into a degenerate distribution [1]. In practical terms, the desired baseline must be identified together with temperature, token filtering and other probability processing. Hugging Face's v4.49.0 guide also distinguishes decoding strategy from logits manipulation [4]. Comparing raw target logits in one path with transformed probabilities in another would not implement the same stated problem. Our toy distributions below are already normalized effective distributions for one fixed prefix, with no hidden temperature, truncation or vocabulary conversion step.

For one fixed prefix, let the target probabilities be A: 0.60, B: 0.30 and C: 0.10. Let the draft probabilities be A: 0.50, B: 0.25 and C: 0.25. Each list sums to one. The letters are invented token labels, not words from a real model vocabulary. The draft gives C more probability than the target does, and gives A and B less. Accepting every draft sample would therefore produce the wrong distribution for this example.

Notice that this is more specific than calling the draft less intelligent. We have a precise mismatch that can be repaired mathematically. A draft sample of C is not a false statement; it is one of the allowed outcomes, proposed too frequently relative to the chosen target. Likewise, A is not guaranteed to be the correct answer because it has the largest target probability. The distinction keeps the worked example focused on sampling. The research names the target distribution p and the approximation distribution q [1]; we will use those labels while keeping the same three numbers throughout the probability calculation.

Let the common candidate alphabet be {A,B,C}, with target p=(0.60,0.30,0.10) and proposal q=(0.50,0.25,0.25) at one specified prefix. Both distributions normalize to one and all proposal probabilities are positive. The setup follows the paper's target/proposal distinction [1] while using original numerical values. It is deliberately a shared-alphabet calculation; tokenizer alignment and cross-vocabulary proposal mechanisms are outside this small example.

The discrepancy is directional: q overallocates 0.15 probability mass to C, while underallocating 0.10 to A and 0.05 to B. This supplies an explicit accounting problem. Keeping every proposal would reproduce q rather than p. A valid correction must preserve enough proposal mass where useful and distribute the rejected mass to the target's deficit. These statements concern probability mass, not semantic correctness or measured model capability. Calling the operation target verification should not obscure that it evaluates compatibility with a specified sampling law, rather than certifying the truth of an invented token label.

The cited rule accepts a draft token with probability min(1, p divided by q) for that token [1]. The min instruction means choose the smaller of one and the ratio. For A, 0.60 divided by 0.50 is 1.2, so acceptance is capped at 1: always accept an A proposal in this example. B also has ratio 1.2 and is always accepted. C has ratio 0.10 divided by 0.25, or 0.4, so a C proposal is accepted 40 percent of the time.

Those are acceptance chances after a particular token has been proposed. To find their contribution across all trials, multiply by the chance of proposing that token. A contributes 0.50 × 1 = 0.50; B contributes 0.25 × 1 = 0.25; C contributes 0.25 × 0.4 = 0.10. The accepted branch totals 0.85. The remaining 0.15 is the probability of rejecting a proposal. It is not another 15 percent to discard permanently; the procedure must still produce an output token on that branch. That is where the replacement distribution becomes important.

For a proposal x drawn from q, the algorithm uses acceptance a(x)=min(1,p(x)/q(x)) [1]. Our values give a=(1,1,0.4). The conditional acceptance probability for C is therefore 0.4, while its unconditional accepted mass is q(C)a(C)=0.10. Confusing those two quantities would overcount C. The accepted masses for A and B are 0.50 and 0.25, respectively.

In general, q(x)a(x)=min(p(x),q(x)). Here the total acceptance probability is beta=0.50+0.25+0.10=0.85, leaving rejection probability 1-beta=0.15. This is a complete branch probability calculation at the stated prefix, not an observed acceptance rate for a real draft model. The formula requires evaluating the ratio only for a proposal that can actually be sampled from q; our example avoids zero-probability proposal cases entirely. The rejected branch must now add exactly the missing target mass without reintroducing excess C probability. The paper's residual distribution is designed to accomplish that accounting [1].

For the replacement, first subtract the draft probability from the target probability for each token, keeping only positive results. A has 0.10 left, B has 0.05 left and C has zero. The total is 0.15. Divide each value by that total to make a fresh distribution: A gets two-thirds, B gets one-third and C gets zero. This is the residual, or remaining-mass, distribution used when a proposal is rejected. [1]

The replacement is therefore different from simply sampling again from the original target list. Why? The accepted branch has already supplied the full target allowance for C: 0.10. Giving C more mass on the rejected branch would make it too frequent again. A and B are the outcomes that still need their missing contributions. The word replacement can sound like an arbitrary second chance, but here it has a precise job. It completes the probability budget left after the accepted proposals. These are analytical probabilities; no claim is made that a small finite set of experimental draws would contain these exact fractions.

Define the residual weights as r_unnormalized(x)=max(0,p(x)-q(x)). In the example they are (0.10,0.05,0), whose sum is 0.15. Normalization gives r=(2/3,1/3,0), agreeing with the residual formula in the paper [1]. Multiplying by the rejection-branch probability yields unconditional replacement masses (0.10,0.05,0). The denominator is essential: the residual weights themselves are not a normalized distribution until divided by their sum.

This construction has a useful boundary case. If p=q, every proposal is accepted and the residual sum is zero; the rejected branch is never needed under the exact rule. An implementation should not attempt to normalize a zero residual merely because it has allocated a replacement buffer. Our worked case has a positive rejection probability, so the division is well-defined. The argument also depends on using the matching p and q for the same prefix and sampling setup. A residual assembled from stale or differently conditioned probabilities would no longer supply the demonstrated correction.

Add the two branches. A receives 0.50 from accepted proposals plus 0.15 × two-thirds from replacement, giving 0.60. B receives 0.25 plus 0.15 × one-third, giving 0.30. C receives 0.10 plus zero, giving 0.10. The final probabilities exactly match the invented target list. This is a numerical version of the paper's proof: accepted mass and correctly allocated replacement mass add to the desired distribution [1].

Now try the tempting shortcut of using the unchanged target distribution after rejection, while keeping the same acceptance rule. A would receive 0.50 + 0.15 × 0.60 = 0.59. B would receive 0.25 + 0.15 × 0.30 = 0.295. C would receive 0.10 + 0.15 × 0.10 = 0.115. Those values still sum to one, but they are the wrong distribution. A normalized result is not automatically the intended result. The shortcut specifically fails with this acceptance rule; it is not a claim that every other rejection-sampling algorithm must use the same correction formula.

The complete law is accepted_mass(x)+(1-beta)r(x). Substitution gives A:0.50+0.15×2/3=0.60; B:0.25+0.15×1/3=0.30; C:0.10+0.15×0=0.10. More generally, min(p,q)+max(0,p-q)=p. The paper's appendix establishes this accepted/rejected decomposition for arbitrary normalized distributions under the specified sampling construction [1]. Our arithmetic checks the same invariant with concrete original inputs.

A useful counterexample holds the acceptance rule fixed but substitutes unmodified p for r on rejection. The output becomes (0.59,0.295,0.115), which is normalized but not equal to p. This demonstrates why a fresh draw from the target is not a valid interchangeable repair for this particular acceptance mechanism. Other rejection samplers can use different acceptance bounds and corrections; the result is not a criticism of those separately defined algorithms. Verification should examine the complete output law, including the rejected branch, rather than checking only that proposed tokens are evaluated or that the final probabilities sum to one.

A longer draft contains a sequence of guesses, with each later guess depending on the earlier ones. Suppose the draft proposes A, then B, then C after the current prefix. If the first proposal is accepted but the second is rejected, the third was prepared after a history that is no longer the accepted history. The algorithm cannot simply skip the rejected B and keep C because C looks plausible. It keeps the accepted prefix, supplies the appropriate replacement and continues from that corrected history. [1][3]

In the cited block algorithm, if all proposed candidates are accepted, an additional token can be drawn from the target's distribution after that accepted sequence. A round with two proposals can therefore produce up to three committed tokens, while a rejection earlier in the round produces fewer. [1] These are properties of that specified method, not a promise about every implementation's token count. The same boundary also applies to temporary cached work: information computed for an abandoned candidate suffix must not silently become the state of the corrected continuation. Useful speculative work still has to belong to the history the system actually keeps.

Candidate verification preserves a contiguous accepted prefix. If proposal i is rejected, subsequent candidate states were conditioned on a path containing that rejected proposal and cannot simply be spliced into the corrected continuation. The paper's algorithm returns the accepted n candidates plus one token from the residual distribution at the first rejected position; if all gamma candidates are accepted, it uses the target distribution for the additional token [1]. Hugging Face's historical assisted-generation explanation likewise emphasizes invalidating candidates after the first mismatch [3].

Thus a round commits between one and gamma+1 tokens in this algorithm, rather than an arbitrary subset of individually acceptable-looking candidates. The conditional probabilities and associated cached states must correspond to the same retained history. Even if a discarded later token happens to equal a future output token, equality of its label does not validate the old conditional computation. This connects speculative execution to ordinary causal dependence, without implying that every implementation uses the same cache data structure or rollback API. Runtime state management is part of implementing the mathematical procedure correctly.

If a system always chooses its highest-probability next token, it is using greedy selection. In that special setting, matching the target's chosen token is a useful way to verify a draft path. Random sampling is different: several outcomes can legitimately be chosen. Two identical probability distributions can produce different individual draws. Rejecting a draft merely because an unrelated target draw picked another token can throw away useful work even when the distributions agree. Google explains this distinction in its retrospective [2].

The acceptance-and-replacement calculation handles probabilities directly. It should be applied to the effective sampling rules that the system actually intends to preserve [1]. Hugging Face's v4.49.0 documentation specifically distinguishes greedy assisted decoding from sampling with the paper's resampling method [4]. That is a versioned implementation statement, not a claim that all newer engines share its batching, tokenizer or feature limits. When reading an optimization announcement, ask which decoding path was checked. A greedy example does not by itself prove the correctness of a stochastic implementation, and an exact probability proof does not test an installed program.

A second invented comparison makes the distinction sharper. Suppose both draft and target use exactly [0.60, 0.30, 0.10]. The exact acceptance rule then accepts every proposal. But if you take two independent draws and require their letters to match, the chance of a match is only 0.60 squared plus 0.30 squared plus 0.10 squared, or 0.46. You would reject 54 percent of the pairs despite the distributions being identical. This calculation describes a different equal-distribution case, not the earlier biased draft.

Greedy decoding is a degenerate selection policy: the effective distribution concentrates on the chosen maximum, subject to the implementation's tie handling. Stochastic sampling generally has multiple possible outcomes, so matching independently drawn token values is not the same criterion as preserving the target law. Google's retrospective illustrates the inefficiency of requiring identical independent samples even when the two distributions are equal [2]. The speculative acceptance rule instead uses their probability relationship.

Sampling transformations must be included consistently in the effective p and q, as the paper specifies [1]. Hugging Face's v4.49.0 guide identifies a resampling path for do_sample=True and separately documents implementation constraints [4]. The earlier 2023 assisted-generation blog is retained here for prefix invalidation and historical implementation context, not as a replacement definition of the exact stochastic algorithm. These boundaries matter because a system's name can remain familiar while its proposal mechanism, vocabulary alignment and supported generation modes change. A correctness review should identify the actual algorithm and baseline before interpreting a result labelled lossless or equivalent.

For an additional original check, set both distributions to (0.60,0.30,0.10). The exact proposal acceptance probability becomes one. Requiring equality of independently drawn target and draft tokens instead succeeds with probability sum_x p(x) squared = 0.36+0.09+0.01=0.46. The remaining 0.54 is needless rejection under that alternative matching test, even though the distributions are identical. Independence is essential to this calculation; it is not a statement about a deliberately coupled sampler. This equal-distribution case is separate from the earlier biased q and illustrates the retrospective’s distinction using different teaching numbers [2].

The draft takes time, and the target needs resources to check its candidates. Speculative work can also be discarded. The paper's speed analysis therefore includes both acceptance behavior and the cost of running the approximation, with explicit assumptions about parallel target evaluation [1]. Keeping more proposals is useful, but acceptance percentage alone is not a measured latency result. A slower or poorly matched draft can consume time without supplying enough useful candidates.

For a separate invented timing model, assume two proposals per round, independent acceptance probability one-half at each step, a 100-millisecond target verification round and two 20-millisecond draft steps. This timing example does not reuse the earlier 85-percent acceptance case. Expected committed tokens are 1 + 0.5 + 0.25 = 1.75 per round. The round costs 140 milliseconds, so the long-run budget is 140 divided by 1.75, or 80 milliseconds per token, compared with the assumed 100-millisecond baseline. That is a calculated 1.25-times rate under those assumptions. It is not a benchmark: real verification cost, scheduling, prompt length and short-generation boundaries can change the outcome.

Change only the assumed draft cost to 50 milliseconds per step. The two-step round now costs 200 milliseconds for the same expected 1.75 tokens, or about 114.3 milliseconds per token. In this separate calculation, the speculative arrangement is slower than the assumed 100-millisecond baseline. The acceptance rule remains mathematically valid. What changed is the cost of obtaining proposals, demonstrating why correct sampling and worthwhile acceleration must be checked separately.

The paper separates reduced serial target rounds from wall-clock cost and total arithmetic work [1]. Its simplified analysis assumes sufficient resources for concurrent verification and includes the draft-to-target cost ratio. A lower serial-call count can coexist with extra operations on rejected paths. Consequently, acceptance rate, latency, throughput, memory traffic and energy use should not be collapsed into one improvement claim.

Use an explicitly separate hypothetical model with gamma=2, independent identical acceptance alpha=0.5, target round T=100 ms and draft step 20 ms. Expected committed tokens are 1+alpha+alpha squared=1.75. Assuming the target verifies the full candidate round within T, total round cost is 100+2×20=140 ms. The asymptotic cost is 80 ms per token, corresponding to a baseline-to-new rate ratio100/80=1.25. The earlier three-token distribution had beta 0.85 at one prefix and does not establish these new assumptions. Finite generation length, changing conditional acceptance and a target verification cost that grows with candidate count can invalidate the toy budget. No hardware timing or energy saving was measured.

A sensitivity check changes only the assumed draft-step cost from 20 ms to 50 ms, leaving the acceptance model and 100 ms verification round fixed. Round cost becomes 200 ms, so 200/1.75 is approximately 114.286 ms per token, above the 100 ms baseline. The baseline-to-new rate ratio is 0.875. Sampling correctness has not changed, but the assumed execution benefit has disappeared. These numbers isolate the cost tradeoff; they neither predict a particular model pair nor establish that real verification time stays constant when candidate count or batch load changes.

A useful review starts by naming the target model and its intended sampling rules. Then identify how candidates are proposed, how acceptance is calculated and how the first rejection is repaired. Keep the retained history and any cached work consistent with that repair. The cited paper provides the probability argument, while the implementation documents show why version and generation mode belong in the record. [1][3][4] A reassuring phrase such as target verified is too vague without those details.

The three-token example leaves a compact check: the accepted branch contributes [0.50, 0.25, 0.10], and the rejected branch must add [0.10, 0.05, 0]. Together they reproduce [0.60, 0.30, 0.10]. A real deployment also needs observed runtime results for its actual hardware and workload, plus ordinary evaluation of the target model's usefulness and errors. Exact sampling preserves the chosen target's behavior; it does not improve that target's factual judgment by definition. Speculation earns its place when the implementation preserves the intended rule and its measured execution benefit justifies the additional work.

A reproducible handoff identifies the target baseline, effective sampling distributions, proposal mechanism, acceptance criterion, residual calculation and prefix/state handling. It distinguishes a proof for an algorithm from evidence that a particular implementation follows that algorithm. Versioned documentation and the exact source revision belong beside any implementation claim [1][3][4]. Numerical behavior, random-number handling and workload measurements then require their own observed checks.

For this article, the verified artifact is analytical: accepted masses(0.50,0.25,0.10) plus replacement masses(0.10,0.05,0) yield the target(0.60,0.30,0.10), while the naive unchanged-target replacement produces the counterexample(0.59,0.295,0.115). The separate timing model illustrates cost accounting under explicit assumptions. Neither calculation is a trained-model benchmark, a factuality result or an installed-runtime certification. The central lesson is precise: a draft can be useful without matching the target everywhere, provided the entire accept-and-correct procedure preserves the target law and the retained causal history. Performance and answer quality remain questions requiring their own evidence.