A causal attention mask decides which earlier or current token positions a model may use when processing a particular position. The first row in a fresh batch of queries can be misleading: row zero might represent the fifth token in an existing sequence. If the mask treats it as the first token instead, it can hide valid earlier context even though every array has an acceptable size.

Caching makes that distinction useful. Hugging Face describes storing earlier keys and values and combining them with newly computed ones, so the new query block can be shorter than the available key sequence [4]. A query asks which stored information to use; keys help assign attention scores, and values supply the information being combined.

Our original seven-position example holds the scores and values fixed while changing only the allowed positions. It shows a wrong result from an incorrectly aligned mask, then follows the same reasoning through a single query, missing key positions and unused storage slots. These are exact teaching calculations, not results from running a language model.

A causal relation is defined over sequence positions, whereas an attention-score matrix is indexed by local query and key rows. Restarting a query tensor at local index zero does not reset the corresponding absolute sequence position. A shape-valid triangular mask can therefore express the wrong temporal relation for cached attention.

Hugging Face's retrieved Transformers 5.22.1 explanation describes combining past and current keys and values, with attention weights shaped by new-query length and combined key length [4]. The resulting rectangle needs an alignment convention. PyTorch's upper-left and lower-right bias utilities provide different such conventions [1][2]. Their difference is meaningful when the axes represent different spans of the sequence.

The construction below uses one sequence, inclusive causal access, hypothetical scalar values, equal finite logits and no dropout. We calculate exact masks and weighted outputs in pure Python with rational arithmetic. No tensor backend, model, generation loop or hardware benchmark is executed. This isolates position alignment from numerical precision, model quality and the separate memory cost of keeping a cache.

Assume four earlier positions, numbered zero through three, already have cached keys and values. We now process three new positions, four, five and six. Their new keys join the earlier ones, making seven key positions in total. The new query block has only three rows, corresponding to positions four, five and six in that order.

Use an inclusive rule: a query may see keys at its own position or earlier, but none from later positions. Position four may therefore use keys zero through four: five choices. Position five has six choices, and position six has seven. Including the current position means including the input currently being processed; it does not give that input access to the following position.

For this example, assign the seven values 2, 5, 11, 17, 23, 31 and 47. They are invented numbers with no claimed relationship to words or model predictions. We will use them to make different allowed sets produce visibly different outputs, while preserving the same seven stored values throughout the first comparisons.

Set past length P=4, new query length L=3 and combined key length S=P+L=7. Key positions are kpos=[0,1,2,3,4,5,6]; query positions are qpos=[4,5,6]. For local row i and column j, define Allowed(i,j) as kpos[j]≤qpos[i]. This relation is inclusive and applies within the same sequence.

The three Boolean rows, with columns in increasing key-position order, are 1111100,1111110,1111111. The corresponding allowed counts are five, six and seven. All current-block keys are present in the combined tensor, but the first new query must still exclude positions five and six. Combining the keys does not remove causal ordering inside the new block.

Assign V=[2,5,11,17,23,31,47] as one-dimensional value vectors. All finite query-key logits are stipulated to be zero, so the later output calculations isolate the membership selected by the mask. This is a mathematical attention fixture, not a claimed embedding, tokenizer sequence or measurement from a trained network. Its exact position labels remain part of every comparison.

An upper-left triangular mask starts by allowing just the first key, then the first two, then the first three. Across our three query rows and seven key columns, its allowed sets are zero; zero and one; then zero, one and two. That would be appropriate for queries at positions zero, one and two. Our queries are at four, five and six, so those sets hide valid context.

PyTorch's documented upper-left construction has this first-row alignment. Its lower-right construction instead shifts the diagonal by the number of columns minus the number of rows [1][2]. Here seven minus three is four. The allowed sets become zero through four, zero through five, and zero through six, matching our declared positions.

Neither corner name is a universal quality label. The same three-by-seven dimensions could describe queries at the beginning or the end of a longer key sequence. The correct answer comes from what the rows and columns represent. Merely checking that the mask has three rows and seven columns cannot choose the right alignment.

An upper-left triangular relation admits j≤i. For L=3,S=7 it yields 1000000,1100000,1110000. Those rows match queries at absolute positions 0,1,2 against keys 0..6, but not the suffix queries 4,5,6 in our fixture. Both interpretations have identical tensor dimensions.

The retrieved PyTorch lower-right utility constructs an offset diagonal with offset=S−L, while the upper-left utility uses the unshifted diagonal [1][2]. Here the lower-right condition j≤i+4 produces the intended rows exactly. This is a correspondence between our declared positions and those documented constructions, not a test of kernel dispatch.

The position interpretation must precede the corner choice. An upper-left mask is not inherently incorrect, and lower-right is not automatically correct for every rectangular attention operation. Rectangular shape alone omits the query origin, key origin and any irregular position mapping. A later example will keep the same lengths while changing that mapping and make the length-only rule insufficient.

Attention normally gives different allowed values different weights. Our teaching case deliberately gives every allowed key the same score. After normalizing those equal scores, the output is the ordinary average of the allowed values. This removes score preference from the comparison and lets the mask's selection do all the work.

For query position four, the intended sum is 2+5+11+17+23=58, divided by five, giving 11.6. Position five includes 31 as well, giving 89 divided by six, about 14.8333. Position six also includes 47, giving 136 divided by seven, about 19.4286. These are three different averages because each query may see a different prefix.

The upper-left mask instead produces 2, then 3.5, then 6. It has kept only the first one, two and three values. The array sizes still fit, and every result is an ordinary finite number. A successful calculation therefore would not by itself prove that the intended context was used. Checking allowed identities reveals an error that a shape check cannot.

With equal zero finite logits, excluded positions removed from normalization and dropout disabled, every allowed key receives weight 1/n_i, where n_i is the nonzero allowed count for row i. Thus output_i is the mean of the selected scalar values. This follows the masked-score, softmax and value-product formulation in the retrieved SDPA reference [3].

The intended outputs are 58/5, 89/6, 136/7, respectively 11.6, approximately 14.833333 and 19.428571. Under upper-left alignment the selected prefixes instead yield 2, 7/2, 18/3, or 2, 3.5, 6. The underlying values and finite logits have not changed. Only the permitted index sets differ.

All displayed decimals are rounded presentations of retained exact fractions. The example makes no statement about trained-model answer quality or the size of a typical alignment error. Other value assignments could accidentally give equal outputs under different masks. That is why a diagnostic should compare the allowed relation itself as well as one output vector; numerical agreement alone can conceal a membership difference.

What happens if every key is allowed for all three queries? Every output becomes 136 divided by seven. The last row is correct because position six really may see all seven keys. The first two rows are wrong: position four can now use future positions five and six, and position five can use future position six.

Checking only the final output would miss that failure in the three-query block. This matters to the design of a small example: it needs an earlier query with genuinely later keys, not just the last row where everything happens to be available. A mask can get one boundary case right while expressing the wrong rule elsewhere.

Now process only the query at position six against the same seven keys. Its intended mask permits all seven, while an upper-left one-row mask permits only key zero. The exact outputs are about 19.4286 and 2. In this particular single-query case, allowing every valid key is causally acceptable because none lies in the query's future. That conclusion depends on the positions, not merely on there being one row.

An all-allowed 3×7 mask returns 136/7 on every row. It agrees with the intended result at qpos=6 but violates the causal relation at qpos=4 and 5. The former gains future keys 5,6; the latter gains future key 6. A last-row-only comparison is therefore insufficient for the original multi-query fixture.

For the separate single-query case L=1, qpos=(6), S=7, the intended relation is 1111111. Upper-left instead gives 1000000, producing 2 rather than 136/7. Lower-right has offset 6 and matches the intended relation. This is the same positional accounting with a different query slice, not an assumption that decoding always uses a particular library convention.

When every valid key belongs to the same sequence and has position at most the sole query's position, an all-valid-keys relation meets the causal requirement. That conditional observation is not a general instruction to remove masks: unused slots, other sequences or keys from later positions would require their own exclusions. The key-position evidence, rather than query count alone, establishes the condition.

Suppose the retained keys occupy consecutive positions, beginning at some position we call the key origin. The queries also occupy consecutive positions, beginning at the query origin. A local row number tells us how far to move from the query origin, while a column number tells us how far to move from the key origin.

Our causal question is then simple: has this key's actual position been reached by this query? The difference between the two origins tells us how far to shift the triangle. In the seven-key example, the key origin is zero and the query origin is four, so the shift is four. Counting only rows would throw away that information.

Removing some earliest keys does not automatically break lower-right alignment. Retain consecutive keys at positions three, four, five and six, while keeping queries at four, five and six. Both spans still end at six. Four key columns minus three query rows gives a shift of one, correctly allowing two, three and four keys. The relevant condition is the alignment of the represented spans, not whether any old key was removed.

For contiguous ranges, write kpos[j]=k₀+j and qpos[i]=q₀+i. Then kpos[j]≤qpos[i] becomes j≤i+(q₀−k₀). The required diagonal offset is therefore the origin difference q₀−k₀. This is an original algebraic derivation from the declared inclusive relation.

If the final represented positions align, q₀+L−1=k₀+S−1, then q₀−k₀=S−L. The lower-right length formula follows under that condition. PyTorch's tutorial explains the last-query/last-key alignment behind the lower-right convention [5]. Equal lengths or a rectangular shape alone do not establish the corresponding origins.

As a separate contiguous-retention case, use keys [3,4,5,6] and queries [4,5,6]. Here q₀−k₀=1=S−L, giving 1100, 1110, 1111. The length-derived relation remains correct despite removing earlier positions 0,1,2. Thus the presence of eviction is not by itself a counterexample; what matters is whether the retained mapping satisfies the positional assumptions used in the derivation.

Consider a different hypothetical selection of keys: positions zero, two, four and six. Keep the same queries at four, five and six. This is a deliberately constructed gathered set, not a claim that a named cache normally discards these particular positions. The column numbers zero through three now stand for four nonconsecutive sequence positions.

The query at four may use keys zero, two and four, so it has three choices. The query at five has those same three choices because there is no retained key at five. The query at six may use all four. The intended counts are therefore three, three and four.

A length-only lower-right triangle uses four minus three as its shift and instead allows two, three and four keys. Its first row wrongly hides position four. Using the corresponding values 2,11,23,47, the intended first average is 12, while the mistaken first average is 6.5. The remaining two outputs agree. Again, agreement in some rows does not validate the rule for every row.

Let the hypothetical gathered key positions be [0,2,4,6], with corresponding values [2,11,23,47], and retain qpos=[4,5,6]. Applying the position predicate directly gives 1110, 1110, 1111. The key axis no longer has a single contiguous origin-plus-index representation.

Using S−L=1 without that mapping information gives 1100, 1110, 1111. The first row excludes the eligible key at position 4. Intended outputs are 36/3, 36/3, 83/4, or 12,12,20.75. The length-only outputs are 13/2,12,83/4, or 6.5,12,20.75. Only one row differs, which is enough to invalidate the relation for this fixture.

This is a mathematical counterexample for an irregularly selected key sequence. It does not assert an implementation policy for Hugging Face, PyTorch or a particular compressed cache. A custom gathered representation must retain whatever position information its masking rule requires. Counts identify axis lengths; they cannot reconstruct which absolute positions were retained or determine eligibility when those positions contain gaps.

Storage capacity introduces another length. Imagine nine allocated key slots, but only seven contain real keys at positions zero through six. The final two are unused. Our three queries still represent positions four, five and six. A separate validity check can mark the two unused slots unavailable.

If we calculate the triangular shift from all nine slots, nine minus three gives six. The first query row then allows real keys zero through six. Removing the two unused slots does nothing to fix that error, because future positions five and six are real, valid keys. The same problem affects query position five.

The correct rows still allow five, six and seven real keys, with both unused slots excluded throughout. Hugging Face documents fixed-length cache storage as one supported approach [4]; this invented example does not claim that its implementation makes our mistake. It shows why a validity rule and a causal-position rule answer different questions. Marking storage as occupied is not the same as establishing that its content is in a particular query's past.

Use an allocated key axis of length 9 with valid positions [0,1,2,3,4,5,6] followed by two invalid slots. For qpos=[4,5,6], the intended rows are 111110000,111111000,111111100. The relation combines slot validity with the original absolute-position comparison.

A naive allocated-length offset 9−3=6, followed by intersection with the valid-slot mask, instead produces 111111100 on all three rows. The first two queries gain future real keys. Their outputs become 136/7 rather than 58/5 and 89/6. Validity masking removes unused storage but cannot repair the incorrect causal origin assigned to occupied slots.

The retrieved cache explanation distinguishes dynamically growing and fixed-length storage [4]. Our nine-slot construction is an independent counterexample, not an audit of those implementations. A correct implementation can track actual positions and valid lengths explicitly. The distinction is between allocation length, active key count and represented position span; substituting one for another requires evidence about the layout and the intended relation.

There is another reason not to copy a mask shape from one example into a different call. A model's generation interface may accept a row of flags covering its input and cached positions. A lower-level attention operation must ultimately decide separately for each query which keys it may use. Those interfaces are describing related information at different stages.

Hugging Face's cited cache guide describes an attention mask covering the combined past and current length for each batch item [4]. PyTorch's scaled-dot-product attention reference describes a mask that can expand to the dimensions of its attention weights, including query and key axes [3]. These statements do not mean that either interface accepts every shape accepted by the other.

In our construction, a seven-entry validity row cannot by itself express why query four excludes two real keys while query six excludes none. A causal rule supplies the query-dependent part. Record the meaning of the flags and the stage that adds this rule, instead of assuming that a successfully expanded array already contains the full intended temporal relationship.

The retrieved Transformers 5.22.1 guide describes a higher-level attention_mask with batch and combined past-plus-current key dimensions [4]. The retrieved PyTorch 2.15 SDPA reference instead requires its attention mask to be broadcastable to attention weights shaped by batch, heads, query length L and key length S [3]. The abstraction boundary belongs to the API contract.

In our single-sequence example, key validity is a predicate on columns, while causal eligibility depends jointly on row and column positions. Broadcasting a column-validity vector across query rows repeats the same validity information; it does not derive the missing position comparison. Conversely, a query-dependent relation cannot generally be compressed into one identical validity row without another stage reconstructing its row dependence.

Keep separate records for what the caller supplies, what the model wrapper constructs and what the attention primitive consumes. This is a representation distinction rather than a claim that one specific internal expansion was observed. A mask accepted at one API level is not evidence of identical semantics at another, even when its final dimensions appear compatible.

The PyTorch 2.15 reference describes its is_causal option as upper-left aligned for a nonsquare score matrix [3]. That matters in our suffix-query example: the word causal does not tell the call where the new queries begin in the old sequence. The lower-right utility expresses a different alignment, and PyTorch's tutorial shows these bias objects being supplied to the attention operation [1][5].

The same reference says an explicit attention mask cannot be combined with is_causal set to true [3]. This article reports that documented interface boundary; it does not run a call to test a particular installation. A proposed implementation needs the documentation for its actual version and the chosen backend's supported inputs.

The important decision comes before the API spelling. Write the allowed position pairs, then identify which documented option represents them. For contiguous suffix queries ending with the key sequence, a lower-right relation fits our example. For the gathered or partly filled layouts, preserve their extra position and validity information. A familiar option name cannot supply information that its inputs never received.

The retrieved PyTorch 2.15 SDPA contract specifies upper-left alignment for nonsquare is_causal=True and disallows simultaneous explicit attn_mask with that flag [3]. Its Boolean attn_mask uses True for participation. The bias-utility references and tutorial distinguish upper-left from lower-right alignment [1][2][5]. These are version-scoped documented semantics, not a universal convention across attention libraries.

Our comparison maps mathematical relations to those descriptions without executing source examples or asserting observed dispatch behavior. For an explicit mask or supported bias object, follow the applicable call contract rather than adding a causal flag by analogy. A separate API may use another convention; this article establishes no equivalence merely because the option has a similar name.

The position predicate remains the reference when adapting a representation. The valid nine-slot fixture and the gathered four-key fixture require more information than allocated dimensions alone. Correct alignment and correct Boolean polarity are separate obligations. The earlier attention-mask article addresses polarity; here the new issue is whether the participating entries represent the intended sequence positions at all.

A small check should contain a rectangular case, an earlier query with a future key, and different values at positions that one candidate mask hides or exposes. Our three-by-seven example supplies all three. A square seven-by-seven case is also useful, but upper-left and lower-right triangles are identical there, so that case alone cannot distinguish their alignment.

We calculated fifty-four suffix layouts using zero through eight past positions and one through six new positions. In every constructed layout, the position-based mask matched the lower-right relation and the corresponding rows taken from a full-sequence causal mask. Upper-left matched those suffix cases only when there were no past positions. This is a finite mathematical check, not a performance result.

There is also an empty-row boundary. If a query precedes every retained key, it has no allowed value to average in this simple reference. We leave that average undefined instead of inventing a zero. A real implementation's handling needs its own documented behavior; the teaching calculation does not decide how a particular kernel responds.

The retained calculation enumerates P=0..8 and L=1..6, with S=P+L: 54 suffix configurations. For each, direct comparison of absolute positions matches both the lower-right construction and rows P..P+L−1 of the full S×S causal relation. Upper-left matches precisely the P=0 cases within this enumeration. Another 48 origin/shape cases verify j≤i+(q₀−k₀).

These checks validate the constructed relations, not a production cache or backend. The square case S=L has identical corner offsets and cannot discriminate the two conventions. We also verify the exact Fraction outputs, gathered-key discrepancy and allocated-slot counterexample. Checking masks before outputs avoids hiding a membership error through coincidental value equality or output rounding.

If no key is allowed, the equal-logit reference has a zero normalization count and no defined mean. The saved calculation records that separately, without claiming a kernel-specific NaN, zero or exception policy. SDPA's documented dropout and backend numerical differences are additional execution concerns [3]; all hypothetical outputs here deliberately use no dropout and exact arithmetic.

For a cache-related attention result, keep the query positions, retained key positions and slot-validity information together. State whether the current block's keys are included, and whether keys belong to one sequence. Then record the intended past-and-current rule and the API option used to represent it. These details explain why the mask is correct; an array size alone cannot.

The seven-key construction gives a clear comparison. Intended suffix attention averages five, six and seven values. An upper-left triangle averages only one, two and three. An all-allowed mask gets the final row right while leaking later positions into the earlier rows. The gathered and allocated-storage cases show why even the familiar lower-right length formula has assumptions.

The practical outcome is a precise question for any new cache layout: which sequence position does this local index represent? Answering it lets you derive the allowed pairs, identify the necessary validity checks and compare the complete relation with a reference. It establishes the mask's position accounting without pretending to measure model intelligence, answer quality or hardware speed.

Retain qpos, kpos, active-slot validity, sequence membership, current-key inclusion and the intended inclusive causal predicate. For contiguous ranges, the origin difference yields the diagonal offset; aligned ends justify replacing that difference with S−L. For irregular or partially allocated layouts, preserve the actual mapping instead of treating storage dimensions as sequence coordinates.

Our primary result is an exact distinction between three-by-seven relations: intended suffix rows 1111100,1111110,1111111 versus upper-left 1000000,1100000,1110000. The resulting means differ while the scores, values and dimensions stay fixed. Single-query, retained-contiguous, gathered and nine-slot examples each change one specified boundary rather than adding an unmeasured model claim.

A completed implementation check would also need its actual version, call inputs, returned values and numerical conditions. This article supplies the mathematical reference and documented alignment distinctions, not that execution evidence. Its result is narrower and useful: causal permission follows represented sequence positions, and any shortcut based on lengths must preserve the assumptions that make those lengths an adequate description.