You save a passage, turn it into tokens, then decode those tokens back into text. The result reads correctly. Has the process preserved the exact passage? That depends on which version of the text reached the token encoder and what the decoder promises to return.

A tokenizer can be a pipeline rather than one indivisible step. Hugging Face’s Tokenizers guide separates normalization, preliminary splitting, the vocabulary model and later processing; decoding has its own behavior too [1]. Some steps can change the input string before the vocabulary model chooses tokens. Reversing the last step cannot automatically reverse every earlier change.

We will build a deliberately small byte-token system whose entire rule is visible, then place Unicode normalization in front of it. Four short strings reveal what survives and what becomes indistinguishable. The byte and normalization examples were checked with an original Python script. They are teaching examples, not token counts from a commercial model. The practical question is whether an application needs readable normalized text, equivalent text or the exact original representation.

Define the round-trip contract before evaluating a tokenizer. Equality of the decoded display, equality of Unicode scalar sequences, equality after a chosen normalization and equality of original encoded bytes are different tests. A successful check against one does not automatically establish the others.

The Tokenizers 0.20.3 documentation presents distinct normalization, pre-tokenization, vocabulary-model and post-processing stages, followed by configurable decoding [1][2]. A representation may be reversible relative to the string presented to one stage while the complete pipeline is not reversible relative to the earlier raw input.

The original construction below specifies a complete 257-entry vocabulary and a deterministic byte-consumption rule. It is not a trained BPE vocabulary or an execution of Hugging Face’s tokenizer. We use valid Unicode scalar strings, strict UTF-8 conversion and explicitly named NFC or NFKC operations. The examples were computed with Python 3.12.14 and its Unicode 15.0.0 database; the historically assigned characters used here also fall within the normalization-stability scope described by Unicode [3].

Start with an accented e. One string contains the single code point U+00E9, written é. Another contains U+0065, the ordinary letter e, followed by U+0301, a combining acute accent. These can represent the same accented letter while using different code-point sequences. Naming the code points avoids relying on how a particular font displays them [3].

UTF-8 gives those strings different byte sequences. The single-code-point version uses decimal bytes (195, 169), while the two-code-point version uses (101, 204, 129). UTF-8 represents different code points with different numbers of bytes, so a byte count and a code-point count need not match [4].

Our second pair is the fi ligature, U+FB01, written fi, and the two ordinary letters fi, U+0066 followed by U+0069. The ligature uses bytes (239, 172, 129); the two letters use (102, 105). We now have four precisely identified inputs. Their visual similarity is useful context, but the code-point and byte lists tell us exactly what any later operation receives.

Let A be U+00E9 and B be the sequence U+0065 U+0301. Their UTF-8 encodings are respectively hex C3 A9 and 65 CC 81, or decimal (195,169) and (101,204,129). A contains one code point and two bytes; B contains two code points and three bytes. The sequences are canonically equivalent, while their raw encodings differ [3][4].

Let C be U+FB01 and D be U+0066 U+0069. Their encodings are EF AC 81 and 66 69, or decimal (239,172,129) and (102,105). C has one code point and three bytes; D has two code points and two bytes. This pair illustrates compatibility equivalence, which Unicode distinguishes from canonical equivalence [3].

These inventories concern code points and UTF-8 octets, not a font-independent count of visible glyphs. The byte encodings were computed and retained for these explicit strings. The UTF-8 reference supplies the variable-length encoding rules [4]; it does not supply a language-model vocabulary or determine how many model tokens each sequence must receive.

Our invented vocabulary gives every possible byte its own token. Identifiers 0 through 255 stand for the byte with that same numeric value. Add one extra token: identifier 256 stands for the two-byte sequence (102, 105), which spells fi. That makes 257 vocabulary entries.

To encode a string, first obtain its UTF-8 bytes. Read from left to right. Whenever the next two bytes are (102, 105), emit token 256 and move forward by two bytes. Otherwise, emit the current byte’s identifier and move forward by one. There are no other merges, omitted spaces, added prefixes or special markers in this example.

The two ordinary letters fi therefore become (256). The ligature fi becomes (239, 172, 129), since its bytes do not match our merge. The two accented-e forms become (195, 169) and (101, 204, 129). These counts belong to our stated rule. The useful feature is that every output token has a known byte payload, so we can examine preservation without guessing how a real vocabulary was trained.

Define vocabulary V by V(i)=the one-byte payload i for every integer 0≤i≤255, and V(256)=bytes 102,105. The inventory is complete: 256 byte entries plus one two-byte entry. Token identifiers are labels in this construction, not borrowed from a model.

Define encoder E over the UTF-8 byte string. At each scan position, emit 256 and consume two bytes if the next pair is 102,105; otherwise emit the current byte value and consume one byte. This is an explicitly chosen left-to-right substitution rule. It uses no learned frequency ranking, preliminary split, unknown token, padding, truncation or special-token post-processing.

Consequently E(A)=(195,169), E(B)=(101,204,129), E(C)=(239,172,129) and E(D)=(256), with sequence lengths 2,3,3,1. Hugging Face distinguishes the vocabulary model and byte-level components in a real pipeline [1][2]. Our construction isolates a simpler property: segmentation can combine adjacent bytes without deleting any of the bytes they represent.

Our decoder looks up each token’s byte payload, joins the payloads in order and interprets the complete result as UTF-8. Token 256 expands to (102, 105), so decoding (256) returns fi. Decoding (239, 172, 129) returns the original ligature fi. The distinction between the two source strings survives this encoder and decoder.

The reason is visible at every step. A one-byte token stores exactly the byte the encoder consumed. The extra token stores exactly the two bytes it consumed. Nothing is deleted, reordered or replaced by a guess. Joining those payloads recovers the input byte sequence, whether or not a merge occurred.

A token containing only part of a multi-byte character need not be readable by itself. Our decoder joins all payloads before decoding the complete UTF-8 string. Readability of each separate token is therefore not the preservation test. The right check for this stage is whether the complete decoded bytes equal the complete bytes supplied to it. For our four examples, they do.

Let decoder Dec concatenate V(t) for all token identifiers t in order, then strictly decode the resulting UTF-8 bytes. Each encoder step emits a payload equal to the next consumed one- or two-byte segment. By induction on the scan position, concatenating emitted payloads reproduces the consumed input prefix. At termination it reproduces the entire input byte string.

Thus Dec(E(s))=s for every valid scalar string s under this construction. In particular, the merge fi→256 is lossless because V(256) stores both original bytes. The ligature’s three-byte representation remains distinct. This proof depends on the stated encoder, payload table and decoder, rather than on an empirical sample of language-model outputs.

Some individual byte tokens can be incomplete UTF-8 character sequences. Decoding each such payload separately would be a different procedure. The defined Dec reconstructs the byte sequence first, so the original valid encoding supplies complete character boundaries. Byte-level token displays and whole-sequence textual reconstruction should therefore be evaluated as separate interfaces.

Now insert an NFC normalization step before our byte encoder. Unicode defines this form using canonical decomposition and composition [3]. In our accented-e pair, both the single U+00E9 and the sequence e plus combining acute become the same normalized string, U+00E9.

Both inputs now reach the encoder as bytes (195, 169) and produce tokens (195, 169). Decoding returns é in each case. The accent has not been removed. What disappeared was the distinction between writing the accented letter as one code point and writing it as two. Plain unaccented e remains a different string in this example.

The byte encoder is still doing its lossless job on the bytes it receives. It never sees the earlier two-code-point representation after NFC has replaced it. If the application’s goal is comparison under canonical equivalence, this shared form may be useful. If the goal is to reproduce the exact earlier bytes, the normalized tokens alone no longer supply enough information. The preservation question has moved to an earlier stage.

Let N_C denote NFC. For the declared pair, N_C(A)=N_C(B)=U+00E9, even though A≠B as scalar sequences and UTF-8 bytes. Both complete pipelines therefore emit (195,169), and decoding returns the shared normalized form.

The construction now satisfies Dec(E(N_C(s)))=N_C(s). That identity establishes recovery of normalized text, not necessarily raw text. Since the two distinct inputs produce the same complete token sequence, a decoder receiving only that sequence cannot identify which raw representation preceded normalization. This is a direct collision in the chosen preprocessing map, not a failure of the byte-payload decoder.

Unicode’s canonical equivalence distinguishes representation from abstract character identity [3]. The example retains the acute accent: neither NFC result is the unaccented U+0065. Avoid describing this operation as accent stripping or as byte-preserving cleanup. Its usefulness depends on the application’s equality criterion, while exact historical reconstruction requires the original representation or enough additional information to distinguish it.

NFC leaves our fi ligature distinct from the ordinary letters fi. NFKC makes a different choice: the ligature becomes the two letters. Unicode documents this compatibility decomposition and distinguishes it from the canonical normalization used in the previous example [3].

After NFKC, both forms reach our encoder as (102, 105), which becomes the single token (256). Decoding returns fi. That result faithfully represents the normalized input, yet it cannot tell us whether the source used the ligature. Both accented-e inputs also become the same accented-e form under NFKC in our example.

The broader change is not automatically appropriate for every document. A search application may deliberately want certain alternative forms to match. A quotation archive may also need to retain the exact source representation. Unicode cautions that compatibility normalization can erase distinctions that matter in some contexts [3]. Preserve the original separately when those distinctions are part of the record, and describe the normalized copy by the operation that produced it.

Let N_K denote NFKC. The compatibility pair obeys N_K(C)=N_K(D)=U+0066 U+0069, whereas NFC preserves C as U+FB01. Thus E(N_K(C))=E(N_K(D))=(256). The raw-token lengths 3 and 1 become the same normalized-token length, 1, in this particular construction.

That count change follows from both the named Unicode mapping and the deliberately chosen fi token. It is not a universal compression result for NFKC or a prediction for another vocabulary. The two accented-e strings similarly converge under NFKC, but their normalized token sequence remains (195,169), length 2.

Unicode describes compatibility equivalence as weaker than canonical equivalence and warns that compatibility normalization can remove contextually significant distinctions [3]. A system may choose that folding for a defined matching task while retaining the raw source for another task. Neither NFC nor NFKC should be silently equated with lowercasing: the operations discussed here concern the stated normalization forms, and any case conversion needs its own specification.

Extend the ligature example into the word fish, with a ligature followed by s and h. Our raw encoder produces (239, 172, 129, 115, 104). The ordinary-letter version fish produces (256, 115, 104). The sequences differ because the first encoder receives the ligature’s three bytes, while the second can use our fi merge.

Apply NFKC first and both words become ordinary fish. Both now produce (256, 115, 104), and both decode to fish. The system can represent every byte it receives, and it has no unknown-token problem. It still cannot recover the earlier distinction from those token identifiers alone.

This separates two useful questions. Coverage asks whether the encoder can represent its input. Preservation asks whether the complete process retains the distinctions that matter about an earlier source. Our vocabulary has complete byte coverage by construction. The loss of the ligature distinction happened before byte encoding, so adding more byte coverage would not repair it. Keeping the original source would.

For raw S₁=U+FB01 U+0073 U+0068, the declared encoder returns (239,172,129,115,104). For S₂=U+0066 U+0069 U+0073 U+0068, it returns (256,115,104). The raw sequences are distinct and both decode exactly under Dec.

After NFKC, N_K(S₁)=N_K(S₂)=fish, giving the identical complete token sequence (256,115,104). The normalized word has three tokens under our rule. This is an original finite construction with a complete input definition, so no file name, source identifier or hidden metadata is available to distinguish the two histories.

Complete byte coverage and raw-input recoverability are therefore independent properties here. Every byte presented to E has a representation, yet normalization has already identified two raw strings. An absence of unknown tokens says nothing about that earlier collision. The conclusion is about this pipeline boundary, not a claim that byte-level language-model tokenizers universally normalize ligatures or discard source distinctions. Their actual normalization and decoding configuration must be inspected [1][2].

Hugging Face’s components guide describes tracking alignment during normalization so generated tokens can be related back to the input [2]. That is useful for highlighting a source passage, but a location and the text stored at that location are different pieces of information.

In our fish example, the normalized fi token corresponds to original code-point positions from 0 up to, but not including, 1. In ordinary fish, it corresponds to positions from 0 up to 2. We have declared those alignments for this example; they are not outputs measured from the library. If the original string is retained, the span identifies the corresponding source substring.

Offsets alone are not a universal substitute for that source. Consider a separate rule that changes ASCII A into a. Original A and original a both become a, both produce token (97), and both occupy the original span [0, 1). Their byte spans are the same too. Neither the normalized token nor those offsets says which letter appeared originally. Alignment metadata needs a clearly identified source and a stated unit.

The Tokenizers components documentation describes preserving normalization alignment to map generated tokens back to input text [2]. Keep this association separate from inversion of a transformed string. An offset names a location in a particular coordinate system; it does not generally contain the characters at that location.

For the explicit NFKC word example, assign the fi token the half-open original code-point span [0,1) in S₁=fish and [0,2) in S₂=fish. The remaining source spans are [1,2),[2,3) in S₁ and [2,3),[3,4) in S₂. These are original declared alignments, not an executed offset API or byte offsets.

A separate ASCII A-to-a preprocessing rule supplies a counterexample to reconstruction from offsets alone. Inputs A and a both yield token (97) with original code-point and byte span [0,1). Their raw values still differ. Retaining an original-source reference makes the alignment useful for extraction; retaining only the common token and span cannot recover both distinct inputs. NFC/NFKC are not asserted to perform this case conversion.

Suppose an application receives e in one piece and the combining acute accent in the next. Each piece is already unchanged by NFC when considered alone. Normalizing them separately and joining the results gives the two-code-point sequence e plus acute.

Normalize the complete joined string instead and NFC produces the single code point é. Unicode explicitly explains that concatenating normalized strings does not always leave the result normalized [3]. The boundary between pieces can create a combination that was not present inside either piece on its own.

Our byte encoder makes the difference concrete. The separately normalized and then joined form yields (101, 204, 129). The form normalized after joining yields (195, 169). Applying NFC once more to the joined two-code-point string brings it to the latter form. The lesson is to record where normalization happens in a pipeline that assembles pieces. The word normalized is incomplete unless it identifies the form and the text boundary to which that operation was applied.

Take X=U+0065 and Y=U+0301. For these singleton strings, N_C(X)=X and N_C(Y)=Y. Nevertheless, N_C(X)+N_C(Y)=U+0065 U+0301 differs from N_C(X+Y)=U+00E9. This directly instantiates the Unicode annex’s statement that normalization forms are not generally closed under concatenation [3].

Our encoder maps the former result to (101,204,129) and the latter to (195,169). Reapplying NFC to the concatenated normalized pieces produces U+00E9, so the discrepancy is about the boundary at which normalization was performed. It does not mean that the two forms have different abstract accented-letter identities.

A streaming or document-assembly procedure therefore needs a defined normalization policy across joins. This article does not implement a streaming normalizer or prescribe an arbitrary fixed buffer size. Its finite example establishes that independent per-piece normalization and whole-string normalization are different operations. The source discusses boundary-aware normalization; an actual implementation must supply the relevant rules rather than infer them from the token count.

There is another boundary effect even when no normalization occurs. Encode f alone with our rule and the result is (102). Encode i alone and the result is (105). Joining those two token lists produces (102, 105). If we instead join the text first and encode fi, our merge produces (256).

Both token lists decode to the same two bytes and the same text, fi. One list contains two tokens and the other contains one. The difference comes from whether the encoder was allowed to see the byte pair together. It does not indicate that either list lost the text it represents.

This is a separate result from the earlier normalization example. There, different original strings became the same normalized tokens. Here, the same decoded text has two possible token sequences under different assembly procedures. For a system that stores tokens, records where the text was divided can therefore matter. Exact decoded-text equality does not, by itself, establish that an independently assembled token sequence matches the encoder’s output for the complete text.

Without normalization, E(f) concatenated with E(i) is (102,105), whereas E(fi)=(256). The whole-input encoder can apply its two-byte rule across the join; independent encodings cannot. Yet Dec((102,105))=Dec((256))=fi because both payload concatenations equal bytes 102,105.

Our decoder is therefore not one-to-one on its valid decoding domain, even though Dec(E(s))=s for every valid input string. There is no contradiction: the encoder chooses a particular segmentation, while another legal sequence can carry the same byte payload. Re-encoding the decoded result gives (256), not necessarily the supplied sequence (102,105).

This distinguishes text round-trip fidelity from token-sequence identity. It also prevents an unwarranted additive-count rule for separately encoded pieces. The result is proved only for the declared merge; another tokenizer’s preliminary splits, vocabulary and boundary treatment determine its behavior [1][2]. For a concrete application, compare the actual complete input encoding and the assembled sequence under the relevant tokenizer configuration.

Our decoder is intentionally strict: it expands the known payloads in order and joins bytes before interpreting the string. Real tokenizer pipelines can have additional decoding choices. The Hugging Face guide discusses converting identifiers back to vocabulary pieces, handling special tokens and choosing a decoder that understands the tokenizer’s piece markers [1][2]. Those details belong in a round-trip check.

Choose the comparison that matches the task. For a raw-text archive, compare the reconstructed source representation with the saved original. For canonical matching, compare the chosen normalized forms. For our token-assembly example, compare the token identifiers as well if identical sequences are required. A readable preview alone does not perform all these checks.

The criterion should also name the boundary. Comparing decoded output with the string after normalization can correctly pass while comparison with the earlier raw string fails. Both results can be accurate. Reporting which inputs were compared makes the result useful and prevents a successful check of one stage from being presented as preservation by the entire pipeline.

The proof for Dec relies on exact payload expansion and ordered byte concatenation. A real decoder may have additional behavior associated with special tokens, subword markers or formatting; the pinned Hugging Face guide demonstrates why the decoder must match the tokenizer components [1][2]. Its example behavior is not a universal promise that every decoder inserts spaces or removes every special token.

Define the comparison explicitly. Raw UTF-8 equality asks whether bytes are identical. Scalar-sequence equality compares the decoded code points. Equality after NFC applies a chosen equivalence relation before comparison. Token-sequence equality compares identifiers and order under the same vocabulary. These tests answer different questions even when their displayed strings look similar.

Also identify which pipeline output serves as the reference. A check against N(s) can establish Dec(E(N(s)))=N(s) without establishing equality to s. Preserve that distinction in a result or regression record. The original examples specify no cleanup, truncation or padding, so their success cannot stand in for checking those behaviors in another implementation.

A useful record keeps the original string, its encoding, the normalization choice, the string actually submitted for tokenization, the tokenizer’s identity and the decoding settings. If text was processed in pieces, record those boundaries too. These are the details needed to explain why a decoded passage does or does not reproduce its source.

For our four inputs, the results are precise. Raw encoding preserves all four strings. NFC makes the two accented-e representations identical while keeping the ligature distinct from fi. NFKC also makes the ligature and ordinary letters identical. Each token count follows from the complete 257-entry rule we declared, rather than from an estimate based on appearance or word length.

An application can retain a normalized working copy and the original source for different purposes. That gives readers and future processing steps a way to inspect the actual quotation when exact representation matters. The decision is about which distinctions the task needs. A readable decoding is useful evidence, but its meaning comes from the source, transformations and equality test that accompany it.

Retain the raw scalar string or original bytes with their encoding, normalization form and boundary policy, tokenizer definition or immutable identity, submitted string, emitted IDs and decoder configuration. Where offsets are used, specify their unit and which retained source they reference. This is a proposed reproducibility record derived from the examples, not a claim that every library stores all those fields automatically.

The fixture’s exact counts are raw 2,3,3,1; after NFC 2,2,3,1; and after NFKC 2,2,1,1 for A,B,C,D respectively. The checked mappings use characters assigned in the runtime’s Unicode 15.0.0 tables. The cited revision of the Unicode annex supplies normalization definitions and the stability rule; these finite checks do not certify a complete Unicode 18 implementation [3].

The central result is compositional: reversibility of one stage does not imply reversibility of every preceding transformation. Keep the raw source when its distinctions are required, and evaluate normalized text under its declared contract. No token count in this teaching vocabulary establishes a real model’s context cost, quality or speed.