Imagine a library where every book is one enormous card. To answer a small question, a librarian would have to read the whole book, even if the answer is one paragraph. Retrieval systems work similarly: they search smaller passages, called chunks, so a question can find a relevant piece without carrying an entire document into the answer step [1][2].

Splitting also helps a system stay inside model limits. Embedding and chat models accept only a limited number of tokens, and text that is too long can lose information through truncation [1]. A chunk is therefore a practical unit: small enough to embed and pass to a reader model, but large enough to contain meaning. The same passage may need to satisfy both the embedding model and the reader, so its size cannot be chosen for one model alone [1].

A finer grain can also improve representation. A long page covering many sub-topics may fit a model limit as a single unit, yet its overall meaning is mixed [1]. Dividing it lets each sub-topic be represented separately. That does not guarantee a better answer. It changes what the search can match, what context is preserved, and what irrelevant material may be retrieved alongside the useful passage [1][3].

The main methods choose boundaries from different signals. Fixed-size chunking uses a length budget; structure-based chunking uses headings or markup that group related material [1][2]. The comparison below shows why the choice matters before any search takes place. A length rule gives predictable units but may divide an explanation. A heading can keep a topic together, but a long section still needs a size check. Both approaches need inspection of the chunks they actually produce.

Chunking partitions a source document into retrievable units whose embeddings are indexed and later matched against a query embedding [1][3]. The partition is required when source length exceeds embedding or chat-completion input limits, where truncation would otherwise lose content [1]. It is also a representation decision: content may be poorly represented as a single vector, for example a page covering numerous varied sub-topics, and chunking at a finer grain might give better results [1].

The chunk is the granularity at which similarity is computed. Once chunks are embedded and stored, a query is embedded with the same model and nearest neighbours are returned for the reader [3]. Consequently, chunking determines the candidate set before any reranking or generation occurs. A passage that was never formed, or whose embedding does not reflect the query, cannot be recovered by a downstream reader that only sees retrieved context [3].

This creates a coupled design problem across indexing and query time. Chunk size must respect the embedding model's maximum input, the reader model's context budget, and the semantic coherence of the source [1]. In the tutorial example, the retrieved context consists of five documents of 512 tokens each, and the authors aim for a reader window of at least 4k tokens to accommodate the complete prompt [3]. The five passages account for 2,560 tokens before the question and other prompt text; 4k is the tutorial's chosen window target, not the sum of the passage lengths. Chunking is therefore not a preprocessing convenience; it is a constraint on both representation and generation.

Boundary selection and size validation are separate decisions. Fixed-size methods use an explicit length measure, while structure-based methods use markup such as section headings or logical code blocks [1][2]. The comparison below separates their boundary rules from their limitations. Authorial structure can group related content, but it does not establish token fit; a repeatable size rule can control length, but it does not establish semantic completeness. Evaluate those properties independently before comparing retrieval results.

Text already has natural layers: paragraphs contain sentences, and sentences contain words. Recursive splitting uses that hierarchy. It tries to keep a paragraph together, and only breaks it into sentences if it is too large, then into words if necessary [2][3]. This is like unpacking boxes: open the large box first, and only open smaller boxes when the contents will not fit.

Sentence-based splitting can preserve complete thoughts better than cutting at an arbitrary point [1][2]. A chunk that ends at a sentence boundary is less likely to leave a half-finished idea. Recursive methods still need a maximum size, so a very long paragraph will be divided, but the division follows the structure of the writing instead of ignoring it [2][3].

The trade-off is uneven chunk sizes. Some chunks may be short because a paragraph or section was short, while others approach the limit [3]. That variation is not a defect by itself. It reflects the source. The risk is assuming that a natural boundary always matches the question being asked. A paragraph can be coherent as writing yet still contain several different facts that a search might need to separate [1][2].

Recursive character splitting implements a hierarchical fallback: attempt to retain larger units such as paragraphs, descend to sentences when a unit exceeds the size budget, and continue to words if required [2]. Separators are ordered from most to least significant, for example double line breaks before single line breaks and sentence punctuation, so global structure is preserved at the expense of chunk-size variation [3]. LangChain presents this as a default starting strategy because it balances context retention against size control [2].

Sentence-aware splitting, including splitting by sentences in built-in skills, can maintain semantic coherence within a chunk [1]. However, coherence at the sentence level does not imply topical homogeneity at the chunk level. A paragraph may contain multiple claims, and a chunk formed from several sentences may still mix topics. The method reduces syntactic fragmentation; it does not solve retrieval granularity, which depends on how query-relevant information is distributed across the resulting units [1][3].

Many documents announce their own organisation. Headings divide a report into sections, Markdown marks levels of importance, and code is grouped into functions or classes [1][2]. Structure-based splitting respects those signals. Instead of cutting every 500 characters, it keeps a section together or starts a new chunk at a heading [1][2].

This approach can help retrieval because related ideas are often placed under the same heading [2]. A chunk taken from one section is more likely to discuss one topic than a chunk that crosses several sections. It can also help a reader understand where a passage came from, because the heading provides context that a bare fragment may lack [1][2].

Structure is not always clean. A heading may introduce a very long section, a table may be split across pages, or a document may use formatting inconsistently [1]. A system may therefore combine methods: split at headings first, then divide any section that is still too large [1]. The lesson is to treat structure as evidence about meaning, not as a guarantee that every section is a perfect chunk [1][2].

Structure-based splitting exploits explicit markup, including HTML and Markdown heading syntax, to partition content by sections that often group semantically related text [1][2]. For code, analogous logical blocks such as functions and classes provide natural partition points [2]. This can improve downstream retrieval and summarisation because chunk boundaries align with authorial organisation rather than arbitrary length thresholds [2].

The approach has failure modes when markup is sparse, inconsistent or misleading, or when a single section exceeds model limits. Variable-sized chunking based on end-of-sentence punctuation, end-of-line markers or document-structure detection is therefore often combined with a size cap [1]. Custom combinations may also append document titles to chunks from the middle of a document to reduce context loss [1]. Such augmentation changes the embedded text and must be evaluated as part of the chunking strategy, not assumed to be neutral [1].

Semantic chunking tries to break text where the meaning changes, rather than where a fixed number of characters ends [1]. Imagine sorting a pile of notes by topic instead of cutting the pile into equal stacks. Sentences about the same idea stay together, and a new chunk begins when the subject shifts. The aim is to preserve context and relationships across sentences and paragraphs [1].

This can produce chunks that read as coherent units, which may help search and answering [1]. A focused chunk may make a sub-topic easier to represent, but whether it is retrieved more reliably must be checked on the actual questions. Semantic coherence and measured retrieval performance are separate considerations [1][3]. Semantic methods can also span page boundaries when a topic continues, instead of forcing a break because a page ended [1].

Semantic does not mean perfect. A chunk can still be too large for a model or too small to answer a question that needs background, and the sources do not describe a universal grouping rule for every document. Semantic chunking should therefore be compared with simpler methods on real questions, not chosen because its name sounds more intelligent [1].

Semantic chunking partitions content into units that preserve context and semantic relationships across sentences and paragraphs, and can span page boundaries [1]. The sourced description presents these chunks as better maintaining semantic coherence [1]. This describes the objective of the chunking approach rather than a universal algorithm. A coherent unit still needs a model-length check, and its usefulness depends on whether the resulting passage contains the information a real question requires.

A semantically coherent chunk is not necessarily an optimal retrieval unit. Retrieval depends on the alignment between query phrasing and chunk content, the density of relevant information within the chunk, and the amount of competing material included [1][3]. Overly broad units may include competing material, while overly narrow units may omit prerequisites needed to interpret a passage. Semantic segmentation therefore trades one form of fragmentation for another and requires task-specific evaluation [1][3].

A chunk boundary can separate a question from its answer in a simple way: the words that make a passage understandable end up in different chunks. A heading may name a topic, while the details appear after the cut. A pronoun may refer to something mentioned earlier, or a step may depend on a warning placed just before it [1][3]. The search may find one fragment, but the fragment alone may not contain enough context to answer correctly.

This is why context matters as much as keywords. A chunk containing the word a user asks about may still omit the condition, exception or definition that determines the answer [1]. Larger chunks and overlap can preserve more surrounding context, which helps queries that target specific information but need semantic richness [1]. Structure-based chunks can also help by keeping a heading with its section [1][2].

No boundary strategy removes the risk entirely. Even a well-formed chunk may lack information from another part of the document. A source-grounded answer needs support in the retrieved passages; missing context can leave the reader without enough evidence [3]. The system should retrieve the missing material or acknowledge the limitation instead of guessing an absent qualification. The tutorial instructs its reader to answer from context and decline when that context is insufficient, but an instruction is not a guarantee of model compliance [3]. Test questions whose answers depend on headings, references, lists and exceptions, not only questions matching a single sentence [1][3].

Boundary-induced context loss occurs when anaphoric references, headings, definitions, qualifications or procedural prerequisites are partitioned away from the span containing the answer [1][3]. The retrieved chunk may have high lexical or embedding similarity to the query while lacking the discourse context required for correct interpretation. Larger chunks and overlapping strategies can preserve context and semantic richness for specific-information queries, but they also include more non-relevant material [1].

Mitigations include structure-aware boundaries, overlap, and appending document titles or section headings to chunks to reduce context loss [1]. Each mitigation alters the indexed representation: added titles may improve matching for title-dependent queries while diluting content for others. When retrieved passages omit necessary context, the reader may lack sufficient evidence; an answer could then contain unsupported inference or miss a qualification [3]. These are possible failure modes, not inevitable outcomes or a guarantee that a grounding instruction will be followed. Chunking evaluation should therefore include context-dependent queries, not only keyword-matching queries [1][3].

A token is a small piece of text that a model processes. It is not exactly a word or a character, so counting words alone cannot guarantee that a chunk fits a model [1][2]. One model described in the sources accepts up to 8,191 tokens, which is roughly 6,000 words for common models, but that figure belongs to that model and should not be copied to others [1].

Embedding models turn a chunk into numbers that represent its meaning. The search then compares the numbers for a question with the numbers for chunks to find close matches [3]. If a chunk is too long for the embedding model, part of it may be lost. If it is too short, its numbers may represent only a fragment of the idea [1][3]. The chunk must therefore fit the embedding model and still be meaningful.

The reader model has its own limit. Retrieved chunks are combined into a prompt, so several chunks together must fit the reader's context [3]. A system can use small chunks for precise search but still overflow the reader if it retrieves too many of them. Chunk size, number of retrieved chunks and reader limit must be planned together [1][3].

Token limits impose hard constraints on both embedding and generation. Partitioning content helps meet input token requirements and prevents data loss from truncation [1]. The cited maximum of 8,191 tokens for text-embedding-3-small, approximated as around 6,000 words at roughly four characters per token for common OpenAI models, is model-specific and must not be generalised [1]. Token-based splitting aligns chunk construction with these limits, while character-based splitting provides only an approximation [2].

An embedding represents the input actually supplied to its model. If a long passage is truncated, omitted material cannot contribute to that embedding; the supplied guidance does not specify which part is retained in every implementation [1]. A much shorter chunk may instead omit context needed to interpret its topic. In the cited retrieval tutorial, vectors are used to match queries with stored records, and the original retrieved passage text is then placed in the reader prompt [3]. The passage is not reconstructed by decoding its embedding. This creates two separate size checks: each chunk must fit the embedding input, and the combined passage text, question and instructions must fit the reader’s total context budget [1][3].