Introduction: The Boundary Problem
When a document is split into chunks for embedding and retrieval, a subtle but serious problem emerges at the boundaries. A sentence at the end of one chunk might depend on a sentence at the beginning of the next. A definition in chunk one might be referenced in chunk two. If a query matches the second chunk but not the first, the retrieved context is incomplete, and the model lacks the information needed to answer correctly.
Chunk overlap is the technique of including a portion of the previous chunk at the beginning of the next chunk. This ensures that no semantic unit is split across a boundary without some contextual continuity. It is a simple idea with profound implications for retrieval quality.
Why Boundaries Break Meaning
Natural language does not respect arbitrary token limits. A paragraph might flow logically across what would be a five-hundred-token boundary. Consider a medical guideline that states a diagnostic criterion in one sentence and the recommended follow-up procedure in the next. If the split falls between these sentences, a query about follow-up procedures might retrieve a chunk that mentions the procedure but omits the qualifying criterion. The model, lacking the criterion, might incorrectly generalize the procedure's applicability.
In technical documentation, boundaries are even more dangerous. A code example might span multiple chunks. A configuration instruction might be split across a boundary. An API parameter definition and its example usage might end up in different chunks. Each of these splits degrades the utility of the retrieved context.
How Overlap Works
Overlap is implemented by configuring the text splitter to include a specified number of tokens or characters from the end of the previous chunk at the start of the next. A typical configuration might use a chunk size of five hundred tokens with an overlap of fifty tokens. This means each chunk shares fifty tokens with its predecessor.
The overlap region acts as a semantic bridge. Queries that match concepts near the boundary have a higher chance of retrieving a chunk that contains the complete context, because that context appears in both the preceding and following chunks. The retrieval system effectively gets two chances to find the relevant information.
The Cost of Overlap
Overlap is not free. Because content is duplicated across chunks, the total number of chunks increases, which increases storage costs in the vector database. Each chunk must be embedded separately, so overlap also increases embedding computation costs. And because chunks are duplicated, the same information might be retrieved multiple times for a single query, wasting valuable context window space in the generation prompt.
These costs are generally modest compared to the benefits. A ten percent overlap on a five-hundred-token chunk adds only fifty tokens of duplication per boundary. For most applications, the improved retrieval accuracy justifies the incremental cost.
Tuning Overlap Size
The optimal overlap depends on the document type and the chunk size. For small chunks of one to two hundred tokens, an overlap of twenty to fifty tokens provides substantial boundary coverage. For larger chunks of one thousand tokens or more, a smaller proportional overlap may suffice because the boundary represents a smaller fraction of the total semantic unit.
Documents with highly structured content, such as JSON or XML, may benefit from boundary-aware splitting that respects structural delimiters rather than relying solely on overlap. Recursive character text splitters that prioritize paragraph breaks, sentence breaks, and word breaks over raw token counts often produce better results with less need for aggressive overlap.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(documents)
Overlap and Retrieval Redundancy
One side effect of overlap is retrieval redundancy. When a query matches content near a boundary, both the preceding and following chunks may be retrieved. This is usually beneficial, as it provides richer context. However, if the overlap is too large relative to the chunk size, the retrieved set may contain mostly duplicated content, leaving little room for diverse perspectives.
Managing this requires tuning the number of retrieved chunks and, in some cases, deduplicating overlapping content before injecting it into the prompt. Some advanced systems use maximal marginal relevance to balance similarity with diversity, ensuring that retrieved chunks provide complementary rather than redundant information.
Conclusion
Chunk overlap is a simple but essential technique for preserving semantic continuity across document boundaries. By duplicating a small portion of content between adjacent chunks, it ensures that queries matching boundary regions retrieve complete context rather than fragmented fragments. The incremental storage and compute costs are a small price to pay for the significant improvement in retrieval accuracy and answer quality.