AI Infrastructure

Chunking Is Where RAG Silently Fails

8 min read
A highlighted sentence sliced in half by scissors cutting across a page, evoking how RAG chunking severs meaning at the boundary.

You have tuned everything. A better embedding model, a reranker, hybrid search. Retrieval quality is still not what it should be, and you cannot find the bug, because there is no bug. The problem is upstream of everything you tuned, in a decision you made before any of it: how you cut the documents into chunks.

Chunking looks like preprocessing. A default you set once and forget. In practice it is the single decision that most shapes what your system is able to retrieve, and you make it at the worst possible moment, at index time, before you have seen a single query.

A chunk is the atom of retrieval. It is the smallest thing your system can return. When you choose chunk boundaries you are deciding, in advance and blind, which units of meaning will be retrievable at all, and every boundary you draw is a bet about questions you have not been asked yet. That bet is lossy, in three specific ways.

Three ways chunking loses information

Fragmentation. The answer to a question spans a boundary you drew, so retrieval returns half of it. The peer-reviewed taxonomy of RAG failures (Barnett et al., "Seven Failure Points of Retrieval-Augmented Generation," CAIN 2024) puts this in context: of its seven failure points, five sit in the retrieval and consolidation stages, not in the model. One of them, "incomplete," is fragmentation's endpoint, the pieces needed to answer are scattered across chunks and never reassembled. A question like "what is the dose, and what are the side effects" against a leaflet that keeps dose and side effects in separate sections is a fragmentation waiting to happen: retrieval brings back one section or the other, rarely both, and the model answers confidently from half the picture.

Dilution. Make the chunk bigger to avoid fragmentation and you create the opposite problem. A large chunk holding one relevant sentence and twenty irrelevant ones produces a muddy, averaged embedding that no longer points cleanly at the relevant idea, so it ranks below where it should. Chroma measured exactly this in their chunking evaluation: recursive splitting at 200 tokens reached roughly 7% precision, while the same method at 800 tokens fell to around 1.5%. The relevant span is still in the chunk. It is just drowned.

Decontextualization. A chunk that cannot stand on its own. Anthropic's example is the cleanest: a chunk reading "the company's revenue grew by 3% over the previous quarter" is nearly useless in isolation, because which company and which quarter lived in sentences that are now in a different chunk. Embedded alone it means almost nothing, and a query that names the company will not find it. Two independent labs documented this in the same month, Anthropic's Contextual Retrieval and Jina's Late Chunking, both September 2024, which is a good sign it is a real property of chunking and not one vendor's framing.

Three ways chunking loses information: fragmentation (the answer spans a chunk boundary), dilution (a relevant line buried in an oversized chunk), and decontextualization (a chunk that cannot stand on its own).

Why it fails silently

Here is what makes chunking dangerous rather than merely imperfect. None of those three failures produces an error. Retrieval returns chunks. The model reads them and writes a fluent, confident answer. The fragmented answer reads as complete. The diluted miss is invisible, you never see the better chunk that ranked just below the cutoff. The decontextualized chunk gets interpreted as if its missing context were present.

From the outside, a system built on badly cut documents is indistinguishable from one built on well cut documents, right until someone checks an answer against the source. It is the same shape as the other vector-search failure modes that only surface in production: the system reports success while quietly returning the wrong thing, and a system's report of its own health is not the same as its correctness.

You cannot chunk your way out of a chunking problem

The instinct here is to reach for a smarter chunker. If fixed-size splitting is the problem, surely a semantic splitter, one that places boundaries where the topic shifts, is the answer.

The evidence does not support that. A study on exactly this question (Qu et al., "Is Semantic Chunking Worth the Computational Cost?", Findings of NAACL 2025) tested semantic chunking against plain fixed-size chunking across document retrieval, evidence retrieval, and answer generation, and found the extra compute "not justified by consistent performance gains." And there is no universal target to tune toward in the first place: separate work (Bhat et al., 2025) found the best chunk size depends on the dataset and even the embedding model, so a size that wins on one corpus loses on another. There is no value you dial in once and keep.

None of this means all splitters are equal. The pragmatic baseline, the one that is genuinely hard to beat, is unglamorous: split along the document's own structure, on paragraph and section breaks, markdown headers, function boundaries in code, rather than at arbitrary token counts, and tune size and overlap to the kind of question you expect. Structure-aware recursive splitting respects the boundaries the author already drew. It just does not, by itself, solve the three losses above, which is the whole point.

So the fix is not a cleverer boundary. It is a different architecture.

Decouple what you retrieve from what you generate on

The core move is to stop forcing one span of text to do two incompatible jobs. Retrieval wants a small, precise unit, so the embedding is sharp and ranks well. Generation wants a large, coherent unit, so the model has enough context to answer. One chunk size cannot be both, so use two.

  • Parent-document retrieval (LangChain): embed and retrieve on small child chunks, but hand the model the larger parent they came from.
  • Sentence-window retrieval (LlamaIndex): embed a single sentence, then at query time swap in a window of the surrounding sentences before generation.
  • Auto-merging / hierarchical retrieval (LlamaIndex): index chunks at several sizes at once; when enough small children from the same parent are retrieved, merge up to the parent automatically.

All three are the same idea: retrieve small, generate large. You get the precise embedding and the coherent context without betting everything on one boundary.

Decoupling retrieval from generation: a query retrieves a small precise chunk, which expands to its larger parent chunk or window before the model generates from that coherent context.

The other family of fixes binds the context at index time instead. Anthropic's Contextual Retrieval prepends a short, document-aware description to each chunk before embedding it, so the "revenue grew 3%" chunk carries "this is from Acme's Q2 2024 report" into its own vector. On their evaluation the failed-retrieval rate drops from 5.7% to 3.7% with contextual embeddings, to 2.9% adding a contextual keyword index, and to 1.9% with a reranker on top. Jina's Late Chunking reaches the same goal differently: embed the whole document in one pass, then pool per chunk, so each chunk embedding already carries its neighbors' context. Treat the numbers as directional, they are each a vendor's own evaluation, but the direction is consistent and the mechanism is sound.

Measure it, because you cannot see it otherwise

Every choice above is a tradeoff with no universal winner, which means the only way to know whether your chunking is hurting you is to measure retrieval quality directly, not to inspect chunks by eye. Chroma's method is a good template: take real questions, mark the exact spans of text that answer them, and measure how much of that ground truth your retrieval actually returns, at the token level, for each chunking choice. Frameworks like Ragas formalize the same split as context precision and recall on the retrieval side. The specific tool does not matter. What matters is that chunking quality is a number you have to compute, because the failure never announces itself.

The pattern underneath

Step back and every fix here is doing the same thing. Chunking forces you to commit, at index time, to a single granularity of retrievable state, and that commitment is premature, you are fixing the unit of meaning before you know what will be asked of it. The fixes all defer the commitment: retrieve on the small unit, but bind the surrounding context late, at query time or at generation time, instead of baking it into one boundary you can never revisit without rebuilding the index.

That last part is what makes it expensive, and it connects chunking to its sibling problem. Like changing your embedding model, a chunking decision is an index-time commitment you cannot undo at query time. Get it wrong and the fix is not a setting, it is re-chunking and re-embedding the whole corpus. Which is exactly why it is worth getting right before you scale, and worth measuring after.

Chunking is not preprocessing. It is the architecture of what your system is able to know.

You cannot chunk your way out of a chunking problem. You have to change what the chunk is for.

Sources

The figures here are reported by the vendors and papers cited, each on their own evaluation, not independently verified. Read them as directional.