Chunking decides your RAG quality more than your model does
A retrieval system can only be as good as the passages it hands the model. If the answer lives in a paragraph that was cut in half at ingestion time, no amount of model quality recovers it.
Fixed-size chunking loses the thing you were looking for
Splitting on a character count is easy to implement and easy to reason about, which is why almost every project starts there. It also cuts through the middle of clauses, table rows and numbered lists. A contract term that reads "either party may terminate with thirty days written notice" becomes two chunks, and neither one answers the question on its own.
Split on structure first, size second
In practice the ordering that works is:
- Respect the document's own structure — headings, list items, table rows, contract clauses.
- Merge neighbouring units until you approach the target size.
- Only fall back to a hard split when a single unit is genuinely too large.
That keeps the semantic unit intact and treats the size limit as a ceiling rather than a grid.
Overlap is a patch, not a design
A sliding overlap of a couple of hundred tokens hides some of the damage from fixed-size splitting, at the cost of storing the same text several times and pushing duplicates into the model's context. It is worth having, but if overlap is doing heavy lifting, the chunker is the thing to fix.
Measure it before you tune it
Groundedness and citation coverage are the two numbers that move when chunking changes. Set up a small evaluation set of real questions with known correct sources, then change one thing at a time. Most teams find a bigger jump here than from any model swap.