Chunking, chunk size, overlap, and metadata

Overview

• Chunking divides documents into retrieval-sized pieces.
• Chunk size controls how much content each piece contains; overlap repeats a small amount at boundaries.
• Metadata identifies the piece's source, permissions, version, and other useful attributes.

Why this matters

Whole documents are often too broad for precise retrieval, but isolated sentences may lose essential conditions.

An everyday example

You bookmark a complete paragraph and its heading rather than one word or an entire book.

A technical example

• Start with section-aware chunks around 400–800 tokens and modest overlap, then tune using actual evaluation results.
• These are experiment settings, not universal defaults.

How it works

1. Detect headings/sections
2. Group coherent text
3. Split oversized sections with bounded overlap
4. Carry titles and source coordinates
5. Store stable chunk IDs and metadata.

Architecture and flow

Parsed sections → coherent chunks → source/ACL metadata → embedding + storage.

The insurance assistant example

Keep an exclusion and its exception together. Attach policy ID, effective dates, page, product, region, and tenant.

When to use it

Larger source documents and retrieval requiring precise passages.

When to avoid it

Do not split small self-contained FAQ answers unnecessarily or use a fixed character limit that destroys tables and legal conditions.

Interview answer

“I chunk by meaning and structure, tune size and overlap through retrieval tests, and retain metadata for eligibility, freshness, and citations.”

Benefits, trade-offs, and production considerations

• Smaller chunks improve precision but can lose context; larger chunks preserve context but add irrelevant tokens.
• Overlap increases storage and duplicate retrieval.
• Token-aware chunking must account for the embedding model's input limit, not only the generation model's limit.