RAG retrieval basics

Retrieval-augmented generation (RAG) grounds a language model in your own data by fetching relevant context at query time and stuffing it into the prompt. The generation half gets the attention, but retrieval quality is where most RAG systems live or die.

The pipeline

1. Chunk documents into passages small enough to be specific but large enough to be self-contained.
2. Embed each chunk into a vector with an embedding model.
3. Index the vectors in a vector store.
4. At query time, embed the question and fetch the nearest chunks.
5. Augment the prompt with those chunks and generate an answer.

Chunking is a real decision

Too small and a chunk loses the context that makes it meaningful; too large and you dilute the embedding and waste tokens. A reasonable starting point:

chunk size:    ~500–800 tokens
overlap:       ~10–15%
split on:      headings > paragraphs > sentences

Overlap keeps ideas that straddle a boundary retrievable from either side.

Similarity search

Most stores rank chunks by cosine similarity between the query vector and each chunk vector. Cosine measures the angle between vectors, so it compares direction (meaning) rather than magnitude (length):

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = sum(x * x for x in a) ** 0.5
    nb = sum(y * y for y in b) ** 0.5
    return dot / (na * nb)

Why hybrid retrieval wins

Pure vector search is great at semantic matches (“car” ≈ “automobile”) but weak at exact tokens — names, error codes, rare identifiers. Keyword search (BM25) is the opposite. Hybrid retrieval runs both and fuses the rankings (e.g. Reciprocal Rank Fusion), which reliably beats either alone.

What I still want to test

• Re-ranking the top-k with a cross-encoder before generation.
• How much a small overlap change moves answer quality.
• Query rewriting for under-specified questions.

Next note: turning this static retriever into an agent that decides when and what to retrieve — see LangGraph agent loops.