Production RAG: where the textbook architecture fails

The standard tutorial on Retrieval-Augmented Generation (RAG) makes it sound trivial: take your company documents, split them into 500-token chunks with a 50-token overlap, embed them using an off-the-shelf vector model, and fetch top-\(k\) nearest neighbours by cosine similarity.

If you deploy this architecture to enterprise users, accuracy degrades rapidly. Users search for exact error codes, table lookups spanning multiple sections, or negated queries, and the vector store returns semantically related fluff that pollutes the model's context window.

The three fatal flaws of naive chunking

Vector embeddings compress an entire text window into a single fixed-dimension vector (e.g., 768 or 1536 floats). This mathematical bottleneck causes specific systematic failures:

• Loss of local granularity: Exact identifiers (UUIDs, model serials, numerical values) are smeared across latent dimensions, failing exact-match queries.
• Context tearing: Splitting at arbitrary token boundaries divorces a sentence from its qualifying preconditions or table headers.
• Context poisoning: Injecting irrelevant but semantically similar chunks consumes context budget and increases hallucination rates by 40%.

# Hybrid Search via Reciprocal Rank Fusion (RRF)
def reciprocal_rank_fusion(
    bm25_results: list[str],
    vector_results: list[str],
    k: int = 60
) -> list[tuple[str, float]]:
    scores = {}
    for rank, doc_id in enumerate(bm25_results):
        scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
        
    for rank, doc_id in enumerate(vector_results):
        scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
        
    # Sort descending by fused reciprocal score
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

Cosine similarity in a dense embedding space measures topical similarity, not factual relevance. A paragraph discussing why a feature was deprecated will have high cosine similarity to a query asking how to configure it.

The production pipeline: Hybrid retrieval & Cross-Encoders

Robust retrieval pipelines combine three distinct stages:

1. Hybrid sparse-dense retrieval: Query both a BM25 inverted index (for exact keywords, part numbers, and terminology) and an HNSW vector index in parallel.
2. Reciprocal Rank Fusion (RRF): Merge rankings without needing to normalise disparate distance metrics into raw scores.
3. Cross-Encoder re-ranking: Pass the top 50 candidates through a lightweight cross-encoder model (such as BGE-Reranker) that performs joint self-attention over the query-document pair before feeding the top 5 chunks into the generation prompt.

Parent document retrieval and metadata filtering

Instead of feeding small embedding chunks directly to the LLM, use small chunks (128 tokens) for embedding indexation, but maintain a reference mapping back to the full parent section (1,000–2,000 tokens). When a sub-chunk matches, inject the coherent parent document into the prompt.

Combining parent document expansion with pre-retrieval SQL/metadata filtering eliminates 85% of retrieval hallucinations while keeping token usage tightly bounded.

Experiencing retrieval failures in production search or RAG systems? Reach out to discuss architecture patterns.