Building a Production RAG Pipeline from Scratch

The Problem with Naive RAG

Most RAG tutorials stop at: chunk text, embed, retrieve, generate. Production RAG is harder.

The Chunking Problem

Fixed-size chunks (500 tokens) break semantic boundaries:

"...the function returns a Promise that"
"resolves to the user object. IMPORTANT:"
"Always validate the token before..."

The split corrupted meaning.

Semantic Chunking

hljs python

[object Object], ,[object Object],(,[object Object],) -> ,[object Object],[,[object Object],]:
    ,[object Object],
    sections = split_by_markdown_headers(document)

    chunks = []
    ,[object Object], section ,[object Object], sections:
        ,[object Object],
        sentences = nltk.sent_tokenize(section)
        current_chunk = []
        current_length = ,[object Object],

        ,[object Object], sentence ,[object Object], sentences:
            ,[object Object], current_length + ,[object Object],(sentence) > MAX_CHUNK_SIZE:
                chunks.append(,[object Object],.join(current_chunk))
                current_chunk = [sentence]
                current_length = ,[object Object],(sentence)
            ,[object Object],:
                current_chunk.append(sentence)
                current_length += ,[object Object],(sentence)

    ,[object Object], chunks

The Embedding Model Matters

Model	MTEB Avg	Speed	Cost
text-embedding-3-large	64.6%	Fast	$$
BGE-large-en-v1.5	64.1%	Medium	$
E5-mistral-7b	66.4%	Slow	$$

For most cases: text-embedding-3-small is sufficient and 10x cheaper.

Hybrid Search

Pure semantic search fails on exact matches (product IDs, error codes).

hljs python

[object Object],
semantic_results = vector_search(query_embedding)
bm25_results = text_search(query)

,[object Object],
scores = {}
,[object Object], rank, doc ,[object Object], ,[object Object],(semantic_results):
    scores[doc.,[object Object],] = scores.get(doc.,[object Object],, ,[object Object],) + ,[object Object],/(rank + ,[object Object],)
,[object Object], rank, doc ,[object Object], ,[object Object],(bm25_results):
    scores[doc.,[object Object],] = scores.get(doc.,[object Object],, ,[object Object],) + ,[object Object],/(rank + ,[object Object],)

Reranking

Initial retrieval: Top 100 documents Cross-encoder reranker: Reorder top 100, return top 5

The reranker is slower but more accurate. Use it only on candidates.

Evaluation

Measure before optimizing:

Context precision: Retrieved chunks contain answer?
Context recall: All relevant chunks retrieved?
Answer faithfulness: Generated answer supported by context?
Answer relevance: Answer addresses the question?

hljs python

evaluate(
    questions=test_set,
    pipeline=your_rag_pipeline
)
,[object Object],

Building a Production RAG Pipeline from Scratch

The Problem with Naive RAG

The Chunking Problem

Semantic Chunking

The Embedding Model Matters

Hybrid Search

Reranking

Evaluation

Production Checklist