RAG Search in Production: What Actually Matters

I built a search engine over 2,700 apps that answers questions like "something to track my reading that is not a social network". It works, people use it, and the part that made it work was not the model.

Most RAG writing is a tutorial: chunk, embed, store, retrieve, prompt. That part takes an afternoon. What follows is the rest, which took considerably longer.

Retrieval quality is the whole product

If the right document is not in the top few results, no prompt saves you. The model will answer confidently from whatever it was handed, which is worse than saying nothing, because a confident wrong answer is the one users act on.

So the work is retrieval, not generation. When output is bad, the instinct is to edit the prompt. The question to ask first is whether the correct source document was even retrieved. Usually it was not, and no amount of prompt editing will fix a missing input.

What you embed matters more than the model you embed with

I spent early effort comparing embedding models and gained very little. I gained a lot from changing what went into them.

Embedding a raw description gets you documents that match on wording. Embedding a short generated summary of what the thing is for gets you documents that match on intent, and intent is what people actually type. Nobody searches the way a product describes itself.

Include the metadata that matters in the embedded text rather than only in a filter. Category and purpose belong in the semantic space, because a query mixes them freely with everything else.

Pure vector search misses things keyword search catches

Vector search is bad at exact tokens. A product name, a version number, an acronym: these are precisely the queries where a user knows exactly what they want, and semantic similarity dilutes them into a neighbourhood of vaguely related things.

Run both. Vector search for meaning, keyword search for exactness, then merge. The merge does not need to be sophisticated, and skipping it entirely is a much bigger error than merging it crudely.

Chunking is a product decision, not a parameter

Every guide gives you a number. The number depends on what a good answer looks like.

If answers should cite a specific paragraph, chunk small and keep enough surrounding context to make the fragment intelligible on its own. If answers describe an entire item, one chunk per item beats slicing that item into six pieces that each partially describe it.

For a catalogue, one document per product was obviously right in hindsight and not obvious at the time, because the guides were written for long documents.

Say when you do not know

The most valuable behaviour I added was refusal.

If the top result is below a similarity threshold, the honest answer is that nothing matched. A system that occasionally admits this is trusted more than one that always produces something, because users find the fabricated answer eventually and then stop trusting every answer.

Pick the threshold by looking at real failed queries, not by intuition. You want the number where "nothing matched" is more often right than a stretch.

Where the cost actually goes

Not the model. Embedding a catalogue once is cheap and querying is cheaper.

The cost is keeping embeddings current as the underlying data changes, and that is an infrastructure problem rather than an AI one. Something has to notice a record changed and re-embed it. Get that wrong and your search quietly describes a version of the catalogue that stopped existing months ago, which is the same class of failure as a stale cache and just as invisible.

Which is the honest summary: the interesting problems in a RAG system are the ordinary engineering ones sitting around it.