Your RAG demo works. That is the problem.

A retrieval demo takes an afternoon. I have built maybe fifteen of them. You load some PDFs, chunk them at 500 tokens, throw them in a vector store, wire up an answer prompt, and the thing works. It is genuinely magic the first time.

Then you put it in front of forty real users and it falls apart in ways that have nothing to do with the model.

That gap is the whole job. Everything interesting about building with AI lives in it.

The demo path has three steps. Production has nine.

Here is what the afternoon version does: take the question, grab the top five chunks, generate. Done.

Here is what a system people actually rely on has to do:

• Work out what the user meant, because real questions are short, misspelled, and full of internal acronyms
• Search more than one way, because pure vector search cannot find an invoice number
• Filter by who is allowed to see what, before ranking, not after
• Re-order the candidates with something that actually reads them
• Notice when the evidence is thin and say so instead of inventing
• Cite where the answer came from, in a way a person can click
• Log the whole trajectory so you can debug it on Tuesday

None of that is model work. All of it is product work.

Chunking is where I lost the most time

My honest confession: I spent about three weeks tuning prompts on a project where the actual problem was that our PDF parser was silently dropping tables. The model was fine. The prompt was fine. The evidence never made it into the system in the first place.

Fixed-size chunking is the default in every tutorial and it is wrong for most real documents. A 500-token window cuts a table in half. It splits a policy from its exception. It separates a heading from the thing the heading was describing, which means the retrieved chunk is technically about the right topic and practically useless.

What worked better for us:

• Chunk on document structure, not character count. Headings, sections, table boundaries.
• Keep the parent heading in the chunk text. Cheap, and it fixes a surprising number of misses.
• Store the page number and section title as metadata from day one. You will want them for citations and you cannot backfill them without reindexing everything.

The failure that scared me

There is a paper from USENIX Security 2025 called PoisonedRAG. The researchers inserted five malicious documents per target question into a knowledge base with millions of entries and got a reported 90% attack success rate. In one black box setting it hit 97%.

Five documents. Out of millions.

The part that should worry anyone shipping this stuff is what they found about defences. Paraphrasing did not stop it. Perplexity filtering did not stop it. Duplicate detection did not stop it. If your ingestion pipeline lets anything in, your retrieval layer is an attack surface, and it is one most teams have not thought about because it does not look like a security feature. It looks like a content pipeline.

Practical version for a small team: authenticate what goes in, keep provenance on every document, scope retrieval per tenant, and alert when one source suddenly dominates results for a query it never used to.

What I would tell myself a year ago

Build the boring parts first. Authorization, provenance, citations, and a way to say "I do not know". They are unglamorous and they are the difference between a thing people trust and a thing people quietly stop opening.

Also: the demo working is not evidence. It is the null hypothesis.