When RAG returns the wrong passage

Retrieval-augmented generation fails in a specific way: the system fetches a chunk that is nearby in embedding space but not the chunk that answers the question. The model then writes a fluent paragraph that is faithful to the wrong paragraph. Readers see a citation and assume the claim was checked. It was not.

I debug RAG from the retrieved list, not from the generated sentence. If the top chunks are from the related-work section when the question was about the loss function, the index is wrong. Chunks that split a methods sentence in half, missing metadata (year, section, paper id), and an embedding model trained on generic web text all produce that error. Hybrid search—sparse keywords plus dense vectors—often helps on technical terms such as architecture names and dataset ids.

Show the passages. A teaching or research assistant that hides the retrieved text and shows only a polished answer is worse than a search box. I want the student to see which sentence was used, and to say when none of the chunks were good enough. “I do not know from this corpus” is a valid system output.

RAG still does not make the model a clinician, and it does not turn a preprint into a published result. It is a way to keep generation attached to notes and papers you are willing to name. If you cannot name them, you do not have RAG. You have a chatbot with extra latency.