What Is RAG (Retrieval-Augmented Generation)? A Clear Guide

LLMs only know what was in their training data, which stops at a knowledge cutoff and doesn't include your private documents. Ask about your refund policy and the model will guess, fluently and confidently. RAG fixes that by retrieving the actual policy and putting it in the prompt, ideally with citations the user can check.

Indexing (ahead of time):

1. Load documents and clean them.
2. Split them into chunks (for example ~500 tokens with some overlap).
3. Turn each chunk into an embedding vector.
4. Store vectors plus metadata in a vector database.

At question time:

1. Embed the question with the same embedding model.
2. Retrieve the top-k most similar chunks (often with hybrid keyword + vector search and a reranker).
3. Build a prompt: instructions, numbered sources, the question.
4. Generate an answer that cites the sources, or says "I don't know".

• Embedding queries with a different model than the documents.
• Chunks that are too big (noisy) or too small (no context).
• No metadata, so you can't filter or cite.
• Pure vector search missing exact terms like error codes or SKUs. Use hybrid search.
• Never measuring retrieval, so you can't tell whether bad answers come from search or from generation.

Measure the two halves separately. For retrieval, label which chunk answers each test question and track recall@k and MRR. For generation, grade faithfulness (is every claim supported by the sources?) and answer relevance, usually with an LLM judge spot-checked by humans.

Use RAG for knowledge: facts that change, private data, anything that needs citations. Use fine-tuning for behavior: consistent format, tone or a narrow task at lower cost. Fine-tuning is a poor way to add facts: they go stale and can't be cited.