Retrieval-Augmented Generation (RAG) is a prompting technique that grounds AI answers in documents you supply, instead of the model's training data. The lesson covers how RAG works, how to chunk documents without losing context, and how to write prompts that force citations and refusals when information is missing, cutting hallucinations.
What you will be able to do
- Write RAG prompts that answer only from provided context and cite sources
- Chunk documents into 200-500 word pieces with overlap so key facts don't get split across chunks
- Set confidence rules, strict, moderate, or lenient, that control when the model refuses to answer instead of guessing
- Decide when to use RAG versus fine-tuning for a given business use case
- Rank and select which documents to include in a prompt when working with multiple sources under a token budget
Before you start
- Access to a chat interface such as ChatGPT or Claude
- Documents you want the AI to reference, such as reports, handbooks, or product docs, in text form
- Basic understanding of prompts and token limits, covered in the lesson on LLM settings
Reference
| Element | Guidance |
|---|---|
| Simple RAG (small docs) | Paste the full document into the chat plus your prompt; works for documents under 2,000 words |
| Chunk size | 200-500 words per chunk |
| Chunk overlap | Include 1-2 sentences from the previous chunk at the start of the next one |
| Chunk boundaries | Split at section headers, paragraph breaks, complete sentences, or natural topic shifts |
| Never split | Mid-sentence, lists or tables, code blocks, or related facts such as dates paired with numbers |
| Confidence rule: Strict | Only answer if the exact information is explicitly stated; refuse when in doubt |
| Confidence rule: Moderate | Answer if clearly stated or directly inferable from multiple facts; cite all facts used |
| Token budget (4K example) | Instructions about 500 tokens, answer about 500 tokens, context about 3,000 tokens (roughly 2,000 words) |
Common errors and fixes
| What goes wrong | The fix |
|---|---|
| A key fact, like a sales figure, gets split across two chunks and only one is retrieved | Split at section headers or paragraph breaks, and add 1-2 sentences of overlap between chunks |
| The model gives an answer with no way to verify it | Add a rule requiring the model to cite the specific document and section for each claim |
| The model has no permission to say it doesn't know, so it guesses at missing data (e.g. inventing a Q4 revenue figure from Q3 trends) | Add a refusal rule: if the answer isn't in the context, say so explicitly |
| Using a lenient confidence rule lets the model fill gaps with assumptions | Use a strict or moderate confidence rule instead; lenient is not recommended |
| Irrelevant documents (e.g. HR Policies for a marketing spend question) get included and waste context budget | Rank documents by relevance to the question and include only the most relevant ones first |
Read the full walkthrough
The complete lesson, with screenshots and any downloads, is published on Substack as part of Prompt Engineering for AI Automation.
More in this section
- Lesson 8: From Prompting Patterns to Production Workflows: The Complete Integration Tutorial for ChatGPT and Claude
- Lesson 9: Stop Wasting 3 Hours Per Competitive Analysis
Continue the course
Browse all lessons in the Prompt Engineering for AI Automation course, or subscribe to the GenAI Unplugged newsletter to get new lessons in your inbox.
