RAG in plain English

Someone asks the internal chatbot: “What is the official definition of active customer?” The model answers fluently. It sounds like last year’s all-hands. It is also wrong. Finance updated the rule in a Confluence page in March. The model never saw March. It only saw training data and whatever random paragraphs your search slapped into the prompt.

That is the problem RAG is trying to fix. Not magic memory. Not “the AI finally understands our company.” A pipeline: find relevant company documents at ask time, stuff the useful bits into the prompt, then let the model write an answer grounded in those bits. When retrieval works, answers cite the March page. When retrieval fails, you still get confident prose. Same risk, different costume.

This is Part 5 of Practical AI for analytics people. Part 4 covered human evals. Here we stay conceptual: what retrieval-augmented generation is, how it maps to analytics work, when it helps, and when it will not save a junk wiki. We will not install a vector database or walk through index configs. For that machinery, read Getting Up to Speed on Vector Databases and keep this post as the “why and when” layer. LLM basics still live in What are LLMs?. If you use any of this to draft SQL, keep checking AI-written SQL in the loop.

What you’ll learn

• RAG in one plain sentence, then in a workplace diagram
• How docs, chunks, retrieval, and generation fit together without vendor jargon
• What “grounding” can and cannot guarantee for metrics and policies
• When RAG helps analytics teams versus when better catalogs, SQL, or humans win
• A worked example: answering “active customer” with and without retrieval
• Failure modes you should put on an eval card (from Part 4)

RAG in one breath

Retrieval-augmented generation means: before the model answers, a system searches a document store for passages that look relevant to the question, then includes those passages in the prompt so the model can use them. The generation step is still a language model. The new idea is the open-book exam. Closed-book is “answer from training.” Open-book is “answer with these pages on the desk.”

Researchers popularized the RAG name for combining parametric memory (what the model learned in training) with non-parametric memory (what you retrieve at runtime). You do not need the paper to use the idea. You do need the humility: retrieved text can be wrong, outdated, or about the wrong product line, and the model will still sound sure.

Rule of thumb: RAG upgrades the prompt with company context. It does not replace data quality, metric ownership, or reading the source page when the number matters.

The flow: docs, retrieve, stuff, prompt, answer

Strip the product names. Most company “knowledge assistants” do some version of this:

1. Collect docs: wiki pages, metric cards, runbooks, Slack exports (carefully), PDFs, ticket macros.
2. Chunk: split long pages into smaller pieces so search can rank paragraphs, not 40-page manuals as one blob.
3. Index: store chunks so you can find them fast. Often that uses embeddings and a vector index. Sometimes it is keyword search. Sometimes both.
4. Retrieve: turn the user question into a search, pull top chunks.
5. Stuff (or select): put the best chunks into the model prompt, within the context window budget from Part 2 of this series.
6. Generate: model writes an answer, ideally with citations to the chunks.

Vector databases show up in step 3 when you want similarity search over embeddings. That is useful for “find paragraphs about refund exclusions even if the wording differs.” It is not required for every internal FAQ. Keyword search still works when people share vocabulary. Our vector post covers the storage side; this post cares about the decision: do you need retrieval at all, and what fails when you have it?

Why analytics people should care

Analytics work is full of semi-structured knowledge that never lands cleanly in a warehouse column:

• Metric definitions and edge cases (“active means logged in once in 28 days, except enterprise trials”)
• Pipeline runbooks and “do not use this table on Mondays” folklore that should be docs
• Dashboard purpose statements and known caveats
• Access policies and PII handling notes
• Historical incident writeups that explain weird spikes

SQL alone does not answer “what does Finance mean by net.” A model alone invents a plausible Finance. RAG plus a maintained definition page can quote the real rule. Catalogs and stewardship habits still matter; a bot that retrieves a stale card is a faster way to spread the wrong definition. Pair this thinking with data stewardship and metric specs from the metrics series when those paths are on your Learn map.

Grounding is not a warranty

Vendors love the word “grounded.” In practice, grounding means “the model saw some retrieved text.” It does not mean:

• The retrieved text was the latest approved version
• The chunk was complete (half a definition can be worse than none)
• The model used the chunk instead of its prior habits
• Two conflicting docs were resolved correctly
• The answer is safe to put on a board slide without a human

Good systems show citations and let you open the source. Great teams treat citations as a starting point for audit, not as a stamp. Part 4’s eval suite should include at least one RAG case: “answer only from these docs; if missing, say you do not know.”

When RAG helps vs when it does not

Use this as a team conversation card, not as theology.

Notice the pattern. RAG is strong for “what did we write down?” It is weak for “what is true in the data right now?” and weak for “what should the company decide?” Those need queries and owners.

Worked example: “What is an active customer?”

Suppose three artifacts exist in the company:

• Wiki page A (2024): active = any login in 90 days.
• Metric card B (March 2026, certified): active = paid seat with at least one intentional session in 28 days; exclude internal tenants; trials counted separately.
• Slack thread C: a VP saying “just use 30 days for the board, whatever.”

Closed-book chat (no RAG): the model invents a plausible SaaS definition, maybe 30 days, maybe 90, with no company specificity. Sounds fine. Wrong for your Finance process.

Naive RAG: search returns page A because it is longer and older and ranks high on the phrase “active customer.” The model quotes 90 days with a citation. Still wrong for 2026 reporting.

Better RAG design: index prefers certified metric cards, stores effective dates, boosts “certified” tags, and the prompt says: prefer metric cards over informal pages; if sources conflict, list them and stop. The answer cites card B, notes page A as deprecated if that metadata exists, and ignores Slack unless policy allows it.

Human still owns the number: if you need a headcount for the board, you run the certified query or open the certified dashboard. The RAG answer is how you remember the rule, not a substitute for the measure.

A tiny “prompt pattern” that helps when you control the assistant instructions:

You answer using ONLY the passages provided below.
If the passages are missing, incomplete, or conflict, say so explicitly.
Quote the definition and name the source title and date when available.
Do not invent SQL table names. Do not invent numeric results.
Passages:
{{retrieved_chunks}}
Question:
{{user_question}}

That pattern is still only as good as retrieval. If card B never enters {{retrieved_chunks}}, the model cannot quote it. Garbage in, fluent garbage out.

Chunks, context windows, and the suitcase problem

Part 2 of this series treated tokens like suitcase space. RAG fills the suitcase with other people’s packing. If you retrieve ten noisy chunks, you crowd out the question and the schema. If you retrieve one tiny chunk that cuts a definition mid-sentence, you get half a rule. Practical habits:

• Prefer fewer high-quality chunks over many “sort of related” ones.
• Keep metric cards short and self-contained so a single chunk carries the full rule.
• Store titles, owners, and dates in the chunk metadata so answers can cite them.
• Re-index when certified pages change, not “someday.”

Long PDFs of board decks are often poor retrieval fuel. Clean metric cards and runbooks are better. That is a documentation problem wearing an AI hat.

RAG is not a warehouse, and not an agent

Two confusions show up in meetings:

• “We’ll put the data warehouse into RAG.” Usually they mean “embed dashboard PDFs” or “index table descriptions.” That can help discovery. It does not make the model a correct aggregation engine for weekly revenue. Aggregations belong in SQL and certified transforms. See pipelines and metrics series on Learn when you need that stack.
• “RAG is our agent.” RAG retrieves and generates. Agents (Part 6) choose tools, take multi-step actions, and can write back to systems. Different risk class. You can put RAG inside an agent. You should not pretend a doc chatbot is already safe automation.

What to put in a RAG eval (human-sized)

Borrow Part 4. Add cases like:

If you cannot pass “missing topic,” your bot is a fiction generator with footnotes.

Common mistakes

• Indexing everything, curating nothing. Volume without ownership makes retrieval confidently wrong.
• No effective dates. Definitions without “as of” become landfill.
• Treating citations as proof. Citations prove a chunk was present, not that the answer is decision-ready.
• Skipping access control on the index. A search index can become a PII side channel.
• Replacing the catalog with chat. Catalogs and stewards still need named humans. Chat is a UI, not ownership.
• Expecting RAG to fix bad joins. Wrong SQL is still wrong. Check it.

Practice this week

1. List five questions your team asks a chatbot (or wishes they could).
2. For each, name the one document that should answer it. If none exists, write a half-page card before buying tools.
3. Mark which questions need a warehouse query instead of a doc answer.
4. If you already have an internal assistant, run three of those questions and open every citation. Note stale or missing sources.
5. Add one “missing topic” and one “conflicting docs” case to your Part 4 eval sheet.

Next, Part 6 covers agents, tools, and harnesses: what happens when the model can not only read docs but call systems, and how guardrails and humans in the loop keep that from becoming an automated incident. For vector storage detail without redoing this conceptual map, stay with the vector databases post.

Quick recap

• RAG: retrieve company passages, put them in the prompt, then generate.
• It helps “what did we write down?” more than “what is true in the data right now?”
• Vectors and indexes are implementation details; link out for that depth, do not confuse them with judgment.
• Grounding is not a warranty. Citations help audit. Humans still own numbers.
• Curate docs, dates, and access. Eval missing and conflicting cases. Pair with SQL checks for anything query-shaped.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401): https://arxiv.org/abs/2005.11401
• Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv:2312.10997): https://arxiv.org/abs/2312.10997
• NIST, AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
• OWASP, Top 10 for LLM Applications: https://owasp.org/www-project-top-10-for-large-language-model-applications/
• Analytics Made Simple, Getting Up to Speed on Vector Databases: https://analyticsmadesimple.com/data-engineering/getting-up-to-speed-on-vector-databases/
• Analytics Made Simple, What are LLMs?: https://analyticsmadesimple.com/analytics/what-are-llms-chatgpt-generative-ai-and-more/
• Analytics Made Simple, How to Check AI-Written SQL: https://analyticsmadesimple.com/tutorials/how-to-check-ai-written-sql/
• Analytics Made Simple, SQL series: https://analyticsmadesimple.com/series/sql/

More definitions in Key Terms →

Merch on Etsy

Open store →

You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Y Hat Mouse Pad, Data Science Mouse Pad, Analytics Mouse Pad, Y Pred Mouse Pad View on Etsy → You're On Mute White Ceramic Coffee Mug, Data Science Mug, Data Mug, Analytics Mug, Statistics Mug, Programming Mug, Gift Mug, Coffee Mug View on Etsy → Dashboard to Excel Ceramic Coffee Mug View on Etsy →