Fine-tuning vs RAG

Leadership wants the chatbot to “know our business.” Two proposals hit the channel at once. Team A wants to fine-tune a model on last year’s tickets and PDFs. Team B wants RAG: keep a normal model, retrieve the right pages at question time, and ground the answer. Both can be right for different jobs. Both can waste a quarter if you pick by hype instead of failure mode.

Fine-tuning updates model weights (or efficient adapters) so generation behavior shifts: tone, format, domain phrasing, task skill. RAG (retrieval-augmented generation) leaves base weights mostly alone and stuffs retrieved evidence into the prompt so answers can cite fresher company text. One changes the student. The other puts books on the desk during the exam. You sometimes want both. You almost always want evals either way.

This Key Terms guide is a one-shot decision piece for analytics and data people pulled into AI projects. It complements the Practical AI series post on RAG without replaying every pipeline step. For LLM vocabulary, see What are LLMs?. For retrieval machinery, see vector databases and the embeddings Key Term. For SQL that models draft, keep checking AI-written SQL.

What you’ll learn

Fine-tuning vs RAG in plain sentences and a side-by-side diagram
What each is good at: style and tasks vs fresh facts and citations
Cost, latency, data, and maintenance differences that show up after the demo
When to start with RAG, when fine-tuning earns its keep, when to combine
A worked workplace scenario (policy bot vs tone-standardized writer)
Mistakes, practice drills, and honest limits

Two different jobs

Fine-tuning

Fine-tuning continues training (fully or via parameter-efficient methods) on examples you provide so the model’s behavior shifts. Supervised fine-tuning might teach: “Given this ticket, emit this JSON schema,” or “Rewrite in our support voice.” The new behavior is baked into weights or adapters you deploy. Updating facts means new training data and a new training run (or frequent refresh jobs). It is not a live wiki.

RAG

RAG retrieves relevant passages from a corpus at request time, then generates an answer conditioned on those passages (and the user question). Change a Confluence page, re-index, and the next answer can see the update without retraining the generator. Failures move to retrieval quality: wrong chunk, stale index, missing permissions, blended conflicting sources.

Fine-tuning changes model weights. RAG keeps the model and retrieves documents into the prompt at ask time.

Rule of thumb: Need the model to speak or format differently? Consider fine-tuning (or strong prompting first). Need the model to use today’s internal facts with citations? Start with RAG.

Comparison that survives a budget meeting

Dimension	Fine-tuning	RAG
Primary win	Behavior, style, task skill, format reliability	Grounding in external/current docs
Fact freshness	As fresh as last training set	As fresh as the index and sources
Upfront cost	Data labeling, training compute, eval harness	Ingestion, chunking, index, retrieval evals
Runtime cost	Often lower prompt size; model may be specialized	Retrieval + larger prompts with context
Citations	Weak unless you still retrieve or hardcode	Natural fit (show retrieved sources)
Best early bet for internal Q&A	Usually not	Usually yes (AWS and others recommend starting RAG for custom-doc QA)
Failure mode	Confident outdated knowledge; overfitting to noisy labels	Wrong neighbors; prompt stuffing; fluent misuse of bad chunks

Google Cloud’s guidance on using data with LLMs similarly frames RAG as grounding on your data at runtime and supervised fine-tuning as teaching well-defined task behavior. Different knobs. Different maintenance clocks.

Comparison card of fine-tuning versus RAG across freshness, cost, citations, and best first use cases

When RAG is the better first move

Policy, runbook, and metric-card question answering
Docs that change weekly (and leadership expects the bot to notice)
You need audit trails: “which paragraph did we show the model?”
You lack a clean, large, labeled fine-tune set but you do have a wiki
Multiple source systems with permissions that must filter retrieval

RAG still fails if the wiki is garbage. Retrieval cannot invent a single approved active-customer definition from three conflicting pages. Fix the corpus and ownership; then retrieve.

When fine-tuning earns its keep

Strict output schemas (JSON the app must parse) after prompting plateaus
House style that must stick across thousands of generations
Domain jargon and task patterns that base models keep mangling
Classification or extraction quality that offline evals prove needs weight updates
Latency or cost constraints that punish huge RAG contexts every call

Fine-tuning is a poor substitute for “please memorize the March refund policy.” Policies want retrieval or deterministic tools. Fine-tuning is also a poor substitute for teaching the model private facts you are unwilling to put in training data under a clear governance process.

Combining them (without turning into a science fair)

Common pattern: fine-tune (or heavily prompt-engineer) for format and tone; use RAG for facts. Some teams explore retrieval-augmented fine-tuning variants; you do not need the acronym to use the idea. Keep the system diagram boring:

Retrieve top chunks with filters.
Build a prompt that separates instructions, retrieved evidence, and user question.
Generate with a model that is either base + good instructions or fine-tuned for your schema/voice.
Evaluate with a golden set: faithfulness to sources, refusal when empty, format validity.

Tool calling (query warehouse, hit metric API) sits next to both. For analytics numbers, prefer tools over free-text memory, whether tuned or retrieved.

Worked scenarios

Scenario 1: Internal policy assistant

Need: Answer “How do refunds work for EU annual plans?” with links to the current policy. Better first approach: RAG over approved policy pages with ACL filters and citations. Fine-tune later? Only if you need a fixed answer JSON for a workflow engine and prompting fails. Do not: Fine-tune on a ZIP of PDFs and skip retrieval, then wonder why June’s policy never appears.

Scenario 2: Ticket reply drafting

Need: Draft replies in a calm company voice, structured with greeting, steps, closing. Better first approach: Strong system prompts and few-shot examples; maybe fine-tune if volume and evals justify. RAG role: Pull the relevant troubleshooting snippet for the product area so steps stay accurate. Voice from tune/prompt; facts from retrieval.

Scenario 3: Analytics SQL helper

Need: Help analysts draft SQL against your warehouse. Reality check: Fine-tuning on SQL pairs can help dialect and style. RAG over table docs and metric cards can help name columns. Neither replaces dry-run checks, grain checks, or human review. Publish a checklist, not a vibes-based “AI DBA.” Point people at How to check AI-written SQL and the SQL series.

-- Decision pseudocode for a new "know our business" request
if need_current_internal_facts or citations:
    build_rag(sources=approved_corpus, filters=acl, evals=faithfulness)
if need_strict_format or stable_voice and prompting_plateaued:
    consider_fine_tune(task_examples, evals=schema_and_quality)
if need_exact_numbers_from_warehouse:
    prefer_tools(sql_or_metric_api)  # not free-text memory
always:
    human_review_for_high_stakes
    monitor_and_golden_set()

Data and governance differences

Fine-tuning data is training data: labels, licenses, PII scrubbing, retention, and “can this ever leave our VPC?” questions. RAG data is an operational corpus: access control at retrieve time, redaction, and index hygiene. Both need owners. Fine-tunes freeze a snapshot of behavior; RAG freezes whatever you failed to delete from the index. Different incident types, same need for stewardship habits from data quality and stewardship practices.

Security note: prompt injection is more discussion-native to RAG and tool-using agents (untrusted retrieved text gives instructions). Fine-tuned models can still be steered by malicious user prompts. Neither is “safe by architecture.” Design filters and human escalation for high-impact actions.

Common mistakes

Fine-tuning to avoid building search over messy docs. You will retrain forever.
RAG without evals, celebrating demo day until the first confident wrong policy answer.
Retrieving metrics prose instead of querying the warehouse for numbers that must match Finance.
Skipping prompt baselines before fine-tuning. Many “tune” requests die after a better system prompt.
Mixing unvetted sources in one index (HR drafts + customer help) without filters.
No rollback plan for a bad fine-tune or a poisoned document that ranked #1 all week.
Declaring victory on fluency instead of task success rates.

How to practice

Write one paragraph for your VP: fine-tuning vs RAG using the student vs open-book metaphor. No product names required.
Inventory one AI pilot. Label each requirement as behavior, fact freshness, citation, or exact numeric tool. Map requirements to prompt, RAG, fine-tune, or API.
Build a 20-question golden set for any internal Q&A bot. Score retrieval hit rate and answer faithfulness separately.
Try to solve a format problem with prompts and examples first. Only then draft a fine-tune data shape.
Read one vendor comparison (AWS RAG vs fine-tuning, or Google’s tune guide) and note where your use case sits. Link it in your design doc.

Quick recap

Fine-tuning changes model behavior; RAG supplies external text at ask time.
Internal fact Q&A usually starts with RAG; style and hard task formats may justify fine-tuning after prompts plateau.
Freshness, citations, and maintenance clocks differ; budget for the one you choose.
Combine carefully: retrieve facts, tune or prompt for form, tools for exact numbers.
Evals, permissions, and source ownership beat architectural fashion.

When someone says “we should fine-tune so it knows our data,” translate: “Should we change weights, retrieve documents, call tools, or clean the wiki?” That translation is the whole job of this Key Term.

Sources

Further reading and references used for this article:

AWS Prescriptive Guidance, comparing RAG and fine-tuning: https://docs.aws.amazon.com/prescriptive-guidance/latest/retrieval-augmented-generation-options/rag-vs-fine-tuning.html
Google Cloud, “To tune or not to tune?” (RAG vs supervised fine-tuning): https://cloud.google.com/blog/products/ai-machine-learning/to-tune-or-not-to-tune-a-guide-to-using-your-data-with-llms
Oracle overview of RAG vs fine-tuning: https://www.oracle.com/artificial-intelligence/generative-ai/retrieval-augmented-generation-rag/rag-fine-tuning/
AMS: Practical AI series, What are LLMs?, Vector databases, How to check AI-written SQL, Learn