How to give an AI your company’s knowledge: RAG, fine-tuning, or long context

Pick how your AI gets its knowledge by the job: look it up, train it in, or read it all at once. Have the AI look things up in your documents first, an approach called RAG, when your facts change and people need to see sources. Use fine-tuning when you need the AI to write in a particular style or format every time. Use a long context window when the job means reading a whole document at once, and keep an eye on the cost.

Imagine you want an AI assistant that answers questions about your company’s contracts. One colleague suggests training a model on all the contracts. Another suggests pasting every contract into each question. A third suggests looking up the relevant clauses first. Each works for some jobs and fails badly for others, and the wrong choice can cost a lot or go out of date quickly.

This post closes the RAG from scratch series. RAG is short for retrieval-augmented generation. Fine-tuning means extra training that adjusts a model on your own examples. A context window is the amount of text a model can read in one go, and a long one lets you hand it a whole document. Costs are counted in tokens, small chunks of text of about three-quarters of a word each.

Which one fits which job

Choose RAG when facts change often, must stay private, or need exact citations, because the AI looks them up when the question arrives instead of memorizing them. Choose fine-tuning to change how the model writes, not what it knows, because retraining every time a fact changes is slow and costly. Choose a long context window when the question needs the whole document, not one fact from it.

Three questions decide most cases. How often do the facts change? Does the AI need to learn a new format or style? And how much text does it have to consider at once?

• RAG wins when facts change. If inventory updates hourly, travel policy changes each quarter, or different people may see different documents, RAG is usually the practical choice. Updating it needs no retraining: you add or remove documents in the search index. It can also point to the exact passage behind each answer (Lewis and colleagues, 2020, introduced the approach).
• Fine-tuning wins for style and format. It teaches a model how to write, not facts that change. If you need a particular database query style, strict JSON (a common format for structured data) with nothing around it, or a consistent house tone, fine-tuning works well. A small fine-tuned model can sometimes match a much larger general one on a narrow formatting task, at lower cost, so it is worth testing.
• Long context wins for whole-document reading. Some questions need the entire document, such as finding every place in a long filing where two sections contradict each other. Search over short passages can miss that, because the two ideas sit far apart and use different words. Giving the model the whole document lets it compare them directly. Check the current context limits on your provider’s model page, because they change often.

Why training facts into a model goes stale

Training facts into a model is slow and expensive, and the facts go out of date the day training ends. Keep facts in documents that you hand the model when it answers, and use training for behavior and style.

It is tempting to treat a model like a database and expect that training it on your wiki will make it “know” your company. But training does not store facts the way a database does. It nudges billions of numbers inside the model, so a fact is spread thinly across them rather than kept in one place you can look up or change.

That leads to three weaknesses.

• Facts freeze when training ends. If a project date moves next month, the trained model keeps repeating the old one until you train it again.
• You cannot remove one fact cleanly. If a customer asks you to delete their data, there is no simple way to erase it from a trained model without retraining.
• It sounds sure when it is wrong. When a trained model hits a question it does not really know, it tends to produce a confident, made-up detail in the style it learned, rather than saying it does not know.

A four-step way to decide

Before building anything, work through four steps: how often the facts change, how much text each question needs, what it will cost at your volume, and then which pattern, or mix, to build. Running every project through the same steps beats arguing preferences in a meeting.

• Step 1, how often facts change: If they change hourly, daily, or monthly, rule out relying on fine-tuning for facts and plan on RAG. If the material really never changes, move on.
• Step 2, how much text each question needs: A pinpoint question, such as a deductible amount, suits RAG. A question that needs a whole long document suits a long context window.
• Step 3, cost at your volume: Multiply tokens per question by questions per day by your provider’s price. For example, at a placeholder price of $5 per million input tokens, a 500,000-token prompt costs $2.50, and 10,000 such questions a day costs $25,000 a day. A RAG prompt with a few thousand tokens costs a small fraction of that.
• Step 4, build the right pattern: Real systems often mix them. A small fine-tuned model can sort incoming questions, RAG can answer everyday ones, and a long-context model can handle the rare request to review a whole document.

Cost, speed, and setup effort

A long context window takes almost no setup but costs the most for each question. Fine-tuning costs the most up front but little per question afterward. RAG sits in between.

• Long context: You can build a working prototype quickly, because there is no search index to set up. But you pay for the whole document on every question, and long prompts make answers slower to start.
• Fine-tuning: You must collect and clean many examples, rent computing time, and test the result before shipping. Afterward, prompts can be short because the model already knows the format, and a small model can run on cheaper hardware.
• RAG: Setting up document loading, cutting documents into passages, and a search index takes moderate work. After that, each question sends only a few passages, so it costs far less than sending whole documents, and answers usually start quickly enough for a chat window.

An architecture calculator in Python

The script below scores RAG, fine-tuning, and long context for one project, based on how often the data changes, how long the documents are, how many questions you expect, whether answers must be fast, and whether output must follow a strict format. It also estimates monthly cost. The prices in it are placeholders, so replace them with your provider’s current prices before relying on the numbers.

from dataclasses import dataclass
from typing import Dict, Any

@dataclass
class ProjectRequirements:
    project_name: str
    update_frequency_days: float  # e.g., 1.0 = daily, 30.0 = monthly, 365 = annual
    avg_document_tokens: int       # Total size of knowledge corpus for a query
    monthly_queries: int           # Expected query volume
    latency_critical: bool         # True if requires sub-second chat response
    strict_schema_required: bool   # True if requires deterministic JSON/code output
    global_synthesis_needed: bool  # True if requires whole-doc thematic reasoning

class ArchitectureDecisionEngine:
    def evaluate(self, req: ProjectRequirements) -> Dict[str, Any]:
        rag_score = 0.0
        ft_score = 0.0
        lc_score = 0.0

        # Dimension 1: Knowledge Volatility
        if req.update_frequency_days <= 7.0:
            rag_score += 40.0
            lc_score += 25.0
            ft_score -= 30.0  # Training cannot keep pace with weekly changes
        elif req.update_frequency_days <= 90.0:
            rag_score += 25.0
            lc_score += 20.0
            ft_score += 10.0
        else:
            rag_score += 15.0
            ft_score += 30.0  # Stable domain benefits from weight fine-tuning
            lc_score += 15.0

        # Dimension 2: Reasoning Scope & Document Length
        if req.global_synthesis_needed:
            lc_score += 35.0
            rag_score -= 10.0  # Chunking struggles with whole-document synthesis
        else:
            rag_score += 25.0  # Targeted fact lookup favors RAG

        # Dimension 3: Latency Constraints
        if req.latency_critical:
            rag_score += 20.0
            ft_score += 25.0
            lc_score -= 25.0  # Long context TTFT exceeds user patience

        # Dimension 4: Strict Output Formatting
        if req.strict_schema_required:
            ft_score += 35.0  # Fine-tuning excels at deterministic syntax
            rag_score += 10.0

        # Dimension 5: Economic Unit Cost
        # Rough monthly cost in USD. The per-token prices below are placeholders:
        # replace them with your provider's current prices before trusting the result.
        # RAG: 2K prompt tokens ($0.15/1M) = $0.0003/query
        rag_monthly_cost = req.monthly_queries * (2000 * 0.15 / 1_000_000)
        # LC: Full doc tokens ($3.00/1M average)
        lc_monthly_cost = req.monthly_queries * (req.avg_document_tokens * 3.00 / 1_000_000)
        # FT: Small model ($0.10/1M) + $2,000 upfront amortized
        ft_monthly_cost = (req.monthly_queries * 1000 * 0.10 / 1_000_000) + 200.0

        if lc_monthly_cost > 5000.0:
            lc_score -= 30.0  # Penalize economically non-viable setups

        # Compile Recommendation
        scores = {"RAG": max(0.0, rag_score), "Fine-Tuning": max(0.0, ft_score), "Long Context": max(0.0, lc_score)}
        best_choice = max(scores, key=scores.get)

        if scores["RAG"] > 40 and scores["Fine-Tuning"] > 40:
            recommendation = "Hybrid Composite (Fine-Tuned Classifier + Dynamic RAG)"
        elif scores["RAG"] > 40 and scores["Long Context"] > 40:
            recommendation = "Hybrid Composite (RAG Filter + Long Context Synthesis)"
        else:
            recommendation = f"Pure {best_choice}"

        return {
            "project": req.project_name,
            "recommended": recommendation,
            "scores": scores,
            "monthly_costs": {
                "RAG": f"${rag_monthly_cost:.2f}",
                "Long Context": f"${lc_monthly_cost:.2f}",
                "Fine-Tuning": f"${ft_monthly_cost:.2f}"
            }
        }

# A made-up project: an assistant that answers questions about contracts
example_req = ProjectRequirements(
    project_name="Contract review assistant (example)",
    update_frequency_days=2.0,      # Daily contract amendments
    avg_document_tokens=150_000,    # 100-page complex PDF contracts
    monthly_queries=25_000,         # High enterprise usage
    latency_critical=True,          # Real-time attorney interface
    strict_schema_required=False,
    global_synthesis_needed=False   # Targeted clause extraction
)

engine = ArchitectureDecisionEngine()
audit = engine.evaluate(example_req)

print(f"ARCHITECTURE EVALUATION: {audit['project']}")
print(f"Recommended Paradigm: {audit['recommended']}")
print("Scores:", audit['scores'])
print("Estimated Monthly Token Costs:", audit['monthly_costs'])

For the made-up contract assistant, which gets 25,000 questions a month on 150,000-token contracts that change every couple of days, the script recommends RAG. Sending whole contracts every time would cost about $11,250 a month at the placeholder price, compared with about $7.50 for RAG. The table below shows the result.

Your next step

Put your own project into the calculator above: how often the documents change, how long they are, and how many questions you expect each month, plus your provider’s current prices in place of the placeholders. Run it, and send the recommendation and the monthly cost estimate to whoever owns the budget before anyone starts building.

Sources and further reading

• Lewis and colleagues, 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020, arXiv:2005.11401
• Hu and colleagues, 2021, LoRA: Low-Rank Adaptation of Large Language Models, arXiv:2106.09685
• Liu and colleagues, 2023, Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172
• Anthropic, 2024, Introducing Contextual Retrieval, anthropic.com

More definitions in Key Terms →

Merch on Etsy

Open store →

Apache Hadoop T-Shirt View on Etsy → Git Version Control Shirt View on Etsy → Apache Iceberg T-Shirt View on Etsy → Mean Squared Error Shirt View on Etsy →

Series notes

This is Part 8 of RAG from scratch, and it closes the planned RAG track. Previous: light evals non-engineers can run.