Why AI document search gives wrong answers, and how to find the cause

When a document-search AI gives a wrong answer, find out which step failed before you change anything. Did it find an outdated document, find the wrong document, or find the right one and ignore it? Each cause has a different fix, and blaming “the AI” in general fixes none of them.

Imagine you ask your company’s assistant how to store a delicate product, and it answers from a procedure that was replaced last year. The search worked in the sense that it found a document that matched your words. It simply had no idea the document was out of date. Dates were never part of the search. Tuning the AI’s instructions would not have helped, because the problem was in what it was handed.

This post is part of the RAG from scratch series. RAG, short for retrieval-augmented generation, is a setup where an AI first looks up passages in your own documents and then writes an answer from them. The passages it looks up are called chunks.

The four common ways RAG goes wrong

RAG systems usually fail in one of four ways. They find outdated documents, or look-alike documents from the wrong part of the business. Or the AI ignores the right passage, or misses a key fact buried in the middle of a long prompt. A demo on a small, clean set of documents rarely shows these problems. Thousands of real files, chat threads, and wiki pages do.

• Outdated documents: Search ranks chunks by how close their meaning is to the question, and it has no sense of dates. An old travel policy and its replacement can look almost identical to the search, and the old one may even match the question’s wording more closely. Unless you filter by date or status, the outdated version can end up in the answer.
• Look-alike documents from the wrong area: Search compares embeddings, lists of numbers that capture what a passage is about. Passages from very different parts of a company can land close together if they share words. Think of an employee leave policy and a customer return policy that both talk about “replacement” and “eligibility.” The wrong one looks relevant to the search but misleads the AI.
• The AI ignores the right passage: Even when the correct chunk is in the prompt, the AI can contradict it. This happens most when the passage goes against what the model learned in training. Rules with words like “not,” “never,” or “except” are worth testing on purpose, because a dropped negation flips the meaning.
• Facts lost in the middle: Researchers at Stanford and elsewhere tested this. Language models used information near the start or end of a long prompt better than information in the middle (Liu et al., “Lost in the Middle,” 2023). If the one rule that matters is the fifth of ten passages, the AI is more likely to miss it.

Plain similarity search versus filtered, reranked search

Plain similarity search trusts the closest match. A sturdier setup filters out outdated documents first and mixes word matching with meaning matching. Then it re-scores the shortlist with a more careful model before anything reaches the AI.

Similarity search answers one narrow question: which chunks are closest in meaning to the question? It cannot tell whether a document was replaced yesterday, who wrote it, or whether it contradicts your current rules. Those checks have to come from somewhere else. Search alone cannot tell time.

Many production systems use two stages. First, a fast search pulls a shortlist of candidates, for example fifty, using only chunks whose tags say they are current, such as status == 'active'. Then a reranker re-scores each candidate. A reranker is a slower, more careful model that reads the question and the passage together, instead of comparing two separately computed lists of numbers. That makes it better at noticing when a passage shares words with the question but means something different (Nogueira and Cho, 2019).

A four-step triage routine

When a wrong answer is reported, work through four steps: notice it, find which stage failed, fix that stage, and add a test so it cannot come back quietly. Without a routine, teams tend to tinker at random, changing chunk sizes, swapping models, or adding long warnings to the instructions.

• Notice the problem: Record signals as people use the system: thumbs-down votes, answers whose sources nobody opens, and scores from automated checks. Also log questions where even the best match scored low, because those often lead to weak answers.
• Find which stage failed: Look at the chunks the search actually returned and ask one question: was the correct chunk there at all? If it was missing, search failed, usually because of how documents were cut up, missing tags, or a gap in what the embeddings capture. If it was there and the AI ignored or contradicted it, the writing step failed.
• Fix that stage: For search failures, add date or status filters, keep different business areas apart, or adjust how much weight word matching gets compared with meaning matching. For writing failures, put the most important passages at the start or end of the prompt, give fewer passages, or require the AI to quote its sources.
• Add a test that catches the bug if it returns: Turn the failed question, the correct chunk, and the expected answer into a permanent test. Run your tests before every model change or rebuild of the search index, so the same mistake cannot return without anyone noticing. Then close the ticket.

Three layers of defense

Sturdy systems put a check at each of three points: when documents are loaded, when they are searched, and after the answer is written. Rewording the AI’s instructions alone cannot fix problems in what it is given. Bad inputs stay bad.

• When loading documents: Tag every chunk with the dates it is valid. When someone uploads a new version, mark the old one as replaced automatically. Remove near-duplicate passages so they do not crowd out better ones.
• When searching: Filter out replaced documents before the search runs. Combine meaning-based search with plain keyword search, which is better at exact part numbers and policy codes. Rerank the shortlist, and set a minimum score: if nothing clears it, have the system say it found nothing instead of passing weak matches to the AI.
• After writing: Tell the AI plainly that it may decline. For example: “If the passages do not contain enough to answer, say you cannot answer from the available documents.” Then check that each claim matches its source, using the citation checker from the previous post in this series.

A triage checker in Python

The script below scores each search candidate the way a triage step might. It rejects anything marked as replaced and lowers the score of older documents. It also penalizes candidates from a different business area and rejects anything below a minimum score. The documents are a small made-up example, and the penalty values are illustrations you would tune on your own data.

import math
from dataclasses import dataclass
from typing import List, Dict, Any

@dataclass
class RawChunkCandidate:
    chunk_id: str
    doc_title: str
    domain: str
    publish_year: int
    is_deprecated: bool
    raw_cosine_sim: float
    text_snippet: str

@dataclass
class TriagedCandidate:
    chunk_id: str
    doc_title: str
    domain: str
    effective_score: float
    raw_similarity: float
    temporal_penalty: float
    domain_mismatch: bool
    action: str
    reason: str

class RAGDiagnosticTriageEngine:
    def __init__(self, target_domain: str, current_year: int = 2026, min_score_floor: float = 0.70):
        self.target_domain = target_domain
        self.current_year = current_year
        self.min_score_floor = min_score_floor

    def evaluate_candidate(self, c: RawChunkCandidate) -> TriagedCandidate:
        # Check hard deprecation flag
        if c.is_deprecated:
            return TriagedCandidate(
                chunk_id=c.chunk_id,
                doc_title=c.doc_title,
                domain=c.domain,
                effective_score=0.0,
                raw_similarity=c.raw_cosine_sim,
                temporal_penalty=1.0,
                domain_mismatch=(c.domain != self.target_domain),
                action="REJECT_STALE",
                reason="Document is explicitly marked deprecated by ingestion catalog"
            )

        # Calculate temporal decay for unflagged older documents
        years_old = max(0, self.current_year - c.publish_year)
        # Exponential temporal decay: 15% penalty per elapsed year
        decay_factor = math.pow(0.85, years_old)
        temporal_penalty = round(1.0 - decay_factor, 3)

        # Check domain mismatch (Wrong Neighbor Drift)
        domain_mismatch = (c.domain != self.target_domain)
        domain_penalty = 0.35 if domain_mismatch else 0.0

        # Calculate effective calibrated score
        adjusted_score = c.raw_cosine_sim * decay_factor - domain_penalty
        adjusted_score = max(0.0, round(adjusted_score, 3))

        # Determine triage verdict
        if domain_mismatch and adjusted_score < self.min_score_floor:
            action = "REJECT_WRONG_NEIGHBOR"
            reason = f"Cross-domain drift from '{c.domain}' failed confidence floor"
        elif adjusted_score < self.min_score_floor:
            action = "REJECT_LOW_CONFIDENCE"
            reason = f"Calibrated score {adjusted_score:.2f} below threshold {self.min_score_floor:.2f}"
        else:
            action = "ACCEPT_FOR_RERANK"
            reason = "High confidence candidate approved for prompt construction"

        return TriagedCandidate(
            chunk_id=c.chunk_id,
            doc_title=c.doc_title,
            domain=c.domain,
            effective_score=adjusted_score,
            raw_similarity=c.raw_cosine_sim,
            temporal_penalty=temporal_penalty,
            domain_mismatch=domain_mismatch,
            action=action,
            reason=reason
        )

# Made-up candidates for a question about warehouse storage rules
candidates = [
    RawChunkCandidate(
        chunk_id="chk_wh_2026",
        doc_title="Facility Reconfiguration Handbook (2026)",
        domain="warehouse_logistics",
        publish_year=2026,
        is_deprecated=False,
        raw_cosine_sim=0.91,
        text_snippet="Bay 14 is permanently designated for hazardous chemical quarantine."
    ),
    RawChunkCandidate(
        chunk_id="chk_wh_2024",
        doc_title="Warehouse Storage Matrix (2024)",
        domain="warehouse_logistics",
        publish_year=2024,
        is_deprecated=True,
        raw_cosine_sim=0.95,
        text_snippet="Bay 14 is open staging for high volume transfer overflow."
    ),
    RawChunkCandidate(
        chunk_id="chk_chem_clean",
        doc_title="Industrial Cleaning Supplier Terms",
        domain="procurement_contracts",
        publish_year=2025,
        is_deprecated=False,
        raw_cosine_sim=0.88,
        text_snippet="Immediate clearance granted for routine entry of cleaning crews."
    ),
    RawChunkCandidate(
        chunk_id="chk_pallet_specs",
        doc_title="Pallet Packaging Standards",
        domain="packaging_guidelines",
        publish_year=2023,
        is_deprecated=False,
        raw_cosine_sim=0.74,
        text_snippet="Wooden pallets must meet heat treated ISPM 15 standards."
    )
]

triage_engine = RAGDiagnosticTriageEngine(target_domain="warehouse_logistics")
results = [triage_engine.evaluate_candidate(c) for c in candidates]

print("RAG RETRIEVAL CANDIDATE TRIAGE REPORT:")
for r in results:
    print(f"[{r.action}] {r.doc_title}")
    print(f"  Raw Sim: {r.raw_similarity:.2f} -> Calibrated Score: {r.effective_score:.2f}")
    print(f"  Reason: {r.reason}\n")

When you run it, only the current warehouse handbook passes. The 2024 storage document is rejected because it is marked as replaced, even though it had the highest similarity. The cleaning-supplier contract and the packaging standards come from other business areas and fall below the minimum score. The table below shows the result.

Key takeaways

• A high similarity score means a passage is about the same topic, not that it is current or correct.
• Look-alike documents from other parts of the business can crowd out the right one, so keep business areas apart and rerank the shortlist.
• Filter out replaced documents and set a minimum score, so weak or outdated matches never reach the AI.
• Every wrong answer you fix should become a permanent test.

The next post in this series covers simple ways for non-engineers to check RAG quality: whether answers stick to the sources, whether the right passages were found, and whether anything important was missed.

Your next step

Take one wrong answer your AI document search gave recently and run it through the four-step triage routine in this post. Save the question and the passage that should have been found as a test, and run the checker against it after every change, so the same mistake cannot come back unnoticed.

Sources and further reading

• Liu and colleagues, 2024, Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics, 12, 157-173, arXiv:2307.03172.
• Karpukhin and colleagues, 2020, Dense Passage Retrieval for Open-Domain Question Answering, EMNLP 2020, arXiv:2004.04906.
• Nogueira and Cho, 2019, Passage Re-ranking with BERT, arXiv:1901.04085.
• Barnett and colleagues, 2024, Seven Failure Points When Engineering a Retrieval Augmented Generation System, CAIN 2024, arXiv:2401.05856.

More definitions in Key Terms →

Merch on Etsy

Open store →

Apache Hadoop T-Shirt View on Etsy → Naive Bayes T-Shirt View on Etsy → Python Coder Shirt View on Etsy → Apache Flink Shirt View on Etsy →

Series notes

This is Part 6 of RAG from scratch. Previous: citations. Next: light evals non-engineers can run.