Retrieval-Augmented Generation(RAG)

How it works

Retrieval-augmented generation splits answering into two jobs. First, a retrieval step looks for documents that match the question: a web search, a search over a company's internal files, or a lookup in a prebuilt index. Second, a language model reads what came back and writes the answer, drawing on those documents instead of relying on memory alone.

The useful distinction is between two kinds of knowledge. Parametric knowledge is what a model learned in training and stores in its weights. It is broad, but it stops at a cutoff date and the model cannot point to where a given fact came from. Retrieved knowledge is fetched at answer time, so it can be current, and because the system knows which documents it used, it can show them. Answers tied to retrieved sources are called grounded, and grounding is what produces the citations you see in AI search.

Where it came from

The term comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel and Douwe Kiela. It was first posted to arXiv on May 22, 2020 and accepted at NeurIPS 2020.

The authors framed the problem in terms that still apply. Large models store facts in their parameters but struggle to access them precisely, and "providing provenance for their decisions and updating their world knowledge remain open research problems." Their answer paired a generator model (parametric memory) with a searchable index of Wikipedia (non-parametric memory). They split each Wikipedia article into 100-word chunks, about 21 million passages in all, and retrieved the most relevant ones for each question. They also showed that swapping in a newer Wikipedia index updated the model's answers without retraining it. That property is the reason the approach spread to products that need fresh information.

How AI search engines apply it

AI search products run the same pattern with a live web index in place of a fixed Wikipedia snapshot. Google's documentation for Grounding with Google Search describes the flow in plain steps: the model decides whether a search would help, generates and runs one or more queries, synthesizes the results, and returns an answer whose text spans are linked to source URLs. Google says this lets the model cite verifiable sources beyond its knowledge cutoff and reduces hallucinations.

For AI Overviews and AI Mode, Google says the system may use query fan-out, issuing multiple related searches across subtopics and data sources, and that a page must be indexed and eligible to show with a snippet to appear as a supporting link (Google Search Central). On the OpenAI side, the company says OAI-SearchBot is used to surface websites in ChatGPT's search features.

Retrieval systems usually work with passages, a step called chunking. The original paper used 100-word chunks. Commercial AI search providers do not publish how they split pages, so the practical takeaway is general: a section that makes sense on its own is easier to retrieve and quote than a point that depends on context three screens up.

Why it matters for B2B and consumer brands

RAG is the part of AI search a brand can act on this quarter. What a model learned in training changes only when it is retrained. What it retrieves changes whenever the web changes. If an answer engine searches before it answers, your pages, reviews, documentation and third-party coverage can shape the answer today, and the citation links give buyers a path back to the source.

It also explains why AI answers vary. Retrieval depends on which queries the system generates, what the index returns and which passages the model chooses to use, so the same question can produce different sources on different days. Our guide to where LLMs get their information covers the training-data side of the same picture.

What to do about it

1. Be retrievable. Allow the search crawlers that feed answer engines. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that blocking GPTBot (training) is a separate choice. Make sure important pages are indexed in Google and Bing.
2. Be crawlable. Serve key content and links as plain HTML. Our server-log test of AI crawlers and JavaScript links found that six crawlers, including GPTBot, ClaudeBot, PerplexityBot and Bingbot, discovered none of 293 pages reachable only through JavaScript-injected links (a single site with sitemaps deliberately disabled).
3. Be specific. Write sections that answer one question each, with names, numbers and dates stated in the passage itself, so a retrieved chunk carries its own context.
4. Be corroborated. Answer engines pull from many sources per question. Accurate, consistent descriptions of your company on review sites, industry publications and partner pages give retrieval more than your own site to work with.

Common misconceptions

"RAG means the model trusts whatever it retrieves." The model still decides what to use and how to phrase it. Retrieval narrows the material; it does not guarantee a page is cited or quoted accurately.

"There is a special RAG optimization." Google states there are no additional requirements or special optimizations needed for AI Overviews or AI Mode. The work is crawl access, indexing and clear content.

"llms.txt feeds retrieval." llms.txt is a proposal with limited, uneven adoption, and no major AI search provider has confirmed using it for ranking or citation.

Because retrieval shifts from query to query, the useful measure is how often your brand appears and is cited across the questions buyers actually ask, tracked over time and across engines.