How Does a RAG Chatbot Work?

Ask a plain language model what your refund window is and you will get an answer. It might even be a well-written one. It just will not be your refund window. Closing that gap is the entire reason retrieval-augmented generation exists.

So how does a RAG chatbot work? In four moves: it cuts your documents into chunks, turns every chunk into a vector, searches those vectors the moment a question arrives, and hands the winning chunks to a language model that writes the reply. Retrieval first, generation second. That order is the whole trick. If you want the plain-English definition before the mechanics, read what a RAG chatbot is first and come back here for the pipeline.

Retrieval first, generation second

The architecture is not new. It comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, where Patrick Lewis and colleagues paired a generation model with a searchable index of Wikipedia and showed that the combination beat models relying on training alone.

The useful distinction from that paper is between two kinds of memory. A language model has parametric memory: everything it absorbed during training, baked into its weights, impossible to edit and impossible to audit. A retrieval index is non-parametric memory: a store you own, that you can read, correct, and swap out on a Tuesday afternoon.

That has a practical consequence worth stating early. When your delivery terms change, you do not retrain anything. You update the document, re-index it, and the next answer reflects the change. Nobody touches the model.

Step one: your documents get cut into chunks

Ingestion comes first. Your PDFs, help centre articles, product pages, price lists, contracts and spreadsheet exports are pulled in, stripped of formatting noise, and split into pieces the system can retrieve individually. A whole 40-page policy manual is useless as a unit of retrieval. A single paragraph about returns is exactly the right size.

How big should a chunk be? Anthropic's write-up on contextual retrieval describes the common practice as chunks of no more than a few hundred tokens, which lands at roughly a paragraph or two. The trade-off runs in both directions. Chunk too large and every retrieved piece drags in irrelevant text that dilutes the answer. Chunk too small and you slice a sentence away from the condition that qualifies it, which is how a chatbot ends up promising free delivery that only applies over €100.

The chunking strategies worth knowing:

• Fixed-size with overlap: cut every N tokens, repeating the last 10-20% of the previous chunk. Crude, fast, and a reasonable default.
• Structural: cut on headings, list items, table rows or article boundaries. Best when your source documents are actually structured, which help centres usually are and scanned PDFs usually are not.
• Semantic: cut where the topic shifts, detected by comparing sentence embeddings. Slower to build, better on long prose.
• Contextual: prepend a short model-generated sentence to each chunk explaining what document and section it came from, so the chunk still makes sense in isolation.

This stage decides the ceiling on everything downstream. A chunk that never gets created cannot be retrieved, and an answer that was never in the source cannot be generated. Most of the quality work in a RAG build happens here, long before anyone writes a prompt.

Step two: each chunk becomes a vector

Every chunk then goes through an embedding model, which converts text into a long list of numbers. OpenAI's embeddings guide puts real figures on it: text-embedding-3-small returns 1,536 dimensions by default and text-embedding-3-large returns 3,072. Each of those numbers is a coordinate, and the chunk is a point sitting somewhere in a very high-dimensional space.

What makes this useful is that position encodes meaning. Two chunks about cancelling an order land near each other even when one says "cancel" and the other says "withdraw from the purchase". Keyword search cannot make that connection. Vector search does it by default, which is why a RAG chatbot copes with the way customers actually phrase things rather than the way your documentation phrases things.

Bulgarian is handled well enough by the current multilingual embedding models that we treat it as a solved problem on client projects. A question typed in Bulgarian will retrieve the right chunk from an English source document and vice versa, because the embedding captures meaning rather than the surface form of the words.

Step three: the search that happens before the answer

The vectors live in a vector database, which exists to answer one question quickly: given this point, which stored points sit closest to it? When a customer asks something, their question is embedded with the same model, and the database returns the nearest chunks by cosine similarity.

Common stores are pgvector if you already run Postgres and would rather not add infrastructure, and Qdrant, Pinecone or Weaviate if you want a purpose-built service. At the volumes a typical Bulgarian SME deals with, a few thousand to a few hundred thousand chunks, pgvector is usually enough and keeps your knowledge base inside the database you already back up.

How many chunks should come back? More than instinct suggests. Anthropic's evaluation compared passing the top 5, top 10 and top 20 chunks to the model and found top 20 the most effective of the three. Retrieval is cheap and generation is forgiving, so it pays to over-supply context rather than gamble on the single best match being genuinely best.

Most production systems also run hybrid search: vector similarity for meaning, plus classic BM25 keyword matching for the cases where the exact string matters. Product codes, invoice numbers, model names and Bulgarian street addresses are all things a keyword index catches reliably and an embedding sometimes smooths over.

Step four: the model writes the answer from what it found

Only now does the language model get involved. The retrieved chunks are assembled into a prompt alongside the customer's question and a system instruction that sets the rules: answer only from the supplied context, cite which document each claim came from, and say you do not know when the context does not cover it.

That last rule is the one clients care about most, and it is a configuration decision rather than a property of the model. A bot that admits ignorance and offers a human handover is doing its job. A bot that invents a plausible warranty period is a liability with a chat bubble.

Source citations are worth insisting on even when the interface hides them. During the build they are how you tell a retrieval failure from a generation failure, and after launch they are how a support lead audits a complaint about a wrong answer in about ninety seconds.

Where the pipeline actually breaks

When a RAG chatbot gives a bad answer, the reflex is to blame the model. The failure is usually one stage earlier. If the right chunk never made it into the prompt, no amount of prompt engineering will conjure it back.

Anthropic's published numbers show how much headroom sits in that stage. Their baseline retrieval missed the right chunk 5.7% of the time in the top 20. Adding context to each chunk before embedding cut that to 3.7%. Combining contextual embeddings with contextual BM25 brought it to 2.9%, a 49% reduction. Adding a reranking pass on top took it to 1.9%, a 67% reduction against the same baseline. Same model, same documents, same questions, three times fewer misses.

The failure modes we see most often on real knowledge bases:

• Source documents contradict each other, because the 2023 price list was never deleted. Retrieval faithfully returns both.
• The answer only exists inside a table or an image in a PDF, and the extraction step turned it into unusable text.
• A chunk lost its heading, so "this discount applies to orders over €100" no longer says which discount.
• The question is comparative ("which plan is cheaper for two users?") and the answer requires reasoning across several chunks rather than finding one.
• Nobody set up an evaluation set, so quality is measured by whoever last tried it and formed an impression.

That last one matters more than it sounds. Fifty real customer questions with known correct answers, run after every change, is the difference between a system you can improve and a system you can only hope about.

RAG, fine-tuning, or just a bigger context window?

Three options get proposed for the same problem, and they solve different things. Fine-tuning adjusts a model's weights on your examples. It is good at teaching tone, output format and domain vocabulary, and poor at installing facts, because a fine-tuned fact is as unauditable and as hard to update as any other weight. When your prices change, you would be retraining.

Pasting everything into a long context window is the other tempting shortcut. It works for a 30-page handbook and stops working for a 3,000-page one. You pay for every token on every question, latency rises with the input, and recall degrades as the relevant sentence gets buried among tens of thousands of irrelevant ones.

RAG sits between them: your facts stay in a store you control, only the relevant slice reaches the model on any given question, and updating the knowledge base is a file operation rather than a training run. In practice the three combine well. RAG supplies the facts, a short prompt or a light fine-tune supplies the voice.

What this looks like on a real project

For Houzez, Bulgaria's property platform, we built Boro, a bilingual assistant that takes a plain-language brief such as "two bedrooms in Lozenets under €250,000" and returns matching listings, while answering the process questions that used to arrive through the contact form at eleven at night. The retrieval layer sits over listing data and the site's own guidance content; the model handles phrasing, in Bulgarian or English, whichever the visitor started in.

The shape of that work is typical. Roughly two thirds of the effort goes into ingestion, chunking, retrieval quality and evaluation. The chat interface, the part everyone pictures when they say chatbot, is the small end of the job. Our RAG chatbot builds start with a two-week proof of concept against your actual documents so you can see measured answer quality before committing to a production system, and the pricing page sets out how that scopes.

If your problem is less about answering questions and more about moving data between systems without a person retyping it, that is a different tool. Look at AI automation instead, and skip the retrieval layer entirely.

The verdict

A RAG chatbot works by searching before it speaks. Chunk, embed, retrieve, generate. Nothing in that pipeline is exotic, and every stage is inspectable, which is precisely why it belongs in a business context where a wrong answer costs something.

Our honest position after building these: the model is the least interesting decision you will make. Whichever frontier model you pick will write a competent paragraph from good context. What separates a chatbot people trust from one they stop using is unglamorous work on the documents, the chunking and the retrieval, plus an evaluation set that tells you whether last week's change helped. If a supplier wants to talk mostly about which model they use, they are showing you the easy part.

Frequently asked questions

How does a RAG chatbot work in one sentence?

It converts your documents into searchable vectors, finds the passages closest in meaning to the incoming question, and asks a language model to answer using only those passages.

Does a RAG chatbot still hallucinate?

Less, but not never. Grounding the answer in retrieved text removes most invented facts. What remains is usually a retrieval problem: the right passage was not found, and the model filled the silence. Source citations and an instruction to refuse when the context is thin bring the residue down further.

How much data do I need before RAG makes sense?

Enough that a person cannot hold it in their head, and consistent enough to be worth searching. Twenty FAQ entries do not need retrieval; a well-written FAQ page will beat a chatbot. A few hundred pages of policies, product detail and procedures is where RAG starts earning its keep.

Does it work in Bulgarian?

Yes, in both directions. Current multilingual embedding models handle Bulgarian well, and the major providers generate fluent Bulgarian. A visitor can ask in Bulgarian and get a correct answer from an English source document. The real constraint is document quality, not language.

How long does a RAG chatbot take to build?

Two weeks for a proof of concept against your real documents, then typically four to eight more for production, depending on how many systems it has to read from and how clean the source material is. Messy PDFs extend the first number more than anything else.

What does it cost to run each month?

Two lines: model usage, billed per token and driven by conversation volume, and hosting for the vector store, which is close to zero if you use pgvector inside an existing database. At moderate query volumes most of our clients sit in the low hundreds of euros a month.

Can it read from our CRM or ERP rather than documents?

It can, and often should. Live records are fetched through an API at question time rather than embedded, since stock levels and order statuses change faster than any index refreshes. A production assistant usually mixes both: retrieval for stable knowledge, direct lookups for anything current.