RAG 2020: How to Give Language Models an Answer Key

Language models do not store an encyclopedia in their heads. They store patterns — patterns of which words usually follow which others. This makes them fluent, but not factual. The question "who was the president of Indonesia in 1967" gets answered the same way as writing poetry: guessing the most likely next word.

Retrieval-Augmented Generation, or RAG, addresses this honestly. Instead of forcing the model to memorize all facts, RAG lets the model look things up first. "Retrieval" means fetching. The source can be documents, a database, or an entire website. The model receives relevant snippets, reads them, and then answers.

2020: The Paper That Changed How Models Are Used

In 2020, Patrick Lewis and his team at Facebook AI published "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." The idea inside is simple: separate knowledge stored inside the model (parametric knowledge) from knowledge stored outside it (non-parametric knowledge).

Parametric knowledge is what the model learned during training. It is stored in the network weights. Non-parametric knowledge is documents that can be accessed anytime, updated anytime, without retraining the model. RAG combines both.

The paper also introduced two variants: RAG-Sequence and RAG-Token. RAG-Sequence takes one document and generates the entire answer. RAG-Token takes several documents and generates the answer token by token, with different weights for each document.

Chunking: Cutting Into Pieces

The first step of RAG is cutting documents into smaller pieces, or "chunks." Documents that are too long cannot be used directly. The embedding model has a token limit, and chunks that are too large reduce search quality.

A 3,000-word article might be cut into 15 chunks of 200 words each. Common chunk sizes range from 200 to 500 tokens. Too small, and the chunk loses context. Too large, and search becomes less precise.

Embedding: Turning Text Into Numbers

Each chunk is converted into a vector — a list of numbers that represents its meaning. The embedding model is trained so that texts with similar meanings have vectors at similar distances. "President Soeharto" and "Indonesian leader in 1967" will have nearby vectors, even though the words differ.

These vectors are stored in a vector database, which is optimized for finding the vector closest to the question vector. This search is not ordinary keyword search. It searches for meaning, not letters.

Retrieval and Reranking: Finding What Matters

When a question arrives, it too becomes a vector. The vector database finds the chunks whose vectors are closest. Usually the top five to ten are taken.

But the top ten are not always the most useful. Sometimes a relevant chunk ranks eighth because its vector is slightly farther. This is why many RAG systems add a reranking stage — a second model that reassesses which chunks actually answer the question.

Analysis: RAG Reduces Hallucination, Does Not Remove It

The main advantage of RAG is that the model has a verifiable source. If asked about a specific fact, the model can quote the relevant chunk. If that chunk is not there, the model can answer "I could not find that information."

But RAG is no guarantee. There are three ways it can still fail. First, the documents are missing or irrelevant. Second, a relevant document exists but was not among the retrieved chunks. Third, the model misreads the document, or ignores it entirely.

The third failure is the most common and the most dangerous. The model receives a document that contains the answer, but the answer is buried in the middle of a long paragraph. The model skims, misses it, and answers from its own parametric knowledge. The user sees a confident answer that sounds right, and has no way to know it was not grounded in the document at all.

This is why reranking matters. A good reranking step does not just score relevance. It checks whether the chunk actually contains information that could answer the question. A chunk that mentions the topic but does not answer the question is worse than useless — it gives the model permission to bluff.

The table below makes the three failure modes concrete. Each one has a different cause and a different fix. Treating them as one problem is how RAG systems fail in production.

The first two failures are engineering problems. They can be fixed with better chunking, better embeddings, and better reranking. The third failure is a model problem. It gets better with bigger models and better training, but it never goes away. That is the honest limit of RAG.

Why RAG Is Cheaper Than Fine-Tuning

Fine-tuning retrains the model with new data. It is expensive, requires hours of GPU time, and every time the data changes, the process must be repeated. RAG does not touch the model at all. The data lives in the vector database and can be updated anytime.

The comparison is like this: fine-tuning is hiring and training a new employee, RAG is giving an employee a handbook. When the handbook changes, replace the book. No need to retrain the employee.

For small teams in Indonesia, this difference matters. One server with 16 GB of RAM can run a vector database for millions of documents. A GPU for fine-tuning large models can cost hundreds of millions of rupiah.

Honest Limits

RAG is not a magic solution. It adds latency because of the extra retrieval step. It adds cost because vector databases need storage and memory. And it does not guarantee the model will not hallucinate — it only gives it the opportunity not to.

The most common question is: if the model is already large, is RAG still necessary? The answer is yes. Because model size does not determine whether a model knows recent facts. A model trained until 2024 does not know what happened in 2026, no matter how large it is.

RAG 2020, from the start, is an admission that language models are not encyclopedias. They are very skilled translators. They need sources, and RAG is the way to provide them.

Real-World Example: RAG in Indonesia

Imagine a company in Jakarta has 500 pages of internal documents — contracts, procedures, and reports. New employees often ask about specific procedures. Without RAG, they must read all documents or ask seniors.

With RAG, those documents are turned into vectors and stored. When an employee asks "how is the travel reimbursement procedure?", the system finds the relevant snippet, puts it into context, and the model answers by quoting that document.

This is not hypothetical. Many companies in Indonesia already use RAG for customer support, internal document search, and virtual assistants. The technology is mature, affordable, and measurable.

Practical Implementation: From Zero to a Working System

Building a RAG system from scratch is not as complex as it sounds. Its components are available as open-source software. The first step is choosing a vector database. Common choices include Chroma, Qdrant, and FAISS. All three can run on a single server without a GPU.

The second step is choosing an embedding model. For Indonesian, a multilingual model like multilingual-e5-base works well. For better results, a model trained specifically for Indonesian can be used. Embedding model sizes are generally small, under 1 GB.

The third step is building the pipeline. Documents enter, get cut into chunks, turn into vectors, and are stored. When a question arrives, find the nearest vectors, take the snippets, put them into context, and the model answers. This pipeline can be built with pure Python in a few hundred lines of code.

The real challenge is not the technique, but the data. Messy documents produce messy vectors. Unstructured documents are hard to cut correctly. This is why data preparation often takes longer than building the system itself.

The Future of RAG

RAG continues to evolve. Newer versions add the ability to search multiple sources at once, combine results from vector databases and traditional keyword search, and even ask the model to search again if the first results are not satisfactory.

There is also an approach called "agentic RAG," where the model not only retrieves documents but also decides what to search, how to search, and whether the results are sufficient. This brings RAG closer to how humans search for information: try, evaluate, try again.

What does not change is the core idea. Language models are not encyclopedias. They still need sources. And RAG remains the most practical way to provide them.

Why This Matters for Users in Indonesia

For API users in Indonesia, RAG can significantly reduce costs. Instead of sending long documents every time they ask a question, RAG sends only the relevant snippets. That means fewer tokens processed and lower bills.

For developers in Indonesia, RAG opens the possibility of building AI applications connected to local data — company documents, government regulations, or educational content — without training models from scratch. That is no longer a luxury, but a practical necessity.