Retrieval-augmented generation (RAG) and fine-tuning solve different problems. RAG gives a model the right information at the moment it answers, so use it when answers must come from your own documents, change often or need a source. Fine-tuning changes how the model behaves, so use it for a consistent format, tone or narrow task. Most projects need RAG first, and some later add fine-tuning.
Key takeaways
- RAG changes what the model knows when it answers; fine-tuning changes how it behaves.
- If your content changes weekly, RAG wins: you update the documents, not the model.
- If you need citations, RAG wins: every answer can point to the passage it used.
- Fine-tuning earns its cost when a prompt can’t hold the behavior you need, when volume makes a smaller model cheaper, or when the model must run on your own servers.
- Measure both against the same evaluation set before you decide (how to evaluate an LLM application).
What RAG is
RAG was described by Lewis and colleagues in 2020 as a way to combine a language model with a search over an external collection of documents. In practice it works in three steps:
- Prepare your content: split documents into passages, add metadata such as source, date and who may see it, and index them for search.
- Retrieve: when a user asks a question, find the passages most likely to contain the answer, often with a mix of keyword and vector (meaning-based) search.
- Generate: give those passages to the model with the question, and ask it to answer only from them and to cite them.
The model itself is unchanged. What changes is the context it reads before it answers. That is why RAG suits company knowledge: policies, product documentation, contracts, tickets and anything else that is updated more often than you would retrain a model.
What fine-tuning is
Fine-tuning continues training a model on your own examples, so the new behavior is stored in its weights. You show it hundreds or thousands of input and output pairs, and it learns to produce outputs like yours. Parameter-efficient methods such as LoRA (Hu and colleagues, 2021) train a small set of extra weights instead of the whole model, which makes fine-tuning far cheaper than it used to be.
Fine-tuning is good at teaching form: a report structure, a classification scheme, your house style, a specialized vocabulary. It is a poor way to teach facts that change, because every change means new training data, a new training run and a new round of testing.
RAG vs fine-tuning at a glance
| Question | RAG | Fine-tuning |
|---|---|---|
| What changes | The context the model reads | The model’s weights |
| Best for | Answers from your documents and data | Consistent format, tone or a narrow task |
| Keeping content current | Update the index | Retrain and retest |
| Citations | Built in: answers point to passages | Not available from the model itself |
| Access control | Per user, at retrieval time | Not possible inside the weights |
| Main cost | Preparing content, running retrieval, longer prompts | Curating training data, training runs, upkeep |
| Typical uses | Knowledge assistants, support, internal search | Structured output, classification at volume, private small models |
When to choose each
Choose RAG when:
- answers must follow your current policies, prices, products or records;
- users need to see where an answer came from;
- different users may see different documents;
- your content changes faster than you would retrain a model.
Choose fine-tuning when:
- a well-written prompt with examples still gives inconsistent structure or tone;
- you run the same narrow task at a volume where a smaller, tuned model costs less per call than a large general one;
- the model has to run inside your network, and a small tuned open-source model fits on your hardware;
- you have, or can build, a reviewed set of examples that shows the behavior you want.
Using both together
The two combine well. A common pattern is a RAG system whose model has been fine-tuned to follow your answer format, to refuse when the retrieved passages don’t contain the answer, or to work better in a language the base model handles less well. Fine-tuning the retrieval side (the embedding model or a reranker) on your own queries is another option when search quality is the weak point.
A worked example
Example, for illustration. An insurer wants an assistant that answers agents’ questions about policy cover.
- The wordings change every year and differ by product, so the facts belong in RAG: index the current policy documents with product and version metadata, retrieve the right clauses, and show them with the answer.
- Agents complain that answers ramble. A prompt that asks for "a one-line answer, then the clause" works for most questions. If it still drifts, a small fine-tune on a few hundred reviewed question, answer and clause examples can fix the format.
- Both versions are scored on the same evaluation set: real agent questions with the correct clause marked by an underwriter. The team ships fine-tuning only if it improves that score enough to justify its upkeep.
Common mistakes
- Fine-tuning to add knowledge. The model may sound confident about facts it learned once and never relearned. Put changing facts in retrieval.
- Blaming the model for bad retrieval. If the right passage isn’t retrieved, no model will answer well. Check retrieval first.
- Skipping the baseline. Without a prompt-and-retrieval baseline scored on real questions, you can’t tell whether fine-tuning helped.
- No plan for upkeep. A fine-tuned model is a new asset: it needs versioning, re-evaluation and a reason to exist each time the base model improves.
How KaliIO approaches it
On a new project we build the retrieval baseline first, because it is faster to change and easier to inspect. We write an evaluation set with your experts, score the baseline, and only then decide whether a fine-tune is worth its data, training and upkeep. Our RAG development services cover the content preparation, retrieval and citations; LLM fine-tuning services cover training, distillation and the before-and-after evaluation. For the wider build, see LLM application development.
Frequently asked questions
Is RAG cheaper than fine-tuning?
Usually at the start. RAG needs content preparation and a search index, but no training runs. Its running cost is higher per question, because each prompt carries the retrieved passages. Fine-tuning costs more up front and can cost less per call when a smaller tuned model replaces a larger one at high volume.
Can fine-tuning stop hallucinations?
Not on its own. Fine-tuning can teach a model to say "I don’t know" more often, but it can’t make the model aware of facts it was never shown. Grounding answers in retrieved passages, and checking that the answer is supported by them, does more.
How much data do I need to fine-tune?
It depends on the task and the model. A narrow formatting or classification task can improve with a few hundred reviewed examples; teaching broader behavior takes more. Quality matters more than volume: inconsistent examples teach inconsistent behavior.
Should I fine-tune the embedding model instead?
Sometimes. If users ask in words that differ from the wording in your documents, a tuned embedding model or a reranker can improve retrieval more than any change to the answering model.
Sources
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
- Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
- Es, S. et al. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation.