Three approaches get compared as if they were alternatives to the same problem. They are not. They address different failure modes, and picking between them is mostly a question of what kind of wrong your system currently is.
Start by naming the failure
If the model does not know something, that is a knowledge problem. Retrieval fixes it. If the model knows but will not behave the way you need, consistently, in the right format, that is a behaviour problem. Prompting fixes some of it and fine-tuning fixes the rest.
Most teams reach for fine-tuning on a knowledge problem. It appears to work, because the model can now recite facts from the training set, and then the facts change and the model keeps confidently reciting the old ones. There is no document to update, because the information is in the weights.
Prompting, and why it is underrated
Before anything is built, establish what a well-constructed prompt achieves on your actual task, measured against real examples. Few-shot examples, explicit output schemas and clear failure instructions resolve a surprising share of what gets scoped as a fine-tuning project.
It is worth doing properly because it is the cheapest thing to change. Prompts are edited in minutes. A fine-tune is a training run and a redeployment. If prompting gets you to acceptable accuracy, you have avoided an entire category of operational cost.
It stops being enough when you need output to be identical every time, when the instruction has grown so long it costs more per call than the alternative, or when a smaller and cheaper model simply cannot follow it.
RAG, for anything that changes
Retrieval-augmented generation puts the relevant documents in front of the model at the moment of the question. The knowledge stays in a document store you control.
Use it when the answer depends on information that moves, when different users should see different data, when answers must cite a source, or when someone may need a document removed and forgotten.
That last point matters more than it sounds. If a customer asks you to delete their data, a document store makes that a delete operation. A fine-tuned model makes it a retraining project.
Its costs are structural rather than fixed: storing and indexing the documents, re-embedding when they change, inference per query, and observability so you can tell why an answer was wrong. All of it scales with use, which is an advantage at low volume and a liability at very high volume.
Its hard part is not the model. It is retrieval quality. If the right document does not surface, no model will rescue the answer, and most disappointing RAG systems are disappointing for that reason rather than anything to do with the model.
Fine-tuning, for behaviour
Fine-tuning adjusts the model’s weights on your examples. It is genuinely the right answer for two things.
Locking in form. A specific output structure, house tone, or domain vocabulary that prompting cannot hold reliably across thousands of calls.
Making a smaller model competitive. Distilling the behaviour of a large expensive model into a small cheap one for a narrow task. At sufficient volume this is where the real savings are, and the driver is inference cost and latency rather than accuracy.
It needs a few hundred to a few thousand good examples, which in practice means someone has to produce them. Budget that work. It is usually the largest line item and it is almost always underestimated.
The training run is rarely the expensive part. Keeping a fine-tuned model deployed and available carries a standing hourly cost whether or not anyone uses it. Before choosing this route, work out the monthly bill at zero traffic. For low-volume internal tools that figure alone often settles the decision.
Choosing
The hybrid most production systems end up as
Mature systems usually combine them: a smaller model fine-tuned for format and domain vocabulary, placed behind a retrieval pipeline that supplies the facts. Behaviour comes from the weights, knowledge comes from documents, and each can be changed without disturbing the other.
That is a sensible destination and a poor starting point. Build the retrieval path first and measure it. Add fine-tuning when you can point at a specific behaviour that prompting will not hold, or a specific inference bill that a smaller model would reduce. Doing both at once means that when quality is disappointing, you cannot tell which half is responsible.
How to decide without guessing
Assemble fifty to a hundred real cases with known correct answers. Run prompting alone. Measure. Add retrieval. Measure again. Only if a specific gap remains, and you can name it, consider fine-tuning.
That sequence takes days rather than weeks and it produces a number rather than an opinion. It is also the order that keeps your options open: prompting and retrieval can be revised continuously, while a fine-tune is a commitment you carry.
More on how we scope this work on our AI and machine learning page, and on deciding whether a problem suits AI at all in where AI actually pays off.
If you have a use case and are not sure which of the three it needs, describe what the system has to do.
Common questions
Should we use RAG or fine-tuning?
RAG if the answer depends on information that changes, needs to cite a source, or may need to be deleted. Fine-tuning if the model already knows enough but will not behave consistently, or if you need a smaller cheaper model to handle a narrow repetitive task. Knowledge problems are retrieval problems. Behaviour problems are tuning problems.
Is fine-tuning cheaper than RAG?
It depends on volume. A hosted fine-tuned model carries a standing cost whether or not anyone uses it, while RAG costs scale with queries. At low volume RAG is usually cheaper. At high volume a fine-tuned smaller model often wins on inference cost. Work out both at your real traffic before choosing.
Can we skip prompting and go straight to fine-tuning?
You can, and it usually wastes money. Prompting is the cheapest thing to change and resolves a large share of what gets scoped as fine-tuning work. Establish what a well-built prompt achieves on real examples first, so you know what the fine-tune actually has to add.
Note on sources
The decision framework here reflects established practice rather than a single study, and the cost characteristics are described structurally on purpose. Published per-token and per-hour figures move quickly and vary by provider and region, so precise numbers would be out of date before they were useful. Price your own case against current provider rates at your real volume.
Written September 2026.
