"We need to fine-tune our own model." It is one of the most common requests in AI strategy conversations, and most of the time the right answer is: probably not yet. A strong hosted model, a careful RAG pipeline and a well-built prompt cover most of the territory. The 90% in the title is a rule of thumb rather than a measured figure, but the point stands: there is a real, defensible minority of cases where fine-tuning is the right tool. Here is how to tell which side you are on.
First, be clear about what fine-tuning changes
Fine-tuning changes how a model behaves on a narrow task: its format, style, vocabulary and the patterns it favors. It is a poor way to teach a model facts that change — prices, policies, catalogs — because that knowledge is frozen at training time and hard to audit. That is what retrieval is for. A useful shorthand: RAG for knowledge, fine-tuning for behavior, and better prompting for everything you have not tried yet.
The four scenarios where fine-tuning genuinely wins
- Latency or cost compression at scale. At high, steady volume on a narrow task, a small fine-tuned open-weight model on your own infrastructure can be much cheaper and faster per request than a large hosted model. Do the math with real volumes: GPU hours, utilization, engineering time and monitoring on one side, API spend on the other.
- Strict output format compliance. If downstream systems break on malformed output, first try your provider's structured-output or constrained-decoding features, which enforce a JSON schema. If those are unavailable in your serving stack, or the task needs a consistent style that prompting cannot hold, fine-tuning can close the gap.
- Strong domain shift. Legal Marathi, medical Tamil, telecom-specific jargon — models tuned on the right data can outperform general-purpose ones on narrow tasks. RAG plus a frontier model usually still wins, but not always, so test both.
- On-prem or air-gapped deployment. If data cannot leave your environment, you are running an open-weight model anyway, and fine-tuning a small one is often the most practical route to acceptable quality. This requirement is increasingly common in BFSI and government.
The honest signs you should not fine-tune
- You don't have the data. Without a few hundred to a few thousand clean, representative examples of the exact input and the ideal output, you don't have a fine-tuning project — you have a data project.
- You can't name the metric. If you cannot say which metric should improve, and by how much, you will not be able to tell whether training worked.
- You haven't exhausted prompting. If your prompt is three lines long, try a detailed one with worked examples first. It is far cheaper.
- The knowledge changes weekly. Use retrieval.
A decision table
| Situation | Start with | Consider fine-tuning when |
|---|---|---|
| Answers depend on company documents | RAG | Retrieval is right but answer style is still wrong |
| Output must match a schema | Structured outputs | Schema enforcement is not available where you serve the model |
| High volume, narrow task | A prompted hosted model to prove value | Volume is steady and the cost gap justifies self-hosting |
| Data must stay on-prem | An open-weight model with good prompting | Quality on your eval set still falls short |
LoRA vs full fine-tuning
For most useful fine-tunes, LoRA and similar parameter-efficient methods are the right starting point. They train quickly, need far less GPU memory than a full fine-tune, and let you serve several adapters on one base model. Libraries such as Hugging Face's PEFT make this routine. Full fine-tunes are warranted when adapters hit a quality ceiling — typically tasks where the model must learn genuinely new representations, not just adjust existing ones.
Base model choice in 2026
Common open-weight bases include the Llama, Qwen, Mistral, Gemma and Phi families, as well as OpenAI's gpt-oss models. The decision usually comes down to three things:
- License terms. Some models ship under Apache 2.0; others, Llama among them, use custom licenses with additional conditions. Have someone read the license against your use case.
- Serving cost. Parameter count, memory footprint and how well the model runs on the hardware you have.
- Task performance. Run a quick evaluation on your own examples. Don't pick on public benchmarks alone.
A worked example
Say a support team classifies 200,000 tickets a month into 40 categories with a large hosted model, and accuracy on a held-out set of 1,000 tickets is where the team wants it. The bill is noticeable but tolerable. A sensible sequence: first try a smaller hosted model with a better prompt and examples, and stop there if accuracy holds. If it drops, fine-tune a small open-weight model with LoRA on a few thousand labeled historical tickets, serve it with vLLM, and compare accuracy and total cost per ticket against the original. Adopt it only if quality matches at a clearly lower cost, counting the people who will maintain it.
Hidden costs after training
- Retraining when the base model is updated or your data drifts.
- An evaluation suite that runs before every new version ships.
- Serving infrastructure, autoscaling and monitoring — the work an MLOps pipeline exists to handle.
- Records of training data provenance for audits.
How we run fine-tunes at Velura Labs
Our model fine-tuning engagements start with an audit that tests whether fine-tuning will actually beat RAG and prompting on your task — and we say so when it won't. When it will, the usual result is a LoRA adapter on an open-weight base, served through an MLOps pipeline with evaluation built in. Get in touch if you'd like a second opinion before committing to a training budget.
We ship work like this for clients in the US (California, Texas, Washington, New York), across Europe (France, Italy and the EU), the Gulf (UAE and Saudi Arabia) and India — with an India delivery base that keeps cost down and time-zone overlap high. Talk to us.