A lot of MLOps writing assumes a platform team of twelve and a cloud budget that doesn't blink. Most Indian AI startups have neither. Here is the stack we routinely deploy for early-stage teams that need to be production-ready without spending more on infrastructure than on engineering.
Serving: vLLM is a strong default
For self-hosted models, vLLM is where most teams should start. Continuous batching and paged attention give it high throughput, it exposes an OpenAI-compatible API so application code barely changes when you move between hosted and self-hosted models, and it supports a broad range of model families. SGLang is a credible alternative worth benchmarking on your workload. Put either behind Caddy or a Cloudflare Tunnel for HTTPS without faffing about.
Size the GPU to the model. A 7B or 8B model in 16-bit precision fits on a single 24 GB card such as an A10 or L4. A 70B model needs several 80 GB GPUs, or aggressive quantisation, to run with a usable context length. Quantised formats such as AWQ, GPTQ and FP8 cut memory needs substantially, but always rerun your evals after quantising.
Hosted vs self-hosted: pick by traffic shape
- Bursty, low total volume: hosted (OpenAI, Anthropic, Bedrock). Pay per token; your serving cost is zero when nobody is using the system.
- Sustained, high volume: self-hosted on dedicated GPUs. At high, steady volume a fixed-cost GPU can undercut per-token pricing by a wide margin, but work out the break-even for your own traffic first.
- Mixed: a router. LiteLLM or a homegrown gateway sends cheap calls to your self-hosted cluster and frontier-required calls to a hosted API.
Working out the break-even
Say a dedicated GPU server costs you G per month, fully loaded, including the engineering time to run it, and a hosted model of similar quality for your task costs P per million tokens. Break-even volume is G divided by P, in millions of tokens a month. If G is ₹1,50,000 and P is ₹150, break-even is 1,000 million tokens a month, or about 33 million a day. Below that, hosted is cheaper. Above it, self-hosting starts to win, provided your GPU can serve that volume within your latency target. These figures are hypothetical; plug in your own quotes.
Observability: Langfuse for traces, Datadog or Grafana for everything else
Langfuse is one of the cheapest meaningful tracing stacks: open source, self-hostable, and modest to run at low volume. Wire it up early. For every model call, log the prompt version, model name, input and output token counts, latency, cost and any user feedback. For metrics and logs, use Datadog if you can afford it, and Grafana Cloud's free tier or a self-hosted Loki setup if you can't. Observability is non-negotiable in production; the only question is how cheap you can make it.
Compute: Modal for bursts, dedicated servers for steady state
Modal's serverless GPUs suit irregular workloads: billed per second, no cluster to manage, and they scale to zero. For steady production load, dedicated GPU servers from providers such as Hetzner or OVHcloud are often much cheaper than on-demand hyperscaler instances of a similar spec. Compare current pricing and availability before committing, because GPU supply changes quickly. AWS, Google Cloud or Azure make sense for the parts that talk to their other services, or while you have credits to use.
Data residency and Indian compliance
If you process personal data of people in India, the Digital Personal Data Protection Act, 2023 applies, and enterprise customers in banking, insurance and health will ask where data is stored and who can reach it. AWS, Google Cloud and Azure all run cloud regions in India, and hosting there answers many of those questions early. Check whether your hosted model provider offers in-region processing on your plan, and keep a current data-flow diagram; procurement teams will ask for it.
CI and deploy: GitHub Actions plus Docker
Boring is good. Build a Docker image in CI, push it to a registry, deploy on tag. Pin model versions and prompt versions in configuration, and run your eval suite as a CI step so a model swap cannot reach production untested. Skip Kubernetes until you have a real reason, such as several services with different scaling needs, a platform engineer to run it, or a customer requirement. For many pre-Series A companies, that day never comes.
The minimum stack in one table
| Layer | Default | When to change |
|---|---|---|
| Model access | Hosted API behind a LiteLLM gateway | Sustained volume past break-even |
| Self-hosted serving | vLLM | Benchmark SGLang for your workload |
| Burst compute | Modal | Steady load: dedicated GPU servers |
| Traces | Langfuse | Rarely |
| Metrics and logs | Grafana Cloud free tier or Loki | Datadog once budget allows |
| CI/CD | GitHub Actions and Docker | Kubernetes when several services justify it |
Common mistakes
- Self-hosting before traffic justifies it, then paying for idle GPUs.
- Not tracking cost per request, so nobody notices which feature drives the bill.
- Leaving development GPU instances running over weekends.
- Quantising a model without rerunning the evals.
How we deploy this at Velura Labs
Our AI Deployment & MLOps retainer builds on this stack (vLLM serving, Langfuse traces, Modal for bursts, GitHub Actions CI) with the runbooks and handover your team needs. For tactical infrastructure advice without a retainer, our Backend & Infrastructure service covers it. Talk to us if your cloud bill has grown faster than your usage.
Available to businesses across the United States (Washington, California, Texas, New York), Europe (France, Italy and the wider EU), the Middle East (Dubai and the Gulf) and India. Get in touch to scope your build.