LLM applications fail in ways traditional APM doesn't catch. Latency is fine but the answer is wrong. Load is flat but token usage has tripled since yesterday. The model returned valid JSON but called the wrong tool. Standard observability tells you the request returned 200. Useful observability tells you the request returned a bad answer, why, and what it cost. Here is the minimum stack to have in place before real users arrive.
1. Structured traces around every model call
Every LLM call gets a span with the model and version, input and output tokens, latency (including time to first token when streaming), cost, cache hit or miss, prompt template version, and the feature it served. Use OpenTelemetry where you can. Its generative AI semantic conventions define standard attribute names, though they are still evolving, so pin the version you follow. Without these spans you cannot answer "is my prompt change saving money or burning it?"
2. Full prompt and response capture (with PII filters)
Store the actual prompt and response, not just metadata. You need them to debug specific failures, build regression tests from real traffic and produce audit logs. Redact personal data at the capture layer, before it reaches storage. For regulated industries, encrypt at rest, set a retention period and gate access through your audit trail. If full capture is too sensitive for some flows, capture a sample under stricter access.
3. Tool-call and retrieval telemetry
For agentic systems, every tool call needs its own span with arguments, return value, latency and success or failure, nested under the model call that triggered it. A common production failure for agents is the model calling the wrong tool, or the right tool with the wrong arguments, and you won't see it without tool-level traces. For RAG, log the retrieved document IDs and scores as well, so you can tell a retrieval miss from a generation error.
4. Online quality signals
Wire user feedback (thumbs up or down, citations clicked, edits to generated text, follow-up question rate, escalations to a person) into the same backend as latency and cost. Quality regressions often show up in user behaviour before they show up in offline evals. Tag each request with its prompt template and model version so you can correlate a change with its effect. Where the budget allows, run an automated grader over a sample of production traffic.
5. Cost telemetry with budgets
Roll up cost per user, per feature and per tenant, and alert when daily spend crosses a threshold. Many LLM cost blowouts come from a single buggy code path looping, such as an agent retrying a failing tool again and again. Without per-feature attribution, you find it days later in the invoice. Alongside alerts, set hard limits on steps and tokens per request.
6. Eval runs on every change
CI runs your offline evaluation set on every prompt, model or retrieval change and blocks deploys on regressions. Teams that skip this tend to discover quality regressions weeks later through user complaints. It is a day-one investment with a lifetime payoff.
The minimum schema
| Field | Why it matters |
|---|---|
| trace ID, user or tenant ID | Follow one request end to end; attribute cost |
| model, prompt version | Correlate quality and cost with changes |
| tokens in and out, cost, latency | Spend and performance |
| tool calls with arguments and status | Agent debugging |
| retrieved document IDs and scores | Separate retrieval failures from generation failures |
| feedback and grader scores | Online quality |
Alerts worth having from the start
- Hourly and daily spend above budget, per feature.
- Error rate from the model provider, including rate limits and timeouts.
- Tool-call failure rate outside its normal range.
- Negative-feedback rate rising after a deploy.
- 95th-percentile latency above what the user experience can tolerate.
The tools we currently like
Langfuse, Helicone and Arize Phoenix are open-source, self-hostable options that cover tracing, prompt management and evaluation to varying degrees. If your organisation already runs a general observability platform, check its LLM features before adding another system; Datadog, for example, offers LLM observability. Avoid hand-rolling. Log lines in Postgres look cheap until you need to slice by prompt version and tool-call success.
If you only have a week
- Days 1–2: traces on every model call with tokens, cost, latency, model and prompt version.
- Day 3: prompt and response capture with redaction and a retention policy.
- Day 4: tool-call spans and spend alerts per feature.
- Day 5: wire whatever evaluation set you have into CI, even if it is only 50 cases. A small suite that blocks bad deploys beats a large one that nobody runs.
Dashboards, graders and fine-grained quality metrics can follow once real traffic tells you which questions you actually need answered.
Common mistakes
- Logging prompts without redaction, which creates a new store of personal data to protect.
- Tracking averages only; tail latency and tail cost are where problems hide.
- No prompt versioning, so there is no way to link a regression to the change that caused it.
FAQ
Does this replace our existing APM?
No. Keep your APM for infrastructure, databases and service health. LLM observability sits alongside it and answers questions APM was never designed for: was the answer right, which prompt produced it, and what did it cost.
Should we sample traces to save money?
Keep metadata for every request, since cost and error attribution need it. Full prompt and response bodies can be sampled once traffic is high, provided you always keep the requests that errored or received negative feedback.
How we approach this at Velura Labs
Our Custom LLM Applications and Agentic Systems engagements ship with this stack wired in, paired with Backend & Infrastructure for the platform side. Read our evaluation playbook for the offline side and multi-agent patterns for why tool-call telemetry matters most in agent systems. Talk to us if your first production incident is teaching you what you should have wired three months ago.
Our clients for this span US tech hubs (San Francisco, Seattle, Austin, New York), European markets (Paris, Milan, Rome), the Middle East (Dubai, Riyadh, Abu Dhabi) and India. Start a conversation from anywhere.