LLM Observability, Tracing & Prompt Management
A traditional service fails loudly — an exception, a 500, a stack trace. An LLM app fails quietly: it returns a confident, well-formatted, wrong answer, and nothing in your logs flags it. Observability is how you see inside a non-deterministic system, and prompt management is how you change it without breaking it.
Real-world examples: debugging why a RAG answer cited the wrong document three retrieval hops deep; tracking the daily token bill per feature; rolling back a prompt edit that doubled latency; finding that output quality drifted after a silent upstream model update.
Tracing: the LLM stack trace
A single user request fans out into many steps — retrieval, a planning call, tool calls, a final generation. A trace records that whole tree as nested spans, each capturing its inputs, outputs, latency, token counts, and cost. Tracing is the only way to answer "which step produced the bad answer" in a multi-step pipeline.
Trace waterfall — one request, every span on a shared clock
Indigo bars are LLM generations, teal are ordinary spans, yellow are guardrails. Retrieval and the order-lookup tool don't depend on each other — running them in parallel cuts the trace to its slowest branch, but the cost doesn't move: parallelism buys latency, never dollars.
The waterfall also answers the latency question at a glance: the model call dominates, and the two independent steps (retrieval, the tool call) are running one after the other for no reason — toggle them parallel and the trace shortens while the bill stays identical.
The metrics worth attaching to every span:
- Quality: eval scores, user feedback (👍/👎), guardrail flags.
- Cost: input + output tokens × price; aggregated per request, per feature, per user.
- Latency: per step and end-to-end; for streaming, time-to-first-token matters as much as total time.
A back-of-envelope cost model makes the bill predictable:
where counts tokens in and out of each call and is the per-token price for each direction.
Log the full context, not just the answer
When an LLM step misbehaves, you need the exact prompt that produced it — system prompt, retrieved chunks, tool outputs, parameters, and model version. Logging only the final response makes failures irreproducible. Capture the complete resolved context per span (with PII handled), or you'll be debugging blind.
In practice you don't invent this trace format: OpenTelemetry standardizes the span model and the GenAI attribute names (gen_ai.usage.input_tokens, …), and platforms like Langfuse store, price, and score the resulting traces. For the concrete instrumentation stack, see the deep dive:
Monitoring drift in production
Inputs and outputs drift even when your code is frozen:
- Input drift: users ask new things; the distribution of prompts shifts away from your eval set.
- Output drift: response length, refusal rate, or format changes — often after a silent provider-side model update.
- Quality drift: eval scores on sampled traffic decline.
Because there's no label at inference time, monitor proxies: user feedback, guardrail trigger rates, retrieval relevance scores, format-validity rates, and the online eval scores from the previous lesson.
Example: a provider silently updates the model behind your API. Answers get slightly longer and the JSON format-validity rate drops from 99% to 91% overnight — no deploy happened, no error fired, but the output distribution moved. The proxy metric catches what your logs can't.
Prompt management
The prompt is your app's most-edited, least-version-controlled component. Treat it like code:
- Versioning: every prompt has a version; you know which version served which response (via the trace).
- A registry / templates: prompts live in a managed store with variables, not hard-coded in scattered strings, so non-engineers can iterate and you can audit changes.
- Testing before rollout: run the eval suite against a new prompt version before it ships; canary it to a fraction of traffic and compare metrics.
- Decoupling from code: updating a prompt shouldn't require a full redeploy, but it should go through the same review and eval gates.
An untracked prompt edit is an unreviewed production change
"Just tweaking the prompt" edits live behaviour for every user with none of the safeguards you'd demand of a code change. Without versioning + eval gating you can't tell whether yesterday's quality dip came from a prompt edit, a data change, or a silent model update, and you can't roll back cleanly. Version prompts and tie each one to its eval results.
Exercise
Why is request tracing (nested spans) essential for a multi-step LLM app (retrieval → planning → tools → generation)?
Need a hint?
A bad final answer could originate at any step.
Exercise
A request uses 3,000 input tokens at $2 per 1M and 1,000 output tokens at $6 per 1M. What is its cost in cents?
Need a hint?
cost = 3000×2/1e6 + 1000×6/1e6 dollars, then convert to cents.
Common mistakes
- Logging only outputs: irreproducible failures because the full resolved context wasn't captured.
- No tracing in multi-step apps: unable to localise which retrieval/tool/generation step failed.
- Hard-coded, unversioned prompts: production behaviour changes you can't audit or roll back.
- Assuming a frozen model is a stable model: missing output drift from silent provider updates.
Key takeaways
- Traces of nested spans are the LLM stack trace — the only way to localise failure in a multi-step pipeline.
- Attach quality, cost, and latency (including time-to-first-token) to every span.
- Monitor input/output/quality drift via proxies, since labels aren't available at inference.
- Manage prompts as versioned, tested, registry-stored artifacts — an untracked prompt edit is an unreviewed production change.
Related concepts
- Agents & Tool Use: the multi-step trajectories that most need tracing
- Prompt Engineering & In-Context Learning: the prompts this lesson asks you to version and test
- Monitoring & Observability: the classical-ML monitoring this extends to LLMs
- Observability for Agents: what changes when the traced system decides its own next step at runtime