Latency, tokens, cost, errors, and service objectives

Overview

• Latency metrics describe waiting time, including queue delay, time to first token, and total completion time.
• Token and cost metrics show resource usage.
• Error metrics classify failed, blocked, incomplete, and unknown-outcome runs.

Why this matters

Average response time and total spend hide slow tails, loops, and noisy tenants.

An everyday example

A restaurant tracks preparation, waiting, and ingredient costs separately.

A technical example

Record p50/p95 latency, tool timeout rate, tokens per successful task, escalation rate, and cost by tenant/route.

How it works

1. Capture per-span usage and timing
2. Aggregate by task/model/version
3. Define objectives
4. Alert on regressions
5. Inspect representative traces.

Architecture and flow

Span metrics → task/tenant aggregation → dashboard + alert → investigation.

The insurance assistant example

A reranker improves correctness but breaches interactive p95 latency; decide whether to cache, route, or skip it on simple queries.

When to use it

Before and after rollout and when tuning quality versus efficiency.

When to avoid it

Do not use HTTP 200 rate as the only success metric: a streamed response can fail after headers are sent.

Interview answer

“I monitor outcome quality, tail latency, usage, cost, and classified failures rather than treating a successful HTTP request as a completed AI task.”

Benefits, trade-offs, and production considerations

• Label metrics carefully to avoid unbounded cardinality from user IDs and prompts.
• Account for retries, cached token rates, and background calls.
• Review queues need age and completion metrics too.