Failure handling, evaluation, and operational review
Practice this questionOverview
- A production reference design specifies failed and uncertain outcomes as carefully as successful ones.
- Evaluation checks behavior before release; observability detects failures in actual runs.
- Human escalation is an explicit supported outcome.
Why this matters
A friendly response can hide missing evidence, wrong identity, or an uncertain write.
An everyday example
A clerk distinguishes “not permitted,” “missing information,” and “system unavailable.”
A technical example
Tool timeout after submit produces pending_reconciliation, not “claim failed” or “claim created” without proof.
How it works
- Classify failure
- Choose safe response
- Persist task/operation state
- Retry/reconcile/escalate
- Verify outcome
- Record audit/trace.
Architecture and flow
Failure → deterministic classification → retry / correction / reconciliation / staff → durable result.
The insurance assistant example
Search failure never becomes a confident coverage answer; stale approval never becomes a valid submission.
When to use it
Every route, worker stage, approval, and external effect.
When to avoid it
Do not retry denials, hallucinate missing sources, or silently swallow failed notifications.
Interview answer
“I make no-evidence, denial, transient failure, stale approval, and unknown commit separate states, and test each end to end.”
Benefits, trade-offs, and production considerations
- Monitor authorization denials, unsupported answers, retrieval freshness, queue age, approval age, duplicate suppression, unknown commits, and cost per successful task.
- Redact prompts/traces and retain immutable business audit records under the appropriate policy.
| Test scenario | Expected outcome |
|---|---|
| Customer requests another customer's claim | Denied; no data or existence leak under defined policy |
| Policy changed since last conversation | Fresh applicable version retrieved; old cache invalidated |
| Uploaded invoice contains injected instructions | No unauthorized tools/actions; source treated as data |
| Required clause absent | Abstain and offer permitted escalation |
| Date ambiguous or OCR amount unclear | Ask/review affected field; avoid invented certainty |
| Draft changed after approval | Prior approval invalid; review new exact version |
| Submit duplicated after timeout | One canonical claim; existing or reconciled receipt |
| Model unavailable | Template/search/staff path as applicable; business status remains truthful |
| Email send times out | Reconcile message key/provider reference; avoid duplicate disclosure |
| Worker deploys during approval wait | Authorized run resumes with compatible state/version |
Deployment path: containerize Node and optional Python workers; use a managed database and queue/workflow deployment appropriate to the client's cloud; store secrets through service identity/secret management; provision a staging corpus and fixtures; run security and outcome evals; release to limited traffic; monitor; keep prompt/model/graph version rollback available. Do not create cloud resources merely to match this example.