Errors, retry logic, and recovery

Overview

• Error handling distinguishes recoverable infrastructure failures, invalid user input, unsupported model decisions, and business failures.
• Retry repeats an operation after an appropriate transient failure.
• Recovery uses persisted progress and reconciliation to continue safely.

Why this matters

Provider timeouts and worker restarts are normal production events.

An everyday example

Call a busy department again later; do not keep resubmitting a rejected form with unchanged fields.

A technical example

Retry a read on transient 503; ask for a missing document; do not retry permission denial; reconcile a timed-out submission by idempotency key.

How it works

1. Classify error
2. Select bounded policy
3. Apply backoff with jitter where appropriate
4. Preserve original error
5. Update status
6. Resume or escalate.

Architecture and flow

Node failure → classification → retry / user correction / reconcile / fail.

The insurance assistant example

A claim-create timeout may mean the claim was created but the response was lost; verify before retrying.

When to use it

Every external dependency and consequential action.

When to avoid it

Do not apply automatic retries to non-idempotent writes without reconciliation or repeatedly ask the LLM to fix an authorization failure.

Interview answer

“I retry transient failures with bounds, keep user and permission failures distinct, and make side effects idempotent because recovered nodes may execute again.”

Benefits, trade-offs, and production considerations

• Per-node retry limits multiply with provider retries and whole-run retries.
• Set one total deadline and cost budget.
• Record unknown commit outcomes explicitly; a checkpoint alone cannot prove whether an external service applied a write.