Model selection, routing, fallback, latency, and cost

Overview

• Model selection chooses a model for a task.
• Routing chooses among models or processing paths per request, while fallback handles a failed or unavailable preferred path.
• Latency is waiting time; cost includes tokens, tools, retrieval, infrastructure, and review.

Why this matters

One large model for every request can be slow and expensive, while one small model may fail difficult cases.

An everyday example

Simple service requests go to the front desk; complex cases go to a specialist, with a backup if the specialist is unavailable.

A technical example

Rules handle a known claim ID; a small model classifies intent; a stronger model handles a complex grounded explanation.

How it works

1. Establish quality targets
2. Benchmark models
3. Define routing rules
4. Enforce limits
5. Attempt preferred route
6. Apply approved fallback
7. Monitor quality and cost by route.

Architecture and flow

Task + risk + budget → route → provider → validated result or bounded fallback.

The insurance assistant example

If generation fails, return verified claim status in a deterministic template rather than retrying five expensive models.

When to use it

Routing is valuable when task difficulty, volume, or privacy needs differ meaningfully.

When to avoid it

Avoid routing complexity before you have a baseline. Never silently fall back to a provider outside approved data regions.

Interview answer

“I choose models using task evaluations, latency, cost, and data requirements. Routing and fallback preserve those requirements while improving efficiency and availability.”

Benefits, trade-offs, and production considerations

• A model's self-reported confidence is not a calibrated probability.
• Use empirical route performance, deterministic risk signals, and measured abstention.
• Compare cost per successful task, not just price per token.