Small versus large models and practical routing

Overview

• Smaller models often reduce cost/latency but may perform worse on complex tasks; stronger models may justify higher cost for difficult reasoning or synthesis.
• Routing sends different tasks to appropriate paths.
• The right split comes from evaluation, not model size labels alone.

Why this matters

Task complexity and risk differ across requests.

An everyday example

A junior clerk handles routine classification while a specialist handles complex explanations.

A technical example

Template for verified claim status; small model for bounded intent; larger model for comparing several applicable clauses.

How it works

1. Benchmark tasks
2. Define deterministic routing signals
3. Choose allowed model
4. Validate output
5. Use bounded quality recovery or escalation.

Architecture and flow

Task/risk/context size → route → selected model/template → validation.

The insurance assistant example

A known “claim status” endpoint bypasses a reasoning model entirely.

When to use it

Diverse workload where evaluations show meaningful efficiency gains.

When to avoid it

Avoid relying solely on a small model's claimed confidence to decide whether its own answer is safe.

Interview answer

“I route based on measured task capability and deterministic risk signals, with explicit privacy-compatible fallbacks and quality checks.”

Benefits, trade-offs, and production considerations

A route is a policy, so version and evaluate it. Changes in traffic mix can invalidate earlier benchmarks.

// Provider-neutral routing example; actual adapters implement this interface.
type Task = "claim_status" | "intent" | "policy_explanation";
type Request = { task: Task; evidenceTokens: number; restrictedData: boolean };
type Route = "template" | "small-approved" | "strong-approved" | "private-path";
function route(req: Request): Route {
  if (req.task === "claim_status") return "template";
  if (req.restrictedData) return "private-path";
  if (req.task === "intent" && req.evidenceTokens < 1000) return "small-approved";
  return "strong-approved";
}
// Threshold is illustrative; calibrate against quality and latency datasets.
// Each route still needs documented model capabilities and access policy.