Token optimization, prompt compression, and context trimming
Practice this question🟠 SHOULD KNOW · Optimize after establishing an acceptable quality and security baseline.
Overview
- Token optimization removes unnecessary model input/output while preserving task information.
- Prompt compression shortens instructions or evidence; context trimming selects only relevant history and sources.
- Shorter is useful only if important conditions survive.
Why this matters
Large prompts and long outputs increase cost, latency, and distraction.
An everyday example
Give an employee the relevant pages and a clear brief instead of a box of unrelated paperwork.
A technical example
Send claim status and missing-document codes rather than the full account record; cap requested explanation length.
How it works
- Measure token breakdown
- Remove repetition
- Select evidence
- Preserve exceptions/tool pairs
- Reserve output
- Rerun quality tests.
Architecture and flow
Measured prompt → relevance/minimization → bounded context → model → quality check.
The insurance assistant example
Use three applicable policy sections rather than a whole 100-page document for each question.
When to use it
High-volume chat, RAG, and agent tasks with growing histories.
When to avoid it
Do not compress away permissions, coverage limits, source references, or user-confirmed facts.
Interview answer
“I reduce irrelevant tokens and output verbosity, then verify that completeness and grounded accuracy remain acceptable.”
Benefits, trade-offs, and production considerations
A cheap answer requiring repeat attempts can cost more per successful task. Token reduction should be measured alongside task success, not pursued alone.