Exact caching, prompt caching, and semantic caching

Overview

• Exact caching reuses results for the same normalized request and dependencies.
• Provider prompt caching can reduce processing/billing for supported repeated prefixes.
• Semantic caching reuses results for similar meanings, which is more risky because similar requests may require different answers.

Why this matters

Repeated public or stable knowledge requests can avoid duplicate work.

An everyday example

Reuse an approved answer to the same question, but do not reuse one customer's claim status for another customer.

A technical example

Cache a policy answer under tenant, authorization scope, policy version, question, model, and prompt version.

How it works

1. Define eligible tasks
2. Build complete cache key
3. Check freshness/permissions
4. Reuse or compute
5. Invalidate on source/policy change.

Architecture and flow

Authorized request + source versions → cache lookup → valid hit or model call.

The insurance assistant example

Cache public policy FAQs; avoid semantic reuse for personalized coverage, payouts, or claim status.

When to use it

Stable nonpersonalized content and measured repeated workloads.

When to avoid it

Similar wording is insufficient when tenant, date, policy, or authorization differs.

Interview answer

“I cache only when identity scope, dependencies, and freshness make reuse safe. Semantic caching needs stricter eligibility than exact caching.”

Benefits, trade-offs, and production considerations

• Include corpus/ACL versions and retention.
• Invalidate on revocation and deletion.
• A cache can preserve an injected or incorrect answer, so validate before writing and monitor hit quality.