Temperature and top-p

Overview

• Temperature changes how sharply a model favors its higher-scoring next-token choices.
• Lower values generally reduce variation; higher values allow more variation.
• Top-p, also called nucleus sampling, restricts selection to a set of tokens covering a chosen cumulative probability.

Why this matters

Different tasks need different levels of variation, such as predictable extraction versus varied marketing drafts.

An everyday example

A chef can repeatedly use a standard recipe or allow more variation while experimenting.

A technical example

Use conservative supported settings for a claim-field extractor; allow more variation for alternative customer-friendly phrasings.

How it works

1. The model produces next-token scores
2. Sampling settings adjust the candidate distribution
3. A next token is chosen
4. The process repeats.

Architecture and flow

Task type → supported sampling configuration → generation → evaluation.

The insurance assistant example

A policy answer needs consistent language, but lowering temperature does not verify that the cited policy is applicable.

When to use it

When the selected model supports these settings and evaluation shows a useful effect.

When to avoid it

Do not use temperature zero as a correctness guarantee or tune both settings randomly without controlled comparisons.

Interview answer

“Temperature and top-p affect sampling diversity. They influence variation, not factual truth, and I validate them on the actual task.”

Benefits, trade-offs, and production considerations

• Providers and model families expose different controls; some reasoning models restrict them.
• Even low-temperature runs may vary.
• Reliability comes from constrained contracts, evidence, and tests.