Temperature and Top-P Explained: When (and Why) to Adjust LLM Randomness

Most developers discover temperature by accident — they get a weirdly repetitive output, someone on Stack Overflow says “just set temperature to 0.9,” and suddenly they’re tweaking it for everything without knowing why it sometimes makes things worse. Temperature top-p LLM randomness is one of those topics where five minutes of real understanding eliminates hours of cargo-cult parameter tuning.

This article covers exactly what these sampling parameters do at the token level, why they interact in ways that can break your outputs if you’re not careful, and gives you a decision framework for setting them correctly the first time.

What’s Actually Happening When You Change Temperature

An LLM doesn’t “think” and then “write.” It generates one token at a time, and at each step it produces a probability distribution over its entire vocabulary — potentially 50,000+ tokens. The raw output of the model is called logits: unnormalised scores for every possible next token. Softmax converts those into probabilities.

Temperature is a divisor applied to the logits before softmax. That’s the key mechanic most explanations skip.

import numpy as np

def softmax(logits, temperature=1.0):
    # Divide logits by temperature BEFORE applying softmax
    scaled_logits = logits / temperature
    exp_logits = np.exp(scaled_logits - np.max(scaled_logits))  # numerical stability
    return exp_logits / exp_logits.sum()

# Example: three candidate tokens with raw logits
logits = np.array([4.0, 2.5, 1.0])  # "the", "a", "some"

print("T=0.1 (near-deterministic):", softmax(logits, 0.1).round(4))
print("T=1.0 (default):           ", softmax(logits, 1.0).round(4))
print("T=2.0 (high entropy):      ", softmax(logits, 2.0).round(4))

Run that and you’ll see the effect clearly:

# Output:
# T=0.1 (near-deterministic): [1.0000 0.0000 0.0000]
# T=1.0 (default):            [0.8360 0.2014 0.0450] (approximate)  
# T=2.0 (high entropy):       [0.6479 0.3088 0.1472] (approximate)

At low temperature, the distribution sharpens — the model almost always picks the highest-probability token. At high temperature, the distribution flattens — low-probability tokens get a real shot. Temperature doesn’t make the model “more creative”; it changes how much probability mass flows to lower-ranked candidates. Whether those candidates are creative or garbage depends entirely on what the model learned.

Top-P (Nucleus Sampling): The Smarter Cutoff

Top-P, also called nucleus sampling, works differently. Instead of scaling the full distribution, it ranks tokens by probability, then samples only from the smallest set whose cumulative probability exceeds the threshold P.

def nucleus_sample(probs, top_p=0.9):
    """
    Sample from the nucleus — the smallest set of tokens
    whose cumulative probability >= top_p
    """
    sorted_indices = np.argsort(probs)[::-1]  # descending order
    sorted_probs = probs[sorted_indices]
    cumulative_probs = np.cumsum(sorted_probs)
    
    # Find cutoff: keep tokens until we hit top_p
    cutoff_idx = np.argmax(cumulative_probs >= top_p) + 1
    nucleus = sorted_indices[:cutoff_idx]
    
    # Renormalise within the nucleus and sample
    nucleus_probs = probs[nucleus] / probs[nucleus].sum()
    return np.random.choice(nucleus, p=nucleus_probs)

The practical upside: top-P adapts to context. When the model is very confident (say, completing “The capital of France is”), the nucleus might be just 2-3 tokens. When it’s uncertain (open-ended creative text), the nucleus might expand to 100+ tokens. This adaptive behaviour is why top-P generally produces less incoherent output at high randomness settings than temperature alone.

Top-K: The Dumber Sibling

Top-K just keeps the K highest-probability tokens, always. No adaptation. If K=50 and the model is nearly certain, you’re still sampling from 50 options when only 2 are reasonable. Most current APIs (Anthropic, OpenAI) don’t expose top-K directly — they use top-P instead, and for good reason. I’d avoid top-K unless you’re working with a model that doesn’t support nucleus sampling.

The Three Misconceptions That Break Production Pipelines

Misconception 1: “Temperature 0 is deterministic”

Almost deterministic, not fully deterministic. Floating-point arithmetic on GPUs is non-associative, meaning the order of parallel operations can produce slightly different results across runs. Anthropic’s docs explicitly note that temperature=0 may still produce occasional variation. OpenAI says the same. If you need truly reproducible outputs, you also need a fixed seed parameter (available in the OpenAI API, not yet in Anthropic’s) and even then, model updates will break reproducibility.

For production systems where consistency matters more than occasional variation, pair temperature=0 with output validation rather than assuming identical results. This connects to broader patterns around reducing LLM hallucinations in production — deterministic sampling helps, but it’s not a substitute for output verification.

Misconception 2: “High temperature = better creative output”

High temperature increases the variance of outputs, not their quality. Past a certain point — usually around 1.2-1.4 depending on the model — you start getting syntactic drift, repetition loops, and topic incoherence. The model hasn’t become more creative; it’s sampling from tokens that were low-probability for good reasons.

In practice, for creative tasks I rarely go above 1.0. The sweet spot is usually 0.7-0.9 — you get diversity without incoherence. If outputs feel samey at 0.8, the problem is usually the prompt, not the temperature.

Misconception 3: “You should always tune both temperature and top-P together”

Anthropic’s own recommendation is to adjust one or the other, not both simultaneously. They’re both controlling randomness but through different mechanisms, and stacking them creates interactions that are hard to predict. OpenAI’s API documentation says the same thing. My rule: use temperature for most tasks, switch to top-P if you need more consistent output quality at a given diversity level, but don’t touch both unless you have a specific, tested reason.

Parameter Settings by Use Case: A Practical Reference

These are the settings I actually use in production, not theoretical recommendations:

• Data extraction / structured output (JSON, tables, entities): temperature=0, top_p=1. You want the most probable, correct answer. Randomness here causes malformed JSON and hallucinated field values.
• Code generation: temperature=0 to 0.2. Syntax errors compound fast at higher values. If you want alternative implementations, make multiple calls at 0 and ask explicitly for different approaches in the prompt — see how Claude vs GPT-4 perform on code tasks for model-specific tuning notes.
• Summarisation and factual Q&A: temperature=0 to 0.3. Stay close to the source material.
• Copywriting, marketing, email drafts: temperature=0.7-0.9. Enough diversity to avoid robotic repetition, controlled enough to stay coherent.
• Brainstorming, ideation, creative fiction: temperature=0.9-1.0, top_p=0.95. Let the model roam, but keep a slight nucleus constraint to avoid total incoherence.
• Chatbots and conversational agents: temperature=0.5-0.7. You want natural variation without unpredictable behaviour. Consistent agent personas also depend heavily on your system prompt — if you haven’t read our piece on role prompting best practices, it’s worth pairing with this.

Real Implementation: Adaptive Temperature in a Production Agent

One pattern I’ve shipped that actually works: route different task types to different temperature settings within the same agent, rather than picking one global setting.

import anthropic

client = anthropic.Anthropic()

TASK_CONFIGS = {
    "extract":    {"temperature": 0.0, "description": "Structured extraction"},
    "summarize":  {"temperature": 0.2, "description": "Factual summarization"},
    "draft":      {"temperature": 0.8, "description": "Creative drafting"},
    "brainstorm": {"temperature": 1.0, "description": "Ideation and options"},
}

def run_task(task_type: str, system_prompt: str, user_message: str) -> str:
    config = TASK_CONFIGS.get(task_type, TASK_CONFIGS["summarize"])
    
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=1024,
        temperature=config["temperature"],
        system=system_prompt,
        messages=[{"role": "user", "content": user_message}]
    )
    return response.content[0].text

# Example: extract structured data (temperature=0)
result = run_task(
    task_type="extract",
    system_prompt="Extract the following fields as JSON: name, date, amount, currency.",
    user_message="Invoice from Acme Corp dated March 12 2025 for $4,500 USD."
)

# Example: creative draft (temperature=0.8)
draft = run_task(
    task_type="draft",
    system_prompt="You are a B2B copywriter specialising in SaaS products.",
    user_message="Write three subject line options for a product launch email."
)

At Haiku pricing (~$0.0008 per 1K input tokens, $0.004 per 1K output tokens as of mid-2025), a 500-token extraction call costs roughly $0.0024. Running 1,000 of these per day is about $2.40 — cheap enough that you don’t need to optimise aggressively, but worth logging so you catch runaway loops early.

If you want to handle cases where the model fails or returns invalid output at temperature=0, implement retry logic with a slight temperature bump on the second attempt. There’s a solid pattern for this in our article on LLM fallback and retry logic for production.

When Top-P Beats Temperature (and Vice Versa)

Top-P tends to outperform pure temperature adjustment in two scenarios:

1. Long-form generation where coherence must be maintained — nucleus sampling’s adaptive cutoff prevents the model from drifting into low-probability territory during multi-paragraph output.
2. Tasks where the model’s confidence varies dramatically by sentence — factual claims should draw from a tight nucleus; transitions and stylistic choices can come from a wider one. Top-P handles this automatically.

Temperature wins when you need predictable, tunable diversity across all outputs uniformly — A/B testing copy variants, generating training data with controlled entropy, or building systems where you need to explain the randomness setting to non-technical stakeholders (“we use temperature 0.7” is easier to communicate than top-P mechanics).

What the Docs Get Wrong (or Skip)

Most vendor documentation treats these parameters as independent sliders with obvious semantics. A few things they underemphasise:

• System prompt length affects effective temperature. Longer, more constrained system prompts functionally reduce the impact of high temperature because the model has less “room” to diverge. You can use this — a very detailed system prompt with temperature=0.9 often behaves more like temperature=0.6 in practice.
• Model scale matters. Larger models have sharper, more peaked distributions to begin with. Temperature=0.8 on GPT-4 or Claude 3.5 Sonnet produces less wild variance than the same setting on a 7B parameter open-source model. Calibrate separately for each model you deploy.
• Top-P of 1.0 is not the same as “no top-P filtering.” At top_p=1.0, all tokens remain eligible — it’s the maximum nucleus. Some implementations treat this as “disabled,” which is correct in effect, but it’s worth knowing what’s happening.

The Bottom Line: Who Should Set What

If you’re a solo founder shipping fast: start with temperature=0 for any task with a correct answer, temperature=0.7 for anything creative. Don’t touch top-P until you have a specific problem it solves. These two settings will cover 90% of your use cases correctly.

If you’re a team building a production agent: implement the task-routing pattern above, log your temperature settings alongside outputs, and do periodic reviews to see where high-temperature runs are producing garbage. Pair this with your hallucination reduction strategy — structured output verification becomes especially important when you’re allowing any variance in the sampling parameters.

If you’re working with open-source models (Llama, Mistral, Qwen): you’ll need to recalibrate from scratch — these models have different base distributions and often need lower temperature values than you’d use with frontier models to achieve equivalent output quality. Temperature=0.5 on Mistral 7B might feel like temperature=0.8 on Claude Sonnet.

The core insight is simple: temperature top-p LLM randomness parameters are controls on the sampling process, not on model intelligence. Tuning them correctly means understanding what kind of probability distribution your task needs — tight and peaked for accuracy, wide and flat for diversity. Everything else follows from that.