Temperature and Top-P Explained: When to Adjust LLM Randomness in Production Agents

Most developers treat temperature like a volume knob — turn it up for “creative” tasks, turn it down for “factual” ones. That mental model is close enough to survive demos but breaks down in production. If you’ve ever had a coding agent that works fine in testing then starts hallucinating variable names at temperature 0.7, or a summarization pipeline that produces weirdly identical outputs across thousands of documents, you’re probably misconfigured. Understanding temperature top-p LLM mechanics at the sampling level will let you tune these parameters deliberately instead of guessing.

What’s Actually Happening When You Set Temperature

LLMs generate text by predicting the next token from a probability distribution over the entire vocabulary. Before sampling, the raw scores (logits) are converted to probabilities using a softmax function. Temperature is a scalar divisor applied to those logits before softmax.

Mathematically: P(token) = softmax(logits / T)

At temperature = 1.0, the distribution is unchanged. At temperature < 1.0, you’re sharpening the distribution — high-probability tokens become disproportionately more likely, and the long tail gets compressed. At temperature > 1.0, you flatten it — the model becomes more willing to pick lower-probability tokens.

At temperature 0 (or near 0), you’re doing greedy decoding: always pick the highest-probability token. This sounds like it should produce the “best” output, but it doesn’t — it produces the most expected output, which is often repetitive and occasionally stuck in loops.

The Practical Effect on Output

Here’s what this actually means for outputs you care about:

• Low temperature (0.0–0.3): Near-deterministic. Good for structured outputs, code generation, JSON extraction. Same prompt usually gives the same answer.
• Medium temperature (0.4–0.7): Some variance. Good for summarization, Q&A, most chat applications. The model paraphrases differently across runs without going off-script.
• High temperature (0.8–1.2): High variance. Good for brainstorming, creative writing, generating diverse options. Output can degrade noticeably above 1.0 depending on the model.

One thing the documentation doesn’t tell you: the effect of temperature is model-dependent. GPT-4’s temperature 0.7 behaves differently from Mistral-7B’s temperature 0.7 because the underlying logit distributions are different. You can’t port settings directly across models.

Top-P (Nucleus Sampling) Is Not a Temperature Replacement

Top-p, or nucleus sampling, works differently. Instead of scaling the whole distribution, it dynamically selects the smallest set of tokens whose cumulative probability exceeds the threshold P, then samples only from that set.

At top-p = 0.9, the model considers only the tokens that together account for 90% of the probability mass. The tail — all the low-probability weird tokens — gets excluded. At top-p = 1.0, nothing is excluded.

The key difference: temperature adjusts how peaked the distribution is; top-p adjusts how many tokens are in play. When the model is confident (a few tokens dominate), top-p naturally restricts the sample set to just those tokens. When the model is uncertain (probability spread across many tokens), top-p lets in more candidates.

When Top-P Actually Matters

Top-p has more impact when the model is uncertain. For a coding task where the next token is almost certainly a closing bracket, top-p = 0.9 and top-p = 0.99 produce identical results — the bracket holds most of the probability mass. But for an open-ended creative prompt where dozens of tokens are plausible, top-p meaningfully controls how adventurous the sampling gets.

The most common mistake: setting both a low temperature AND a low top-p simultaneously. These are partially redundant constraints. Low temperature already concentrates probability on the top tokens — adding a tight top-p on top of that doesn’t give you more control, it just increases the chance of degenerate output if the top tokens happen to be bad.

Tuning for Specific Production Tasks

Code Generation and Structured Output

For anything that needs to be syntactically correct — SQL queries, JSON objects, Python functions — you want determinism. Set temperature low and don’t overthink top-p.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-haiku-20241022",
    max_tokens=1024,
    temperature=0.1,  # near-deterministic for structured output
    messages=[
        {
            "role": "user",
            "content": "Write a Python function that validates an email address using regex. Return only the function, no explanation."
        }
    ]
)

print(response.content[0].text)

At temperature 0.1 with Haiku, you get consistent, predictable code. Running this 10 times will give you near-identical outputs with minor stylistic variance. This costs roughly $0.0008 per run at current Haiku pricing ($0.80/M input + $4.00/M output for ~200 tokens total).

Recommendation: temperature 0.0–0.2, top-p 0.9–1.0. Don’t bother tuning top-p tightly here — it doesn’t add meaningful control at low temperature.

Summarization and Information Extraction

Summarization is trickier than it looks. You want faithfulness to the source (low temperature) but you also don’t want identical boilerplate across thousands of documents (which happens at temperature 0 when inputs are similar). A slight bump in temperature helps, but going too high starts introducing hallucinated details.

import openai

client = openai.OpenAI()

def summarize_document(text: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0.3,   # low enough for faithfulness, not zero to avoid monotony
        top_p=0.9,         # exclude the weird long-tail tokens
        messages=[
            {
                "role": "system",
                "content": "Summarize the following document in 3 bullet points. Be concise and factual."
            },
            {
                "role": "user",
                "content": text
            }
        ]
    )
    return response.choices[0].message.content

# Example usage
summary = summarize_document("Your long document text here...")
print(summary)

At GPT-4o-mini pricing (~$0.15/M input, $0.60/M output), summarizing a 2,000-token document costs under $0.001. Temperature 0.3 with top-p 0.9 is the sweet spot I’ve landed on after running summarization pipelines on 50k+ documents. Hallucination rate was measurably lower than at 0.7, and outputs were more varied than at 0.

Brainstorming and Creative Generation

For ideation pipelines — generating product names, ad copy variants, story seeds — you want genuine diversity across outputs. This is where higher temperature earns its place.

import anthropic

client = anthropic.Anthropic()

def generate_product_names(description: str, count: int = 5) -> list[str]:
    names = []
    for _ in range(count):
        response = client.messages.create(
            model="claude-3-5-haiku-20241022",
            max_tokens=100,
            temperature=0.9,    # high variance — we want different ideas each run
            top_p=0.95,         # still exclude the truly weird tokens
            messages=[
                {
                    "role": "user",
                    "content": f"Generate one creative, memorable product name for: {description}. Return only the name, nothing else."
                }
            ]
        )
        names.append(response.content[0].text.strip())
    return names

names = generate_product_names("A mobile app that tracks water intake using AI image recognition")
print(names)
# Example output: ['AquaLens', 'SipSense', 'HydroSnap', 'WaterWise', 'ClearTrack']

Notice I’m calling the API in a loop rather than asking for 5 names in one call. At temperature 0.9, a single call with “give me 5 names” tends to produce names that are structurally similar to each other because the model generates them sequentially and anchors on the first few. Separate calls give you genuine diversity.

Top-K: The Parameter Nobody Talks About

Some APIs (Anthropic, most open-source backends) also expose top-k, which hard-limits sampling to the K most likely tokens regardless of their cumulative probability. Top-k = 40 means only the 40 most probable tokens are ever considered.

Top-k is blunter than top-p. I’d use top-p over top-k in most cases because top-p adapts to the model’s confidence dynamically. The exception: if you’re running inference on smaller open-source models that tend to produce garbage in their tails, top-k 40–50 can act as a safety net. For GPT-4, Claude, or Gemini, you can mostly ignore top-k.

The Parameter Interaction Problem

Temperature, top-p, and top-k interact in ways that documentation rarely makes explicit:

• If temperature is very low, top-p has almost no practical effect (one token dominates anyway)
• If temperature is high AND top-p is high, you’re sampling from a very wide distribution — output quality can collapse on weaker models
• Setting temperature = 0 and top-p = 1.0 gives you greedy decoding, which is fine for deterministic pipelines but tends to produce repetitive text at longer lengths

My working defaults for production, tuned across dozens of deployments:

• Code / structured output: temp 0.1, top-p 1.0
• Factual Q&A / RAG: temp 0.2, top-p 0.9
• Summarization: temp 0.3, top-p 0.9
• Chat / assistant: temp 0.5–0.7, top-p 0.9
• Creative / brainstorm: temp 0.8–1.0, top-p 0.95

Testing Your Settings Before You Commit

Don’t tune by feel. Build a simple eval harness: take 20–50 representative prompts, run each setting 5–10 times, and score outputs on the dimensions that matter for your task (correctness, diversity, length adherence, format compliance). A quick pytest loop with deterministic seed inputs works fine for this.

For production agents where consistency matters, I strongly recommend logging the actual temperature and top-p alongside each LLM response. When something goes wrong (and it will), you want to know if it was a parameter misconfiguration or a genuine model failure. This takes 5 minutes to add and saves hours of debugging.

When You Need Determinism Across Runs

Setting temperature = 0 gets you close to determinism but doesn’t guarantee it on all APIs. OpenAI has a seed parameter that, combined with temperature 0, gives you reproducible outputs. Anthropic doesn’t expose a seed parameter — temperature 0 is as deterministic as you’ll get. This matters for testing and compliance scenarios where you need to prove the system would give the same answer to the same question.

If full reproducibility is a hard requirement, consider caching LLM responses by prompt hash rather than fighting the sampling layer. It’s more reliable and eliminates API costs on repeated identical queries.

Bottom Line: Temperature Top-P LLM Settings by Role

Solo founders building quick automation workflows: Don’t overthink it. Start with temperature 0.3 for anything data-processing, 0.7 for anything user-facing. Adjust based on complaints, not theory.

Teams building production agents: Define per-agent defaults, log them, and version them alongside your prompts. A temperature change is a behavioral change — treat it like a code change.

Anyone running high-volume pipelines (10k+ calls/day): The difference between temperature 0.0 and 0.3 in a summarization pipeline probably doesn’t affect cost, but it does affect output variance. Run an eval before locking in settings.

Open-source model users: Temperature and top-p behavior varies more across model families than it does across OpenAI/Anthropic/Google. Llama and Mistral models tend to need slightly lower temperatures than frontier models for equivalent output quality. Always run your own evals — don’t port settings from GPT-4 benchmarks.

The temperature top-p LLM tuning problem is ultimately an empirical one. The mechanics tell you what direction to move; your specific task and model tell you how far. Build the eval harness once, and you’ll stop guessing forever.