Temperature and sampling

At each step the model gives every token in its vocabulary a score. The scores are turned into probabilities, and one token is picked. How it's picked is called sampling, and a few settings control it.

Temperature reshapes the probabilities before the pick. Below 1, the likeliest tokens get even likelier, so output is more predictable. Above 1, the chances even out, so output is more varied and more prone to odd choices. Here are three possible next words with their scores:

import math

scores = {"blue": 2.0, "green": 1.0, "purple": 0.0}

def probabilities(temperature):
    e = {w: math.exp(s / temperature) for w, s in scores.items()}
    total = sum(e.values())
    return {w: round(v / total, 2) for w, v in e.items()}

print(probabilities(1.0))
print(probabilities(0.5))
print(probabilities(2.0))

At temperature 0 the model (nearly) always takes the top token, so the same prompt usually gives the same answer. Not always exactly the same: many services don't promise identical output for identical requests. Top-p (nucleus sampling) is another dial: sample only from the smallest group of tokens whose probabilities add up to p, so 0.9 cuts off the unlikely tail. Change temperature or top-p, not usually both. Some models, especially ones with a reasoning mode, fix or ignore these settings; check your model's docs.

Low temperature for extraction, classification, code and anything checked against a right answer. Higher for brainstorming and varied wording. A low temperature doesn't make the model right. It makes it consistently pick its first guess, right or wrong.

The other setting you'll always meet is the maximum output length. If the answer hits it, it stops mid-sentence, and the response says why it stopped (a stop or finish reason). Check that field instead of assuming the answer is complete.