AI Temperature: How a Model Picks Its Words
Every word an AI writes comes from a loaded lottery: meet temperature, the setting that keeps a model wise or sets it loose — Level 2 of “The Dive”.
Welcome to Level 2 of “The Dive”. At Level 1, we followed your message’s complete journey: off to the data center, the splitting into tokens — those word fragments that serve as the model’s reading units — and that loop that builds the answer word after word. But I left a door ajar: at every turn of the loop, the model doesn’t really “choose” a word — it draws lots. Today, we descend for a close look at that lottery. By the end of this article, you’ll know why an AI tuned too “wise” drones on, why an AI tuned too “hot” goes off the rails, and what that mysterious setting developers call temperature actually does. One single math notion on the program, I promise: the percentage. Gear up, down we go!
The essentials
- Before each word, the model produces a list of percentages that sum to 100%: the probability distribution.
- Systematically taking the most probable word produces flat texts that go in circles — a flaw documented by research under the name of text degeneration (Holtzman et al., 2019).
- Temperature deforms the list before the draw: at 0, the model almost always takes the most probable word; at the maximum, the choice becomes nearly random. The maximum depends on the provider: 2 at OpenAI, 1 at Anthropic.
- Adjustable through the API (the developer access) and the Playground (its online test console), not in the ChatGPT app.
- Level 2 of “The Dive” series — one single mathematical notion: the percentage.
What is a probability, for a machine that writes?
It’s a numbered bet. Before each word, the model assigns every token in its vocabulary a number between 0 and 100%, and all those numbers together sum to exactly 100%: that’s what we call a probability distribution. Generating a word means drawing lots from that distribution.
And this is where Level 1 gains its full depth: the loop that writes your answer is not a chain of decisions, it’s a chain of draws from bags recomposed at every word. Each token drawn changes the composition of all the bags that follow.
Where do these percentages come from?
From training. The model spent months adjusting its hundreds of billions of parameters so that its bets would match, as closely as possible, the real texts it was shown. But careful: it doesn’t count stored sentences — it learned to generalize, and that makes all the difference.
The natural reflex is to imagine one immense tally: “across all the texts on the web, after the capital of France is, you find Paris 96 times out of 100.” It’s a good starting intuition… and it’s wrong in one crucial detail. Take this sentence: “The cat fell asleep on the”. It’s utterly ordinary, yet odds are that no text in the training data contains it exactly — the possible combinations of words outnumber, by very far, everything that has ever been written. And yet the model proposes “couch”, “radiator”, “keyboard” without a moment’s hesitation…
How? Because it has generalized from millions of neighboring sentences: cats falling asleep, things resting “on the”, cozy radiators. Its percentages are not memories, they are syntheses — which is why it answers in a flash sentences that nobody has ever written. How that synthesis physically settles into the parameters is the big subject of Level 7; for now, simply remember that the bag of tokens is manufactured, not photocopied.
Why not always take the most probable word?
Because that produces dead text. If the model systematically took candidate no. 1, its answers would be stiff, flat — and above all, they would end up going in circles. The flaw is so well known that researchers gave it a name: text degeneration.
The objection raises itself, and it’s perfectly logical: “if it knows the best word, let it take it!” Except here is what really happens when you try — researchers have documented the phenomenon, and here’s the kind of text it produces:
Thank you for your trust. Thank you for your trust. Thank you for your trust.
The mechanism behind this spiral is perversely logical: once a sentence has been written twice, writing it a third time becomes even MORE probable — repetition feeds on itself. The most “probable” text word by word is not the most natural text as a whole; it’s a paradox, but our real human sentences regularly hold surprises, slightly less expected words that give them all their flavor. A text that never surprises resembles nothing we actually write.
Hence the lottery: by regularly giving candidates no. 2, no. 5 or no. 12 their chance, you get living text. All that remains is dosing that randomness — and that is exactly the temperature’s job.
Temperature: the knob that sets the daring of the draw
Temperature deforms the distribution before the draw. Low, it deepens the gaps: the favorite crushes everything, the model becomes predictable. High, it narrows the gaps: the outsiders get a real chance back, the model becomes daring — then incoherent if you push too far. On OpenAI’s scale, it runs from 0 to 2; at Anthropic (Claude), from 0 to 1 — same dial, different graduations.
The name comes from physics (particles get more agitated as heat rises), but the everyday image works just as well: it’s the dial between the colleague who always gives the same answer and the one who goes off in every direction. Look at what temperature does, concretely, to our bag of tokens — filled, this time, for the Level 1 sentence “This weekend, I feel like…”:
Technically, there is nothing magical about it: before converting the model’s scores into percentages, you divide them by the temperature — which deepens the gaps when it’s small, and erases them when it’s large. (The real machinery of that conversion, the famous “softmax”, you’ll get it with its calculation at Level 6 — each thing at its level.)
How I set the temperature in my own projects
It’s a setting I concretely handle every week: my tools call these models through their API, where the dial is within reach. And my rule fits in one sentence: the temperature follows the task’s tolerance for surprise. The more reliability a task demands, the lower I set the dial; the more fresh ideas it calls for, the higher.
A lived example that served me well as a lesson: one of my scripts extracted information from texts to file it as JSON (structured data). Set to the default temperature, it returned slightly different results on every run — sometimes a field phrased differently, sometimes a neighboring category. Impossible to test seriously. Temperature at 0: the script became reproducible overnight — same input, same output. Conversely, when I’m hunting for article angles or project names, I turn the dial up: surprise is precisely what I’ve come for.
| Task | Temperature (OpenAI scale, 0–2) | Why |
|---|---|---|
| Data extraction, classification | 0 – 0.2 | Reproducible, testable — surprise is a bug |
| Code generation | 0 – 0.3 | Only one correct syntax, no need for daring |
| Factual answers, customer support | 0.2 – 0.5 | Stable, with a minimum of phrasing variety |
| Assisted writing | 0.6 – 0.9 | Natural flow and rhythm, without going off the rails |
| Brainstorming, fiction | 1 – 1.3 | Surprise is exactly what you’re paying for |
Two clarifications to finish. First, the scale depends on the provider: 0 to 2 at OpenAI, 0 to 1 at Anthropic — check the documentation before transposing my numbers. Second, in the ChatGPT app, this dial isn’t offered to you: OpenAI has set it for you. But you can get a cousin effect through the prompt — “answer in a strictly factual way” or “suggest unexpected ideas” shift the probabilities in the same direction, with the means at hand from Level 1: by changing the beginning of the document the model extends.
Where does the dive stand?
Level 2 explored: you now know that every word an AI writes comes out of a loaded lottery whose daring the temperature sets. Combined with Level 1 — the journey of your message and the prediction loop — you already hold a view of the machine’s complete behavior.
Ready to descend to Level 3?
Let’s recap this level: at every word, a probability distribution, a lottery draw, and a dial — the temperature — that decides whether the draw is wise or daring. No thought, no plan: a loaded lottery with an immense talent for loading.
But come to think of it… those “tokens” we’ve been drawing lots on for two articles now, why aren’t they simply words? Why does “antidisestablishmentarianism” cost more to process than “cat”? And why does asking ChatGPT to count the letters in “strawberry” put it in difficulty, when it writes whole essays? At Level 3, we open the AIs’ secret dictionary — and you’ll never look at a text the same way again. The article arrives next week.
Until then, reread Level 1, the journey of your message if you’re joining the series along the way — each level builds on the previous ones. And as always, the comments are open, in a spirit of openness and kindness: your questions directly feed the next levels.
– blaminhor
FAQ
What is top-p, often offered next to temperature?
A second filter on the draw: instead of considering every possible word, you keep only the best candidates whose cumulative probabilities reach p (90%, for example), and draw lots within that restricted circle. It cuts off the long tail of absurd words while keeping some variety. Temperature and top-p combine.
Is there a top-k setting on top of top-p?
Yes: top-k limits the draw to the k best-ranked candidates (the top 40, for example), whatever their cumulative probability weight. Anthropic's API (Claude) exposes this parameter under the name top_k; OpenAI's API doesn't offer it and sticks to temperature and top-p.
Temperature or top-p: which one should you adjust first?
Temperature: it's the first-reflex setting, the one whose effect is the most intuitive. OpenAI's documentation actually recommends changing one or the other, but not both at once. So start by adjusting the temperature, and only touch top-p if it isn't enough for your needs.
Does an AI use true randomness to draw lots?
No: like almost everything in computing, it uses a pseudo-random generator — a computed sequence of numbers, unpredictable in practice but fully determined by its starting point (the “seed”). Some APIs even let you pin that seed to replay a draw identically, which is precious for testing.
Does temperature change the price of my API calls?
No. Billing is per token processed, on input and output alike, and temperature doesn't alter that rate. At most, it can indirectly influence the length of the answer — and therefore the number of generated tokens — but the unit price stays identical, whether the dial sits at 0 or at the maximum.
Comments