How many words is a token?

As an order of magnitude: one token is worth about three quarters of an English word, or roughly four characters. A 1,000-word English text therefore weighs around 1,300 tokens. Most other languages cost more, because their accented words and inflections are less well covered by vocabularies that were historically optimized for English.

Do all AI models split text the same way?

No: each family of models has its own tokenizer — OpenAI's recent models use a vocabulary of roughly 200,000 entries (code name “o200k”), Llama's splits differently. The same text therefore doesn't have the same number of tokens from one model to the next. Hence the value of checking with the tool of the provider you actually use.

What happens when I make a typo?

The misspelled word doesn't exist in the vocabulary, so the algorithm splits it into smaller fragments — sometimes letter by letter. The model then sees an unusual sequence, but since it crossed millions of typos during its training, it usually “repairs” the meaning effortlessly. It just consumes a few more tokens.

Do emojis count as tokens?

Yes, and sometimes several: an emoji is recorded, machine-side, as a small sequence of elementary building blocks, and the less common ones get split into two or three tokens. The same goes for Chinese or Japanese characters and rare alphabets. It's a direct consequence of frequency-based splitting: whatever is rare in the training corpus is paid for in fragments.

How can I reduce my token bill?

Four concrete levers: more concise prompts (every repeated word gets billed), a pruned conversation history rather than one sent in full, capped answers (APIs offer a setting, max_tokens, that cuts the output at a chosen length), and the prompt caching when you resend the same context — formidable against repetitive preambles. And if you serve multilingual users, budget for non-English text costing more.

Can I see the tokens in my own text?

Yes, in one click: OpenAI offers an online tokenizer (platform.openai.com/tokenizer) that colors each token of your text and counts them. It's the best way to build an intuition — paste in a sentence, a misspelled word and an emoji, and the first three sections of this article will illustrate themselves before your eyes.

AI Tokens: How a Model Splits Your Text

Since the very start of “The Dive”, I’ve been repeating that an AI reads neither your letters nor your words, but “tokens” — and I now owe you the full explanation. Welcome to Level 3, the level of the secret dictionary. Today, you’ll discover how your text is actually split, why that splitting was learned rather than decided, why it costs you money if you use these models in your projects, and why the innocent “strawberry” became the AIs’ favorite hazing ritual. On the math front, we climb one tiny notch above Level 2’s probabilities: all you’ll need is knowing how to count frequencies. Come on, down we go.

The essentials

The vocabulary of a recent model holds about 200,000 tokens; each token has a number, and that number is what the machine manipulates.
1 token ≈ ¾ of an English word ≈ 4 characters — that’s OpenAI’s own rule of thumb; English is the cheapest language to tokenize — French, German or Japanese cost more for the same meaning.
The splitting wasn’t decided by a human: it was learned by counting frequencies (the Byte Pair Encoding algorithm, BPE).
AI APIs (the developer access to the models) bill per million tokens, and a conversation’s “memory” (the context window) is counted in tokens.
You can visualize the splitting of any text with OpenAI’s tokenizer — the ideal companion to this article.

Why don’t AIs simply read words?

Because the two obvious choices are both bad. Reading letter by letter makes texts endless to process: every sentence becomes a very long sequence. Reading word by word demands an infinite dictionary: proper nouns, typos, new words, foreign languages… The token is the learned compromise between the two.

Let’s make the dilemma concrete. If the model read letters, “Why is the sky blue?” would be 20 characters — letters, spaces and punctuation — to process one by one — and our loop from Level 1, which recalculates everything at every unit, would become excruciatingly slow. If the model read whole words, what would it do with “Wooordpress” and its three o’s, with a username like “xXDragon_87Xx”, or with the name of a molecule? A dictionary of words is either infinite or full of holes — and a model that falls into a hole is a mute model.

The token solves both problems: frequent words (“the”, “house”, “cat”) earn their own full entry, and everything else splits into reusable fragments.

Nothing is ever unreadable: at worst, the algorithm falls back on the letters one by one.

How does the BPE algorithm build the dictionary?

By counting, patiently. But before counting, you need something to count in: the training corpus is the immense library of very real texts — web pages, books, news articles, code — that a model’s creators assemble to build it. That is where Byte Pair Encoding does its tallying (proposed for machine translation by Sennrich and colleagues, it has powered most large models since). Its recipe fits on one line: start from single characters, find the most frequent pair of neighbors, weld it into a new fragment, start over — tens of thousands of times.

It’s the only new mechanism at this level, so let’s unroll it for real. A real corpus weighs terabytes — the equivalent of millions of books; let’s shrink ours down to a single sentence, so we can count by hand: “the cat naps by the catnip”.

At the start, everything is isolated characters. The algorithm then counts every pair of neighbors: “t+h” appears 2 times (the two “the”), “h+e” 2 times, “c+a” 2 times (“cat”, “catnip”), “a+t” 2 times — all the others, just once. Four pairs are tied; it welds one, say “c+a”, which becomes the fragment “ca” everywhere. It recounts: “ca+t” comes out ahead, new merge, and “cat” enters the dictionary. That’s the whole secret — count, weld, recount. (Ties, unavoidable on six words, are vanishingly rare on an entire library.)

BPE on a toy corpus: no grammar rules, no linguist — nothing but frequencies. The fragment “cat”, shared by “cat” and “catnip”, emerges on its own.

The most striking thing about this algorithm is its ignorance: it doesn’t know what a word, a syllable or a language is — it counts, and the structure of language emerges from the counting. Applied to terabytes of real text, BPE rediscovers our prefixes (“anti-”, “re-”), our suffixes (“-tion”, “-ing”), our common words in full, and the most useful pieces of dozens of languages. It’s a foretaste of the series’ grand theme: nobody programs these machines’ knowledge — we program methods for it to emerge.

What does your text look like, as the AI sees it?

Like a sequence of numbered fragments, with a few surprises: spaces belong to the tokens (“cat” and “·cat” — the middle dot · stands for the space — are two different entries), capitals and accents change the splitting, and two sentences of the same length can differ twofold in cost.

Take the sentence “I love the strawberries from my garden”. Seen by the tokenizer, it looks like this:

What the model actually sees: eight numbers. “strawberries” doesn't exist as one block — it's “·straw” + “berries”. Try it on OpenAI's tokenizer; the real splitting varies from model to model.

Three details of this diagram are worth the detour. The space is not a separator, it’s part of the token: “·garden”, with its leading space. Case changes everything: “Hello”, “hello” and “HELLO” are three different splittings — one of the reasons a text IN ALL CAPS can be handled slightly less well. And “strawberries” is in two pieces, which brings us to the promised hazing.

Why does ChatGPT fail to count the letters in “strawberry”?

Because it sees no letters. “strawberry” reaches it as one or two numbered blocks — opaque chips. Counting the “r”s in there is like asking you how many brush strokes are in a painting you’ve only seen in a photo: the information exists upstream, but not in what reaches your eyes.

This is THE answer I’ve owed you since Level 1. When the question “how many r’s in strawberry?” went around the world — there are three, and the best models kept finding two — everyone laughed at the AIs’ “stupidity” — when it actually reveals their architecture. The model never had access to the letters: asking it to count letters means quizzing it about a reality it has literally never seen. If it still often answers correctly anyway, it’s because it has read texts talking about how words are spelled — second-hand knowledge, like yours about the brush strokes of the Mona Lisa.

The same explanation lights up a whole family of blunders: writing words backwards, counting syllables, making anagrams, sticking exactly to an imposed character count. Every time you see an AI stumble on a letter game, you’ll now be able to whisper: the tokens.

Why do tokens cost you money?

Because the token is the AI world’s unit of account: providers bill neither by word nor by page, but by token processed — those you send to the model (the “input”) as well as those it generates (the “output”), priced per million. And since the model rereads the entire exchange before each word — that’s the loop from Level 1 —, a conversation’s “memory”, called the context window, is measured in tokens too; we come back to it at the end of this section.

This is where the subject leaves curiosity behind and reaches the wallet, and I speak from experience: my own tools talk to these models through their API — the direct access providers sell to developers, bypassing applications like ChatGPT — and the invoice at the end of the month is literally a token counter. It taught me four reflexes I’m passing on to you:

Trim the repetitive prompts. A verbose preamble sent a thousand times a day is a budget melting away for nothing.
Cap the answers. Output costs more than input — the rates are just below: “answer in three sentences” is also a cost instruction.
Start a fresh conversation when the topic changes. A never-ending conversation costs more with every message — you’re about to see why.
Watch the language. If you’re reading me in English, you’re playing on the easiest setting: this site lives in two languages, and at identical instructions, my French prompts consistently consume noticeably more tokens than their English counterparts — the vocabulary, carved out of mostly English-speaking corpora, favors English. English is the cheapest language to tokenize; French, German or Japanese pay a surcharge for the very same meaning — worth knowing before you budget a multilingual product.

That surcharge can be measured. Let’s bring back the cat from Level 2 in a twenty-word English sentence — “The cat fell asleep on the old radiator because the living room was cold and the couch was already taken” — and run its variants through the tokenizer:

The same twenty-word sentence, in tokens (orders of magnitude)	Tokens
The original English version	~26
Its French translation	~35
Its Japanese translation	~40
The English version riddled with typos	~45

How much do these tokens cost? Take a real public rate, that of Claude Sonnet, Anthropic’s mid-tier model — one of the major providers; the orders of magnitude are comparable across competitors. As of mid-2026: 3 dollars per million input tokens, 15 dollars per million output tokens — generating costs five times more than reading, hence the “cap the answers” reflex. At that rate, having the model read this entire article comes to a cent; a doorstopper like The Count of Monte Cristo, to two dollars in its English translation — and about two and a half in the French original, the French surcharge at work.

One trap, though: those dollars are not paid once and for all. Every new answer runs the whole exchange through the input again — novel included. Ask ten questions about Monte Cristo in the same conversation, and the book gets re-billed ten times. Providers soften this with “prompt caching” — tokens already seen are re-billed at a fraction of the price, about one tenth at Anthropic — but the fresh-conversation reflex keeps all its point.

That leaves the context window, promised at the top of this section. It’s the ceiling of that memory: your entire exchange — questions, answers, pasted documents — must fit within a token budget set by the model. Today’s large-window models accept up to a million tokens — enough to hold all of Monte Cristo; the more compact ones, far less.

What happens if the conversation overflows anyway? The beginning of the exchange silently drops out of the window. If a long conversation seems to “forget” your first instructions, it’s not distraction: it’s a token budget running dry.

Where does the dive stand?

Three levels explored: the journey of your message, the probability lottery, and now the splitting into tokens. Let’s check the gauge before continuing the descent.

Three levels, and already enough to explain half of the AIs' oddities to your friends and family. The rest of the descent heads into the internal machinery.

Ready to descend to Level 4?

Let’s recap: your text becomes a sequence of numbered fragments, split by an algorithm that learned its dictionary by counting frequencies. Not letters, not words: tokens — which is why AIs juggle whole essays yet stumble over “strawberry”.

But admit there’s still a mystery, and it’s enormous: the token “·garden” carries the number 20,890. A number. How on earth can a number “know” that a garden is closer to a vegetable patch than to a traffic jam? That’s the subject of Level 4, embeddings — the one where numbers start carrying meaning, with the most famous discovery of the field: king − man + woman ≈ queen. The article arrives next week.

Until then, play with OpenAI’s tokenizer, and if you’re joining the series along the way, start with Level 1. The comments are open, in a spirit of openness and kindness — your questions feed the next levels.

– blaminhor