AI Embeddings: When Numbers Carry Meaning
How does token no. 20,890 “know” that a garden resembles a vegetable patch? Enter embeddings, the geometric space where words live. Level 4 of The Dive.
At Level 3, we left the dive on a mystery I owe you: the token “·garden” carries the number 20,890 — a mere entry number in a dictionary. How can a number “know” that a garden resembles a vegetable patch, that strawberries grow there (whole ones, this time) and that it has nothing to do with a traffic jam? Welcome to Level 4: the level of embeddings, where meaning literally becomes geometry. It’s also the level of the contract I made with you in the very first article: by the end of this one, you’ll have an accurate mental model of what an AI is — more accurate than the vast majority of what gets written on the subject. On the math front, we climb one notch, gently: coordinates, like on a map.
The essentials
- An embedding = a list of numbers = the coordinates of a point; every token in the vocabulary has its own.
- These spaces have hundreds to thousands of dimensions (12,288 for GPT-3, 16,384 for Llama 3.1 405B).
- The position isn’t programmed: it’s learned during training, by observing which words get used in the same contexts.
- The founding discovery (word2vec, 2013): directions of the space encode relations — gender, capital cities, verb tense.
- Embeddings serve far beyond chatbots: semantic search, recommendations, duplicate detection, RAG.
How can a token number carry meaning?
It can’t — that’s the whole problem. The number 20,890 is arbitrary: token 20,891 has no reason to be its cousin. The researchers’ solution: replace each number with a scorecard of numerical characteristics, a list of numbers that describes the token — and that list is the embedding.
Real embeddings do exactly that, with two differences of scale. First, not three axes but thousands — 12,288 dimensions for GPT-3 (OpenAI, 2020), 4,096 for Mistral 7B (Mistral AI, 2023), 7,168 for DeepSeek-V3 (DeepSeek, 2024), 16,384 for Llama 3.1 405B (Meta, 2024). Recent closed models — GPT-4, Claude, Gemini — no longer publish that figure, but the order of magnitude is the same: enough to encode nuances that our three naive questions flatten. Second — and this is the key point: nobody chooses the axes. No human ever decreed an “edible” axis. The dimensions emerge from training, and most of them correspond to no nameable concept — we’ll come back to that below.
What does the map of meaning look like?
Like constellations: words from the same semantic family form clusters — the animals together, the vehicles together, the places where things grow together. Here is a two-dimensional projection, the very flattened shadow of a space that has thousands:
Take a second to appreciate what this diagram accomplishes: “near” and “far” — purely geometric notions — have started to mean “synonym” and “unrelated”. When an AI “understands” that your question about strawberry plants belongs to gardening, nothing mystical is going on: points are neighbors in a space. Token no. 20,890 still knows nothing; but its position says it all.
King − man + woman ≈ queen: what does this equation really say?
That the relations between words are directions of the space. The displacement that leads from “man” to “king” — call it “take the crown” — is almost the same as the one that leads from “woman” to “queen”. In 2013, the word2vec team showed you could do this arithmetic on words, and the entire field changed.
And it doesn’t stop at royalty. The same arrows exist for capital cities — start from “Paris”, subtract “France”, add “Italy”: you land near “Rome”. For verb tense: “walked” is to “walk” what “sang” is to “sing”. For sizes, feminine forms, nationalities… The space hasn’t just sorted the words by family: it has aligned the relations themselves, without ever being asked to.
This arithmetic is not infallible — hence the “≈” I force myself to write. It fails on rare relations, and embeddings also inherit the dubious associations of their corpora (well-documented biases — we’ll come back to them at Level 8). But as an intuition of what “understanding” means for a machine, nothing has ever matched it.
Who decides where each word sits?
Nobody — and that’s the most dizzying part. The coordinates are parameters of the model, adjusted millions of times during training to improve next-word prediction. Words that are interchangeable in sentences mechanically end up neighbors: the map organizes itself.
The mechanism rests on an observation linguists made long before machines: you shall know a word by the company it keeps. “Coffee”, “tea” and “juice” appear in the same sentences — “I drink a ___ in the morning”, “a cup of ___”, “this ___ is too hot”. A model trained to predict the missing word therefore has every interest in giving them nearby coordinates: whatever it learns about one will serve for the others.
Which leaves one thing to see: a single one of those famous adjustments in action. On day one of training, the coordinates are drawn at random: the map is a shapeless cloud, where “tea” might sit between “truck” and “jealousy”. The model reads “I drink a ___ in the morning”, guesses wildly — say “lamppost”, nothing stops it yet — then discovers the right answer. The correction is immediate and minuscule: the coordinates that contributed to the failed guess get moved by a hair’s breadth, in the direction that would have made “tea” slightly more believable. That’s all — one point sliding imperceptibly across the map. But multiply that gesture by trillions of sentences, and the drinks clump together, the professions clump together, the emotions clump together — the map of meaning emerges from the simple fact of being useful to prediction. As for how the machine computes the direction and the size of each nudge — which way to slide, and by how much — that’s the grand story of Level 7.
Remember Level 3, where BPE discovered our prefixes just by counting? Same philosophy, one floor up: we don’t program knowledge, we program the conditions for it to emerge.
What do embeddings look like in real life?
Like the most reused building block in all of modern AI. Whenever you come across a search that understands what you meant, a relevant suggestion or a “similar articles” list that hits the mark, chances are embeddings are working backstage.
A concrete example I rely on daily, as a website builder: in SEO, a quiet and underrated trap is called cannibalization — two articles that unknowingly target the same search intent. You think you’re grabbing two spots on Google; in reality you force it to hesitate between your two pages, you dilute their signals, and neither ranks as high as a single, solid page would. Comparing the embeddings of my topics gives me a detector: if two titles in my article list have points that sit too close in the space, they’re telling the same story — and I should merge rather than publish twice. Searching by meaning instead of keywords: that’s exactly this level’s promise, kept.
| Similarity between two texts (illustrative, from 0 to 1) | Score |
|---|---|
| “running shoes” ↔ “sneakers for jogging” | 0.91 |
| “strawberry pie recipe” ↔ “dessert made with garden fruit” | 0.74 |
| “strawberry pie recipe” ↔ “canceling a mobile phone plan” | 0.08 |
| “the cat is sleeping” ↔ “the feline is slumbering” | 0.88 |
| “the cat is sleeping” ↔ “le chat dort” | 0.85 |
The last row of the table deserves a pause: multilingual models place “the cat is sleeping” and its French translation in the very same spot of the space. The point doesn’t encode words: it encodes an idea — language becomes nothing more than a costume.
Where does the dive stand?
Ready to descend to Level 5?
Let’s take stock of the mental model, because the “raw material” side is now complete: your text becomes tokens (Level 3), each token becomes a point in a space where geometry carries meaning (this level), and a loop predicts the next word by producing probabilities (Levels 1 and 2). You know what goes in, what circulates and what comes out. All that’s missing is the engine.
Because between the embedding that goes in and the probability that comes out, something happens — the “big calculation” I’ve been drawing as a blue box since Level 1. At Level 5, we finally open it: you’ll meet the artificial neuron, its weights, its layers — and the series’ first real equation, so simple you’ll wonder why everyone makes such a fuss about it. The article arrives next week.
If you’re joining the series along the way, everything starts at Level 1, the journey of your message. And the comments remain open, in a spirit of openness and kindness — several reader questions have already reshaped the levels to come.
– blaminhor
FAQ
How many dimensions does an embedding have?
From a few hundred to several thousand, depending on the model: 300 for word2vec's historic embeddings, 12,288 for GPT-3, 4,096 for Mistral 7B, 16,384 for Meta's Llama 3.1 405B. More dimensions means more possible nuance — but also more computation for every generated word.
Are embeddings the same from one AI to the next?
No. Each model learns its own during training, and two models place words differently in their respective spaces — the coordinates of “cat” at OpenAI have nothing to do with Claude's. What does look alike from one model to the next is the overall structure: the neighborhoods and the broad directions of meaning.
Can you recover the original text from an embedding?
Not directly: it's lossy compression, with no reverse gear provided. But “embedding inversion” research (Vec2Text, Morris et al. 2023 — arxiv.org/abs/2310.06816) reconstructs a good part of the text — up to 92% of short sentences recovered word for word. So treat your embeddings as sensitive data, not as anonymous numbers.
Can you visualize a 12,000-dimension space?
No, and nobody can: beyond three dimensions, our visual intuition gives up. The 2D maps we use (including in this article) are projections — flattened shadows that preserve neighborhoods as best they can, at the cost of distortions. Useful for intuition, but keep in mind that the original is immensely richer.
Does a word always keep the same embedding?
Initially, yes: the token “date” has a single entry in the table. But in modern models, that initial embedding is then refined by the sentence's context, layer after layer — so “date” ends up with different coordinates in “my date is ripe” and “my date is running late”. That refinement is the attention mechanism, our Level 6.
Comments