Embeddings: How AI Turns Words and Images Into Geometry

Type "the cat sat on the mat" into a search engine, and it can hand you pages about "a feline resting on a rug" — no shared words at all. Somewhere under the hood, a machine has decided that "cat" and "feline," "sat" and "resting," "mat" and "rug" mean approximately the same thing. It didn't look this up in a dictionary. It learned it by turning words into points in space — and measuring how close together they land.

That trick is called an embedding, and it is arguably the single idea that made modern AI possible. Every large language model, every image generator, every recommendation engine that seems to "get" what you want is, underneath, doing geometry on meaning.

The Concept

An embedding is a way of representing something — a word, a sentence, an image, a song, a protein — as a list of numbers, called a vector. That's it. "Dog" might become something like [0.12, -0.44, 0.91, ...] with a few hundred or a few thousand entries. On its own, a single number in that list means nothing to a human. But the position the whole vector occupies, relative to every other vector, means everything.

The organizing principle is old, older than computers. In 1954 the linguist Zellig Harris proposed the distributional hypothesis: words that show up in similar contexts tend to have similar meanings. The British linguist J.R. Firth put it more memorably in 1957: "You shall know a word by the company it keeps." "Coffee" and "tea" show up near words like "cup," "morning," and "caffeine" far more often than "coffee" and "asteroid" do — and that pattern of co-occurrence is, it turns out, enough to reconstruct a surprisingly rich map of meaning.

For decades this was just a linguistic theory. It became a practical tool in 2013, when a team at Google — Tomáš Mikolov, Kai Chen, Greg Corrado, Ilya Sutskever, and Jeff Dean — published word2vec. The method trains a small neural network to do one of two simple jobs: guess a word from its surrounding context (CBOW), or guess the surrounding context from a word (skip-gram). The network never sees a definition. It just gets very good at prediction by reading billions of words. The byproduct — the internal numbers the network learns to represent each word — turned out to be the real prize. Words that behave similarly in text end up as vectors that point in similar directions.

Once you have vectors, you can do arithmetic on meaning. The canonical example, popularized alongside word2vec, is king − man + woman ≈ queen: take the vector for "king," subtract "man," add "woman," and the nearest resulting point in the space is "queen." It's a genuinely striking result, though it comes with an asterisk worth being honest about — the algorithm typically has to be told to ignore the original input words when searching for the nearest match (otherwise "king" minus "man" plus "woman" just lands closest to "king" again). The relationship is real, but it's a little more curated than the popular version of the story suggests.

Why It Matters

Embeddings matter because they solve a problem that plagued computing for decades: computers are exact-match machines, and human language and meaning are not. A keyword search for "affordable laptop" will miss a page that says "budget notebook" unless someone manually links the two terms. An embedding-based search doesn't have that problem, because "affordable laptop" and "budget notebook" land near each other in the vector space automatically, learned from data rather than hand-coded rules.

This is the engine behind retrieval-augmented generation (RAG), the technique that lets a chatbot answer questions about your company's internal documents or a codebase it was never trained on. The documents get embedded once and stored in a vector database; a user's question gets embedded on the fly; the system just finds the nearest neighbors and feeds them to the language model as context. No keyword matching required — the geometry does the work.

The same idea generalizes far past text. In 2021, OpenAI released CLIP (Contrastive Language-Image Pre-training), which trains a text encoder and an image encoder together so that a photo of a golden retriever and the caption "a golden retriever" end up as nearby points in the same shared space. That's what lets you type "red shoes on a wooden table" into an image search and get exactly that, or lets an AI image generator translate a text prompt into a picture — text and pixels are speaking the same numerical language.

It goes further still. Spotify and Netflix embed songs and shows based on listening and viewing patterns, so "users who behave like you" become "points near you" and recommendations fall out as a nearest-neighbor search. Content moderation systems embed images and compare them against embedded descriptions of banned material. Fraud detection systems embed transaction patterns to spot the ones that sit suspiciously far from the norm. Biologists embed protein and DNA sequences to find molecules that behave similarly despite looking different on paper. In every case, the pattern is identical: turn the thing into a point in space, and let distance stand in for similarity.

The Details

Practically, an embedding is produced by a neural network trained to be good at some task — predicting a masked word, matching an image to its caption, predicting the next token in a sentence — and then the network's internal representation, usually a middle layer, is repurposed as the "meaning" of the input. The training task is almost a pretext; what you actually want is the geometry that falls out of it as a side effect.

The number of dimensions varies by system. The original word2vec vectors typically used around 300 numbers per word. Google's BERT, a landmark 2018 language model, represents each token with 768 numbers. OpenAI's current text-embedding-3 models default to 1536 dimensions for the smaller version and 3072 for the larger one — though, thanks to a training technique called Matryoshka representation learning (named for the nesting Russian dolls), you can truncate those vectors down to as few as 256 dimensions and still outperform older, longer embeddings. More dimensions generally mean more nuance captured, at the cost of more storage and slower comparisons — an engineering trade-off, not a philosophical one.

Picture the space itself as an enormous, invisible galaxy with hundreds or thousands of axes instead of three. You cannot draw it, but you can navigate it: "distance" in this galaxy is usually measured by cosine similarity — the angle between two vectors — rather than straight-line distance, because what matters is the direction two concepts point, not how "long" their vectors happen to be. Synonyms cluster into tight neighborhoods. Related-but-distinct concepts form loose regions — "doctor," "nurse," "hospital," and "stethoscope" occupy the same general district without being neighbors. Analogical relationships sometimes show up as consistent directions: the vector you'd add to go from "France" to "Paris" is roughly the same vector that takes you from "Japan" to "Tokyo" — the model has, without ever being told, discovered something like the concept "capital of."

It's worth being clear-eyed about the limits here. Embeddings are learned from data, which means they inherit whatever biases exist in that data — early word embeddings notoriously encoded gender stereotypes (associating certain professions more strongly with one gender), and researchers have spent years since developing techniques to detect and correct for that. And the neat linear-algebra stories — "king minus man plus woman equals queen" — are demonstrations that work well as illustrations but don't fully capture how nonlinear and context-dependent modern embeddings (especially inside transformer models, where the same word gets a different vector depending on its sentence) really are. The map is genuinely useful. It is not the territory.

Takeaways

An embedding turns words, images, sounds, or almost anything else into a vector of numbers, positioned so that similar things land near each other in space.
The idea rests on the decades-old distributional hypothesis — you can infer meaning from context — but became computationally practical with word2vec in 2013.
Distance and direction in embedding space stand in for semantic similarity and relationships, which is what lets vector arithmetic occasionally produce results like king − man + woman ≈ queen.
The same technique now spans modalities: CLIP puts text and images in one shared space, and it underlies everything from RAG chatbots to music recommendations to protein search.
Embeddings are learned from real-world data, so they inherit real-world biases — a reminder that "the geometry of meaning" is only as fair as what it was trained on.

Resources: - Word2vec — Wikipedia - OpenAI: New embedding models and API updates - What Are Word Embeddings? — IBM