Embeddings: How AI Turns Meaning Into Coordinates

In the last lesson, machaMochaLatte explained how backpropagation computes a gradient for every trainable weight. That gives us a mechanism for learning, but it leaves a basic language-model question: neural networks calculate with numbers, while words are not numbers. What enters the network?

The answer begins with embeddings.

An embedding turns an object into a list of numbers called a vector. The important part is not merely assigning numbers. The coordinates are learned so that the resulting geometry becomes useful to the model.

An ID is not an embedding

My first instinct was to think of an embedding as an ID. A database might store 1 = bank, 2 = mortgage, and 3 = banana. Those numbers identify the words, but their numerical distances do not express meaning.

An embedding is different. Imagine learned coordinates such as:

bank      [0.8, -0.2, 0.5]
mortgage  [0.7, -0.1, 0.6]
banana   [-0.4,  0.9,-0.3]

Now bank and mortgage can occupy nearby parts of a learned space while banana sits elsewhere. Real models use far more dimensions, and individual coordinates usually do not have simple human labels.

From one-hot to dense vectors

A vocabulary can be represented with one-hot vectors. With four items, each item gets its own position and every other position is zero. That representation preserves identity but does not contain learned similarity.

An embedding replaces that sparse representation with a dense vector such as [0.31, -0.82, 0.14, ...]. A real model may use hundreds or thousands of dimensions.

The embedding table is a trainable tensor

This connects directly to our earlier lessons about tensors and parameters. If a vocabulary has 50,000 tokens and each embedding has 1,024 dimensions, the model can store an embedding matrix shaped roughly 50,000 x 1,024.

Each row corresponds to one token vector. A token ID selects the corresponding row:

token ID
-> row lookup
-> embedding vector

This was the point where embeddings stopped feeling mysterious to me. The embedding layer is a trainable tensor.

How the coordinates are learned

During the forward pass, an embedding contributes to later calculations and eventually to the loss. Backpropagation computes how changes to the used embedding values would affect that loss, and the optimizer updates them.

token
-> embedding
-> network
-> prediction
-> loss
-> backpropagation
-> embedding update

After many examples, useful relationships can emerge because representations that help solve related prediction problems receive related learning signals. Nobody has to manually program a rule saying mortgage should be more related to loan than to banana.

Similarity is only part of the story

It is useful to say that similar words can be near each other, but that is incomplete. The model is not optimizing a dictionary definition of similarity. It learns representations useful for its training objective.

The space can encode overlapping relationships involving topic, syntax, entities, usage patterns, and context. In modern language models, the representation also changes as it moves through transformer layers. The initial token embedding is a starting point, not the token’s final contextual meaning.

Geometry becomes computation

Once objects are vectors, relationships can be compared mathematically. Euclidean distance asks how far points are from one another. Cosine similarity compares their directions.

The formulas can wait. The important idea is that embeddings turn relationships into geometry that neural-network operations can use.

The dimensions themselves are usually difficult to interpret. Unlike spreadsheet columns named revenue or expenses, embedding dimension 17 does not necessarily mean one clean concept. Information is distributed across combinations of dimensions.

Embeddings are not only for language

The same idea can represent tokens, sentences, documents, images, users, products, and other objects. Recommendation systems can learn user and product vectors. Search systems can embed queries and documents. Multimodal systems can learn representations that connect text and images.

The common pattern is:

complex object
-> vector
-> useful geometry
-> computation

The missing step: tokenization

I have been writing token ID -> embedding row, but where does the token ID come from?

If I type a sentence, a modern language model does not necessarily receive one ID per word. A tokenizer breaks text into tokens according to a vocabulary and encoding procedure.

So the input path is closer to:

text
-> tokenization
-> token IDs
-> embeddings
-> transformer

We learned embeddings first because now we know what those token IDs are ultimately used to retrieve.

The next lesson moves one step backward and asks: what exactly is a token, and why does tokenization sometimes split words in surprising places?