Pretraining: How a Random Transformer Learns From Next-Token Prediction

Yesterday we finally assembled the machine.

A Transformer has embeddings, attention, MLP layers, normalization, residual connections, and many repeated blocks.

But there is an uncomfortable detail:

A newly initialized Transformer does not know language.

Its parameters begin without the useful structure we associate with a trained model.

So where does that structure come from?

The answer is pretraining.

Architecture is not knowledge

This distinction helped me a lot.

The Transformer tells us how computation is organized.

Pretraining determines what values the parameters acquire.

Think about a spreadsheet model with thousands of formulas but random numbers in every assumption cell. The structure can be correct while the result is useless.

For an LLM, we can simplify the situation as:

Transformer architecture
+ initially untrained parameters
+ enormous training dataset
+ prediction objective
+ optimization
= pretrained model

The architecture gives the model capacity. Training fills that capacity with useful statistical structure.

The basic game is surprisingly simple

For an autoregressive language model, the core task is next-token prediction.

Suppose the training text contains:

The capital of France is Paris.

After tokenization, imagine a simplified sequence:

The | capital | of | France | is | Paris | .

The model repeatedly receives earlier tokens and tries to predict the next one.

Conceptually:

The                         -> capital
The capital                 -> of
The capital of              -> France
The capital of France       -> is
The capital of France is    -> Paris

The real implementation processes training examples much more efficiently than literally running these as separate prompts, but this is the right mental model for the objective.

We already learned the machinery needed to train it

This is why the earlier Foundations lessons mattered.

The forward pass produces predictions.

The loss measures how wrong those predictions are.

Backpropagation computes gradients.

The optimizer uses those gradients to update parameters.

Then the process repeats.

text
-> tokens
-> forward pass
-> next-token probabilities
-> loss
-> backpropagation
-> parameter update
-> repeat

Pretraining is not a completely new kind of mathematics. It is the training loop we already studied, applied at enormous scale to a Transformer and a very large corpus.

What exactly is the model predicting?

At each position, the model produces logits over its vocabulary.

After normalization into probabilities, we can imagine something like:

context: "The capital of France is"

Paris       0.71
Lyon        0.05
France      0.03
the         0.02
...

During training, the dataset tells us the actual next token.

If the correct token is Paris, the loss penalizes the model when it assigns too little probability to Paris.

Across many examples, optimization changes the weights so the model becomes better at predicting tokens in contexts resembling its training distribution.

Why can predicting one token teach so much?

This initially seemed strange to me.

If the task is only “guess the next token,” why does the model learn grammar, facts, programming patterns, style, and some forms of reasoning?

Because accurate prediction often requires information about the structure that produced the text.

Consider:

2 + 2 =

Predicting the next token benefits from arithmetic patterns.

The capital of Japan is

Prediction benefits from factual associations.

if x is None:

Prediction benefits from programming syntax and common code structure.

The keys to the cabinet are

Prediction benefits from grammatical number agreement.

Nobody has to create a separate label saying:

lesson = subject-verb agreement

The prediction objective creates pressure to learn whatever internal representations help reduce prediction error.

But “the model memorized the internet” is too simple

Training data absolutely matters, and large models can memorize some training examples.

But memorization alone cannot explain useful behavior on every novel sequence.

The model also needs patterns that generalize across many examples.

For example, it may encounter countless variations of:

A is larger than B.
B is larger than C.

The useful parameter configuration is not necessarily a database row containing every future sentence. Training can produce distributed representations and computations that work across related contexts.

It is safer to think:

pretraining fits statistical structure in the data

rather than:

pretraining copies a giant text database into the weights

Neither slogan captures every behavior, but the first is a better starting model.

One pass over one sentence is not enough

Modern pretraining operates over enormous token counts.

The important scale variables include things such as:

number of model parameters
number of training tokens
amount of compute
batching strategy
learning-rate schedule
data mixture

Research on scaling laws and compute-optimal training showed that model size alone is not the whole story. The amount of training data and compute allocated to the model also matter.

This is one reason comparing models only by parameter count can be misleading.

A larger architecture is not automatically better trained.

An epoch is easier to understand than it sounds

In ordinary machine learning, an epoch means one pass through the training dataset.

For very large language-model training, datasets and token mixtures can be so large and complicated that practitioners often discuss progress directly in tokens or training steps.

For my mental model, the important part is simply repetition:

batch of tokens
-> calculate loss
-> update weights

next batch
-> calculate loss
-> update weights

again and again

Tiny parameter changes accumulate into a trained network.

Where does the training text come from?

A pretraining corpus can contain mixtures of sources such as web text, books, code, academic material, reference material, and other licensed or curated datasets, depending on the model.

The exact mixture is a major part of model development.

Raw volume is not the only issue. Developers also care about filtering, deduplication, quality, language balance, domain balance, contamination, and safety considerations.

Two models with similar architectures and parameter counts can behave differently because they were trained on different data mixtures and with different training recipes.

Pretraining is expensive; inference is a different phase

This distinction becomes important later in the series.

During pretraining we need:

forward pass
loss
backward pass
optimizer state
parameter updates

During ordinary inference we mostly need:

forward pass
next-token selection
repeat

That is one reason training a giant model and running a trained model are very different hardware problems.

A local GGUF model on my computer is normally the result of an expensive training process that already happened elsewhere. Quantization can then compress that trained parameter set for inference.

We will get to quantization later.

Pretraining does not necessarily produce the assistant we want

A pretrained language model has learned to continue text.

That does not automatically mean it will behave like a polished chat assistant.

If I type:

User: Explain bonds to a beginner.
Assistant:

a pretrained model may have learned enough conversational structure to continue it sensibly. But reliable instruction following usually involves additional post-pretraining stages.

Those can include supervised fine-tuning and preference/reinforcement-learning methods.

Our next lesson will start with the simplest next step: fine-tuning.

The mental model I am keeping

Before this lesson, I pictured a trained LLM as though someone had somehow “put knowledge into the Transformer.”

Now I have a more mechanical picture:

Start with a Transformer whose parameters are not useful yet.
Show it huge amounts of tokenized text.
Ask it to predict the next token.
Measure the error.
Backpropagate the error.
Adjust the parameters.
Repeat at enormous scale.

Over training, the parameters become increasingly useful for predicting language.

And because predicting language requires learning many regularities behind the text, the resulting model can acquire capabilities far richer than the wording of the training objective suggests.

Next: fine-tuning — how we take a pretrained model and deliberately change its behavior for a narrower objective.