Inference: What Happens After You Press Send to a Trained Language Model

Yesterday we ended with a specialized trained model.

It has already gone through some combination of:

pretraining
-> fine-tuning
-> trained parameters

Now suppose I open a chat box and type:

Explain what a bond yield is in simple language.

What happens next?

This is inference.

For me, the easiest way to understand inference is to stop thinking about the model as “learning” and start thinking about it as using the numbers it already learned.

Training changes the model; inference uses the model

This is the most important distinction.

During training, we have a loop like:

forward pass
-> loss
-> backpropagation
-> gradients
-> update parameters

During ordinary inference, we do not need the training part of that loop.

We mainly do:

input
-> forward pass
-> token probabilities
-> choose next token
-> repeat

The weights are usually fixed.

No loss needs to be calculated against a correct answer.

No gradient descent step needs to change the parameters after every response.

That difference sounds small, but operationally it is huge.

My finance analogy: a completed model versus rebuilding the model

Imagine a large financial forecasting spreadsheet.

During development, analysts may repeatedly change assumptions, fit parameters, test formulas, compare errors, and rebuild parts of the model.

That is like training.

Once the model is finalized, someone may type in:

interest rate = 4.5%
revenue growth = 7%

and read the forecast output.

The spreadsheet is not relearning its formulas each time someone opens it.

It is applying the existing formulas and parameters to new inputs.

Inference is similar:

trained model + new prompt -> output

Step 1: the prompt becomes tokens

The model does not directly process the sentence as human-readable words.

We already learned that the tokenizer converts text into token IDs.

A simplified prompt might become:

"Explain what a bond yield is."

then:

[4312, 922, 264, 11877, 14622, 374, 13]

The actual IDs depend on the tokenizer.

Those token IDs are integers from the model’s vocabulary.

So the first step is:

text
-> tokenizer
-> token IDs

Step 2: token IDs become vectors

The model cannot do useful neural-network computation on the integer ID 4312 by itself.

The embedding table converts each token ID into a vector.

So:

token IDs
-> embeddings
-> vectors

Those vectors then enter the Transformer blocks we learned about earlier.

At this point, nothing fundamentally new has appeared.

Inference is using the same architecture whose pieces we already studied:

embeddings
attention
MLP layers
normalization
residual connections

Step 3: the Transformer performs a forward pass

The prompt representations flow through the model layer by layer.

Very roughly:

input embeddings
-> Transformer block 1
-> Transformer block 2
-> Transformer block 3
-> ...
-> final hidden representation

Each block transforms the representation using the model’s learned parameters.

Those weights may contain billions of numbers, but during ordinary inference we are mostly reading and applying them rather than updating them.

That is why inference is often described as a forward pass.

Step 4: the model produces logits for the next token

At the end of the forward pass, the model produces scores over the vocabulary.

These scores are called logits.

Suppose our vocabulary contains 100,000 possible tokens.

Then the model may produce something conceptually like:

"A"          8.2
"Bond"       6.1
"The"        5.9
"yield"      4.8
"Banana"    -2.7
...

There is a score for every possible next token.

Higher logits generally mean the model currently considers that token more plausible.

But logits are not yet the final chosen token.

Step 5: logits become a next-token decision

The runtime transforms those logits into a probability distribution and then chooses one next token according to a decoding rule.

The simplest rule is greedy decoding:

choose the token with the highest score

But modern language-model generation often uses temperature, top-p, top-k, or related sampling controls.

We will study those in the next lesson.

For now, I only need this picture:

logits
-> next-token selection rule
-> one chosen token

Suppose the chosen token is:

"A"

The output has now begun.

Step 6: append the new token and run again

This is the part that makes language-model generation feel different from an ordinary classifier.

A classifier might do:

input -> one prediction -> finished

An autoregressive language model does:

prompt
-> predict one token
-> append token
-> predict another token
-> append token
-> predict another token
-> ...

If the prompt is:

Explain what a bond yield is.

then generation might develop like this:

Explain what a bond yield is.
A

then:

Explain what a bond yield is.
A bond

then:

Explain what a bond yield is.
A bond yield

and so on.

Each newly generated token becomes part of the context for predicting the next token.

Autoregressive means the output feeds the next step

This gives me a clean definition.

For a causal autoregressive language model:

P(token 1 | prompt)
P(token 2 | prompt, token 1)
P(token 3 | prompt, token 1, token 2)
...

Every next-token prediction depends on what came before it.

The model is not generating the entire paragraph in one single prediction.

It is building the response token by token.

That is why generation latency can depend strongly on output length.

A 500-token answer requires many more sequential decoding steps than a 20-token answer.

The first pass and later passes are not operationally identical

Suppose my prompt contains 2,000 tokens.

Before generating the first new token, the model must process that prompt context.

After the first generated token, the runtime already has useful intermediate information from the earlier tokens and can avoid recomputing everything from zero in an efficient implementation.

Runtime engineers often separate these phases conceptually into:

prompt processing / prefill

and:

token-by-token decode

I do not need the memory details yet.

The important intuition is:

processing the initial prompt
is a different workload from
generating one additional token

Later we will connect this directly to the KV cache.

Why prompt processing can be parallel but generation is sequential

Inside one prompt, many token positions can be processed together through matrix operations.

But once generation starts, token 101 cannot be selected until token 100 has already been selected, because token 101 depends on it.

So there is an unavoidable sequential dependency:

token n
-> becomes part of context
-> compute token n+1

This is one reason LLM serving has two very different performance metrics.

You may see terms such as:

TTFT = time to first token
TPOT = time per output token

TTFT is influenced heavily by prompt processing and queueing.

TPOT reflects the repeated decode steps after generation has started.

We do not need to benchmark them today; I only want to understand why they are different quantities.

Inference does not mean “the model searches its memory”

This mental model caused me trouble at first.

It is tempting to imagine a model receiving a question and searching an internal database for a stored answer.

But ordinary model inference is better described as repeated numerical computation.

The prompt activates patterns through the model’s learned parameters.

The network transforms vectors layer by layer and produces a probability distribution over the next token.

Then it repeats.

So instead of:

question
-> search database in weights
-> retrieve sentence

I picture:

question tokens
-> neural-network computation
-> next-token distribution
-> one token
-> repeat

That does not mean models never memorize training data. They can.

It means the runtime mechanism itself is not a database lookup API.

What remains fixed during ordinary inference?

Usually:

model architecture
trained weights
vocabulary
tokenizer

What can change from request to request includes:

prompt
system instructions
conversation history
sampling settings
maximum output length

So two users can get very different answers from the same model weights because their context and decoding settings differ.

The model does not need to be retrained between those requests.

Training memory and inference memory are different

During training, the system may need to keep information required for backpropagation and optimization, such as:

activations
gradients
optimizer state

During ordinary inference, gradients and optimizer state are generally unnecessary.

That can make inference much cheaper than full training.

But inference still needs substantial memory for things such as:

model weights
runtime buffers
current context state

And long contexts can make that state expensive.

We will study that more carefully when we reach the KV cache lesson.

Why quantized models can still perform inference

Later in the curriculum we will study quantization in detail.

For now, I only need one connection.

If a trained model’s weights are compressed into a lower-precision representation, an inference runtime can often execute the same basic forward computation using quantized weights.

That is why I can run something like a GGUF model locally without reproducing the original training setup.

The expensive learning already happened.

Local inference is applying a compressed representation of those learned parameters.

Server-side inference adds scheduling around the same model computation

When I use a hosted API, more machinery exists around the neural network.

There may be:

request queues
batching
GPU scheduling
model parallelism
KV-cache management
streaming output
rate limits

Those systems can dramatically affect latency and throughput.

But underneath them, the fundamental model loop is still recognizable:

process tokens
-> run model
-> obtain logits
-> choose next token
-> continue

The serving system optimizes how many requests can share expensive hardware efficiently.

It does not change the basic meaning of inference.

One prompt, mechanically

Let me put the whole path together.

I type:

Why do bond prices usually fall when interest rates rise?

The system roughly does:

1. Convert the text into token IDs.
2. Convert token IDs into embeddings.
3. Run the prompt through the Transformer.
4. Produce logits for the next token.
5. Apply the decoding rule.
6. Choose one token.
7. Append that token to the context.
8. Run the next decode step.
9. Repeat until a stop condition is reached.
10. Decode token IDs back into text for me to read.

No backpropagation is required for that ordinary response.

No parameter update is required.

The model is using learned weights, not learning from my prompt in real time.

Does chatting with a model train it immediately?

Usually, no.

This is another useful distinction.

When I send a prompt to a deployed model, my conversation can become part of the current context, so the model can respond differently later in the same conversation.

That is not the same thing as changing the model’s weights.

conversation context changes
!=
model parameters change

A provider could later use collected data in some separate training process, depending on the system and policy, but that is a different pipeline from the immediate inference request itself.

For understanding model mechanics, I keep those two ideas separate.

When does inference stop?

Generation needs a stopping condition.

Possible reasons include:

model emits an end-of-sequence token
runtime reaches max output tokens
application detects a stop sequence
user cancels the request

Once generation stops, the final token sequence is converted back into readable text.

That is the answer I see in the chat window.

The mental model I am keeping

My compact version is:

Training changes the weights.
Inference holds the trained weights fixed and uses them.
A prompt becomes tokens.
Tokens pass through the Transformer.
The model produces logits for the next token.
A decoding rule selects one token.
That token is appended.
The process repeats until generation stops.

This gives us the next link in the sequence:

pretraining
-> fine-tuning
-> trained model
-> inference
-> generated tokens

But I skipped over one important decision.

The model does not directly hand us a word. It gives us a distribution over possible next tokens.

So how do we decide whether to always choose the most likely token, introduce randomness, use temperature, or restrict the candidate set?

That is the next lesson: sampling.