What Is a Forward Pass? How Data Moves Through a Neural Network

We have spent the first few lessons collecting pieces: models, parameters, weights, biases, and tensors. I understood each definition separately, but I still had a basic problem.

What does the neural network actually do with an input?

The answer is a forward pass.

A forward pass is what happens when data enters a neural network and moves through its layers until the network produces an output. No learning has to happen during this step. The model is simply using its current parameters to calculate a result.

That sounds straightforward. I found it much easier to understand once I stopped looking at a giant language model and followed a tiny example by hand.

Start with one input

Imagine a very small model trying to estimate whether I might buy a product. We give it two input values:

x = [3, 5]

Maybe 3 represents how many times I visited the product page and 5 represents some normalized measure of previous purchases. The exact meanings do not matter for the math.

From the tensor lesson, we can recognize x as a one-dimensional tensor containing two numbers.

Now imagine one neuron has two learned weights:

w = [0.4, 0.2]

and one learned bias:

b = 0.5

The neuron first combines the input with its weights:

3 × 0.4 + 5 × 0.2 + 0.5

which gives:

1.2 + 1.0 + 0.5 = 2.7

That number did not appear by magic. The input values were multiplied by learned weights, the results were added together, and then the bias was added.

This is the basic operation behind a linear layer.

Why the weights matter

This example also helped me connect the forward pass to the previous lesson about parameters.

Suppose the first weight were 0.04 instead of 0.4:

3 × 0.04 + 5 × 0.2 + 0.5 = 1.62

The same input now produces a different result.

So the weights determine how strongly different parts of the input affect the calculation. The bias shifts the result independently of the input.

During a forward pass, the network is not deciding what those weights should be. It is using the weights it already has.

That distinction becomes important when we get to training.

One neuron is not much of a network

Real neural-network layers contain many neurons, so instead of storing one little weight vector for each neuron separately, we organize the weights into a matrix.

Suppose a layer has three output neurons. Its weight tensor might look like this:

W = [
  [0.4,  0.2],
  [0.1, -0.3],
  [0.7,  0.5]
]

and the bias tensor might be:

b = [0.5, 0.2, -0.1]

Our input is still:

x = [3, 5]

Each row of W performs a calculation like the one above. The layer therefore turns two input values into three output values.

In compact notation, we can write the operation as something like:

y = Wx + b

depending on the convention used for tensor orientation.

This is why tensor shapes matter. If the shapes do not line up correctly, the multiplication cannot happen.

The tensor article was not a detour. Tensors are the containers, and the forward pass is where operations start transforming them.

But linear layers alone have a problem

If every layer only performed linear transformations, stacking many layers would not give us the kind of expressive network we want. Multiple linear transformations can collapse mathematically into another linear transformation.

Neural networks therefore usually insert nonlinear activation functions between transformations.

A simple example is ReLU:

ReLU(x) = max(0, x)

So:

ReLU(2.7) = 2.7
ReLU(-1.4) = 0

The rule is almost embarrassingly simple: keep positive values and replace negative values with zero.

Yet adding nonlinear operations allows a network to represent relationships that a purely linear system cannot.

This also introduces another word we see constantly in AI: activation.

An activation is an intermediate value produced as data moves through the network. Weights are persistent learned parameters. Activations are values generated from a particular input during computation.

I used to mix those two up.

Follow a tiny network

Now we can picture a forward pass through several stages:

input tensor
    ↓
linear layer
    ↓
activation function
    ↓
linear layer
    ↓
output

Suppose the first layer converts our two input features into three activations:

[3, 5]
   ↓
[2.7, -1.0, 4.5]

After ReLU:

[2.7, 0, 4.5]

Those three numbers become the input to the next layer. That layer applies another set of weights and biases and produces another tensor.

The important point is that the model is not passing the original input unchanged from one end to the other. Each layer creates a new representation from the previous one.

That is the part of a forward pass I had been missing.

What “forward” actually means

The word forward does not mean the computer is physically moving in one direction. It describes the direction of the computation graph.

We start with the input and repeatedly calculate later values from earlier values until we reach the output:

input → hidden values → output

Later, during training, we will need information to flow conceptually in the other direction to determine how parameters contributed to an error. That process is connected to backpropagation.

For now, it is enough to keep the two ideas separate:

forward pass: input → prediction
backward pass: error information → parameter gradients

We have not learned gradients yet, so the second line can wait.

Inference is mostly forward passes

This gives us a useful connection to local AI.

When we run a trained model to get an answer, we are doing inference. The model repeatedly performs forward computation using parameters that have already been learned.

For a language model, the real architecture is vastly more complicated than our tiny example. There are embeddings, attention, normalization, feed-forward blocks, residual connections, and many layers. Autoregressive generation also repeats computation as new tokens are produced.

But the basic distinction survives the increase in scale:

inference uses the model’s existing parameters to compute outputs.

It does not normally update billions of weights just because we asked a question.

This helped me understand why downloading a model and chatting with it is different from training it.

Why activations use memory

The forward pass also explains something that matters when running models locally.

The model weights occupy memory, but they are not the only tensors involved. As the input moves through the network, the model creates intermediate activation tensors.

Those values also require memory while they are needed.

Training generally needs to retain much more information from the forward pass because the later backward pass needs intermediate values to calculate gradients. Inference can often discard intermediate activations sooner.

That is one reason training a model can require much more memory than simply running it for inference, even before considering optimizer state and other training-specific data.

We will return to this when the series reaches training.

A forward pass does not tell us whether the answer is good

There is one more distinction I think is important.

A forward pass produces an output. It does not by itself tell us whether that output is correct.

Suppose a model predicts:

0.82

but the correct target should have been:

1.00

The forward pass has done its job: it produced 0.82 from the current parameters.

Now we need some way to measure how bad that result is.

That measurement is the job of a loss function.

And once we can measure the error, we can start asking the question that turns a fixed neural network into a learning system:

How should the weights change so the next prediction is better?

The mental model I am keeping

For now, I think of a neural network as a chain of numerical transformations.

input tensor
    ↓
weights + bias
    ↓
activation
    ↓
more layers
    ↓
output tensor

The parameters are learned numbers that control the transformations.

The tensors organize the numbers moving through the system.

The forward pass is the actual computation that transforms an input into an output using the model’s current parameters.

That finally connects the pieces from the previous lessons.

But our network still cannot learn. It can calculate an answer, but nothing we have discussed yet tells it whether the answer was good or how to improve.

That is where training begins.