Gradient Descent: How a Neural Network Takes Its First Step Toward a Lower Loss

In the previous lesson, machaMochaLatte explained loss: a number that tells us how badly the model is doing according to its training objective.

When I first learned this, my immediate question was practical:

If the loss is 0.36, what exactly should the model do with that number?

A loss by itself does not tell a weight whether to go up or down. Training needs a rule for changing parameters in a direction that should reduce the loss.

That rule begins with gradient descent.

Think of loss as a function of the weights

Take the smallest possible model:

prediction = weight × input

Suppose:

input = 2
weight = 0.5

Then:

prediction = 0.5 × 2 = 1

But imagine the target is 2.

Using squared error from the previous lesson:

loss = (prediction - target)²
     = (1 - 2)²
     = 1

Now change only the weight.

If the weight becomes 0.6:

prediction = 0.6 × 2 = 1.2
loss = (1.2 - 2)² = 0.64

The loss fell from 1 to 0.64.

If the weight instead becomes 0.4:

prediction = 0.4 × 2 = 0.8
loss = (0.8 - 2)² = 1.44

The loss increased.

So around weight = 0.5, increasing the weight appears to be the useful direction.

A neural network cannot afford to test random changes to billions of parameters one at a time. It needs a systematic way to estimate the direction and sensitivity of the loss for every parameter.

That information is carried by the gradient.

A gradient is a slope

For one weight, you can picture the relationship like this:

loss
 ^
 |        /
 |      /
 |    /
 |  _/
 |_/____________> weight

At the model’s current weight, the slope tells us how the loss changes when that weight changes slightly.

If the gradient is positive, moving the weight upward would locally increase the loss, so gradient descent moves the weight downward.

If the gradient is negative, moving the weight upward would locally decrease the loss, so gradient descent moves the weight upward.

That is why the update rule contains a minus sign:

new weight = old weight - learning rate × gradient

The name gradient descent is almost literal: use the gradient to find uphill, then move in the opposite direction.

Why we need a learning rate

Suppose a weight is 0.5 and its gradient is -4.

If we simply subtract the gradient:

new weight = 0.5 - (-4) = 4.5

That is a huge jump.

Instead, choose a learning rate such as 0.1:

new weight = 0.5 - 0.1 × (-4)
           = 0.9

The learning rate controls step size.

A very small learning rate can make training painfully slow. A very large learning rate can jump across a useful region, make the loss oscillate, or cause training to become unstable.

So gradient descent needs two pieces of information:

direction: gradient
step size: learning rate

Where did the -4 gradient come from?

For our tiny example:

prediction = w × x
loss = (prediction - target)²

With x = 2, target = 2, and w = 0.5, the loss can be written directly as:

loss = (2w - 2)²

The derivative with respect to w is:

dloss/dw = 2(2w - 2) × 2

At w = 0.5:

dloss/dw = 2(1 - 2) × 2
         = -4

That negative sign agrees with what we observed manually: increasing the weight from 0.5 reduced the loss.

You do not need to become comfortable with all of the calculus immediately. The important idea is that the derivative measures how sensitive the loss is to a small change in a parameter.

For one parameter, that is a derivative. For many parameters together, we usually talk about the gradient.

A neural network has many directions at once

Our toy model had one weight. A real neural network can have millions or billions of weights and biases.

Instead of asking:

Which way should this one weight move?

training asks something closer to:

For every trainable parameter, how would a small change affect the loss?

The collection of those derivatives forms the gradient of the loss with respect to the model parameters.

Then an optimizer applies updates.

In simplified form:

forward pass
    ↓
prediction
    ↓
loss
    ↓
gradients
    ↓
parameter update
    ↓
next forward pass

Repeat that process many times and the model can gradually move toward parameter values that perform better on the training objective.

Gradient descent does not promise the perfect answer

The word “descent” can make the process sound like walking directly to the lowest point on a smooth hill.

Neural-network loss surfaces are much more complicated. There are enormous numbers of dimensions, noisy estimates from batches of data, flat regions, steep regions, and interactions among parameters.

The gradient is local information. It tells us about the slope around the current parameters. It does not provide a map of the entire loss landscape or guarantee that one update is globally optimal.

This is also why modern training uses optimizers such as SGD and Adam rather than imagining a single perfect downhill path.

The foundational principle remains the same: use gradient information to choose parameter updates that are intended to reduce the objective.

Loss, gradient, and optimizer are different things

I found these three concepts easy to blur together, so here is the distinction I use:

A loss function does not update the model.

A gradient does not decide the entire training strategy.

An optimizer does not invent the objective.

They are connected, but they have different jobs.

The missing step: how does a deep network get all those gradients?

Our one-weight example let us write the derivative directly.

A deep neural network is a chain of operations:

input
  ↓
layer 1
  ↓
layer 2
  ↓
layer 3
  ↓
prediction
  ↓
loss

The loss is at the end, but an early weight may be many operations away from it.

How can the network efficiently determine how that early weight contributed to the final loss?

That is the job of backpropagation, which is the next lesson in AI Foundations.

For now, the bridge from the previous lesson is complete:

prediction → loss → gradient → parameter update

Loss tells us how bad the current result is. Gradient descent gives the model a way to take a step toward making that loss smaller.