Parameters, Weights, and Biases: What Are They in a Neural Network?

In the previous lesson, machaMochaLatte explained that machine learning is largely about changing parameters so a model performs better.

That immediately raises a more concrete question:

What are these parameters, physically and mathematically?

When I first saw phrases like “7 billion parameters” or “70B model,” I treated the number almost like a product label. Bigger number, bigger model. But I did not really understand what was being counted.

This lesson is my attempt to make that number less mysterious.

Start with one simple equation

Imagine a tiny model that predicts an exam score from hours studied:

predicted score = weight × hours studied + bias

Suppose:

weight = 6
bias = 42

For four hours of study:

predicted score = 6 × 4 + 42 = 66

There are two adjustable numbers here:

6
42

Those two numbers are parameters.

The important thing is that the model’s behavior changes when those numbers change.

If training updates the weight from 6 to 8, predictions change. If training updates the bias from 42 to 39, predictions change again.

So a useful beginner definition is:

A parameter is a numerical value inside a model that can be learned from data and that influences the model’s output.

Then what is a weight?

A weight is one common type of parameter.

A weight controls how strongly one value influences another.

In our simple equation:

predicted score = weight × hours studied + bias

if the weight is larger, each extra hour of study changes the prediction more strongly.

For example:

weight = 2   → each extra hour adds 2 predicted points
weight = 8   → each extra hour adds 8 predicted points

This is extremely simplified, but the intuition survives inside neural networks.

Neural networks repeatedly combine input values with learned weights. Those weighted combinations are passed through many layers until the model produces an output.

A large language model is vastly more complicated than our one-line score predictor, but it still contains huge arrays of learned numerical weights.

What is a bias?

A bias is another common parameter.

The easiest way I remember it is that a bias lets a calculation shift even when the input itself is zero.

Consider:

y = weight × x + bias

If:

weight = 3
bias = 10

then when x = 0:

y = 10

Without the bias, the output would have been forced to zero whenever the input was zero.

That extra flexibility matters because real relationships do not always pass neatly through zero.

In neural networks, biases can help units shift where they become active and make the model more flexible.

Not every modern architecture uses biases in every layer, but the general concept is still important.

Parameters are usually stored in tensors

This is where local AI started to make more sense to me.

A parameter is conceptually a number, but real models do not store billions of individual variables with names like:

weight_1
weight_2
weight_3
...

Instead, parameters are organized into tensors.

A tensor is basically a structured array of numbers.

For example, a tiny weight matrix could look like:

[
  [ 0.21, -0.84,  0.15 ],
  [ 1.07,  0.03, -0.44 ]
]

That matrix contains six numerical values.

If those values are trainable, they contribute six parameters to the model’s parameter count.

Real neural networks contain many tensors with shapes that can be enormous.

So when we say a model has billions of parameters, we are basically counting the learned numerical entries spread across all of those parameter tensors.

Why are there billions of parameters?

This number sounds absurd until you think about how many transformations a large model performs.

A language model has to transform token representations through many layers. Each layer may contain large matrices for attention, feed-forward networks, projections, normalization-related values, and other learned components.

Suppose just one matrix had a shape of:

4096 × 4096

The number of entries would be:

4096 × 4096 = 16,777,216

That is already about 16.8 million parameters in one matrix.

Now imagine multiple large matrices per layer and dozens of layers.

The total can reach billions very quickly.

This helped me understand that “7B parameters” does not mean there are seven billion separate ideas stored in the model.

It means there are roughly seven billion learned numerical values participating in the model’s computations.

Parameter count is not the same thing as intelligence

It is tempting to treat parameter count as an intelligence score.

That is too simple.

A model’s usefulness depends on much more than its raw parameter count:

• architecture
• training data
• data quality
• training objective
• optimization
• context handling
• post-training
• inference implementation

A smaller, better-trained model can outperform a larger model on some tasks.

Different architectures can also use parameters differently. Mixture-of-Experts models, for example, may have a large total parameter count while activating only a subset for a particular token.

So parameter count tells us something important about model scale, but not the whole story.

Parameters take memory because numbers need bits

This is the part that connects directly to local AI hardware.

Every parameter has to be represented somehow in memory or storage.

If we stored each parameter as a 32-bit floating-point number, each parameter would require 4 bytes.

A simplified estimate for a 7-billion-parameter model would be:

7,000,000,000 parameters × 4 bytes
≈ 28 GB

If the same parameters were represented using 16 bits instead of 32 bits, the rough weight storage would be about half:

7,000,000,000 × 2 bytes
≈ 14 GB

This is why precision matters so much when people talk about fitting models into RAM or VRAM.

The exact file size or runtime memory use can differ because model files include metadata, different tensor types, alignment, architecture-specific data, caches, and other overhead. But the basic relationship is real:

More bits per parameter generally means more storage and memory.

FP32, FP16, and BF16 are different ways to represent numbers

You may see model weights described as:

FP32
FP16
BF16

These are floating-point formats.

For a beginner, the key fact is that they use different numbers of bits to represent values.

FP32 uses 32 bits per value. FP16 and BF16 use 16 bits per value, though they divide those bits differently between range and precision.

The engineering details matter, but the main local-AI consequence is easy to see:

32 bits per parameter → more memory
16 bits per parameter → less memory

This is one reason models are often distributed or converted into lower-precision formats for inference.

Then quantization goes further

Quantization reduces the precision used to represent model values even more aggressively.

Instead of keeping every weight in a 16-bit or 32-bit floating-point format, an inference format may encode weights using something closer to 8, 6, 5, 4, 3, or even fewer effective bits per weight, depending on the quantization method.

That can dramatically reduce model size.

Using an intentionally rough example, if a hypothetical seven-billion-parameter model averaged about four bits per parameter, the raw weight information would be on the order of:

7,000,000,000 × 4 bits
= 28,000,000,000 bits
≈ 3.5 GB

Real quantized formats are more complicated than this because they usually store scales, block metadata, and sometimes different tensors at different precisions. So a real “4-bit model” is not necessarily exactly 3.5 GB.

But the rough calculation explains why quantization is so important for local inference.

A model that would be inconvenient or impossible to fit at full precision may become practical after quantization.

Quantization does not mean deleting parameters

This distinction confused me at first.

If a 7B model becomes much smaller after quantization, did it suddenly become a 3B model?

Usually, no.

The model can still contain roughly the same number of weight values. What changed is how those values are represented.

A crude analogy is saving the same photo with fewer bits per pixel.

The image dimensions may remain the same, but each value is stored less precisely.

Likewise:

same approximate parameter count
+
lower precision per parameter
=
smaller model representation

The challenge is preserving enough numerical information that the model still behaves well.

That is why different quantization methods can produce different quality, speed, and memory trade-offs even at similar nominal bit rates.

Training and inference treat parameters differently

During training, parameters are actively updated.

A simplified training loop looks like:

input
  ↓
model parameters
  ↓
prediction
  ↓
loss
  ↓
calculate updates
  ↓
change parameters

During ordinary inference, the learned parameters are usually fixed:

input
  ↓
fixed learned parameters
  ↓
output

This distinction explains something I used to find strange: why training a model can require far more memory than simply running it.

Inference mainly needs the weights plus runtime state such as activations and the KV cache.

Training also needs information required to compute and apply parameter updates, often including gradients and optimizer state. That can multiply memory requirements well beyond the model weights alone.

So “the model file is 14 GB” does not mean training it requires 14 GB of memory.

Where are the parameters when I run a local model?

When you download a model for local inference, its learned weights are encoded in model files on storage.

When the runtime loads the model, some or all of those tensors are mapped or copied into system RAM, GPU VRAM, or both, depending on the runtime and configuration.

Then inference kernels repeatedly read those values while processing your prompt and generating tokens.

This creates a very concrete chain:

training produced learned parameters
        ↓
model file stores representations of those parameters
        ↓
runtime loads/maps them
        ↓
CPU/GPU uses them during inference

That is why parameter count, precision, quantization, RAM, VRAM, and inference speed are all connected.

They are different views of the same underlying numerical object.

The mental model I want to keep

Here is the compact version I want to remember:

PARAMETER
A learned numerical value inside a model.

WEIGHT
A parameter that controls how strongly information contributes to a computation.

BIAS
A parameter that shifts a computation independently of the input value.

PARAMETER COUNT
How many learned numerical values the model contains.

PRECISION
How many bits are used to represent those values.

QUANTIZATION
A method for representing weights with lower precision so the model can use less storage and memory.

This makes phrases like “7B model,” “FP16 weights,” and “Q4 quantization” feel much less mysterious.

They describe different properties of the same thing: enormous collections of learned numbers.

The next question naturally moves from what the numbers are to how the model uses them.

If billions of parameters are stored in tensors, how do inputs actually move through those tensors and become an output?

That leads to the next lesson: the forward pass.