What Is a Tensor? The Shape of Data Inside AI

In the last article, we looked at parameters, weights, and biases. That left me with a word that shows up almost immediately whenever you read about neural networks: tensor.

At first I assumed a tensor was some advanced mathematical object I would have to understand before I could understand AI. Then I opened PyTorch documentation and found a much more useful starting point: a tensor is a multidimensional collection of numbers.

That sounds almost too simple. The interesting part is not the word tensor. It is how those numbers are arranged.

Start with a single number

Suppose a model has one number:

7

You can think of that as a zero-dimensional tensor, sometimes called a scalar.

Now put several numbers in a line:

[7, 3, 9, 2]

That is a one-dimensional tensor. In ordinary math we might call it a vector.

Arrange numbers in rows and columns:

[
  [7, 3, 9, 2],
  [5, 1, 4, 8],
  [6, 0, 2, 3]
]

Now we have a two-dimensional tensor, or a matrix.

The next step is the one that finally made the word click for me. Imagine several of those matrices stacked together. That is a three-dimensional tensor. Stack groups of those and you have four dimensions. There is no requirement that we stop at the three physical dimensions we can picture.

A tensor is a convenient way for a computer to keep track of numbers along however many axes the problem needs.

Shape is the first thing I would ask

If somebody tells me, “this is a tensor,” I still know almost nothing about it.

A much better question is: what is its shape?

For the matrix above, the shape is:

(3, 4)

because it has 3 rows and 4 columns.

A tensor with shape:

(32, 128, 4096)

contains three axes. The numbers alone do not tell us what those axes mean. In one model they might mean batch, token position, and hidden features. In another application they could represent something completely different.

This was an important correction to my intuition. “Dimension” in programming often means the number of axes, not how many numbers are stored. A tensor with shape (1000,) has 1,000 values but only one axis. A tensor with shape (10, 10, 10) has three axes and also 1,000 values.

The total number of elements is the product of the shape:

10 × 10 × 10 = 1,000

That simple multiplication becomes surprisingly important when thinking about AI memory use.

An image is already a good tensor example

A color image gives us something concrete.

Suppose an image is 1920 pixels wide and 1080 pixels high. Each pixel has three color values: red, green, and blue.

One natural representation has shape:

(1080, 1920, 3)

The axes can mean:

height × width × color channels

The image we see as a picture becomes 6,220,800 numbers if each channel is stored separately:

1080 × 1920 × 3 = 6,220,800

Deep-learning frameworks may arrange those axes differently. The point is not to memorize one order. The point is that shape gives structure to a huge list of values.

Once I saw an image this way, higher-dimensional tensors stopped feeling mysterious.

Add a batch and we get another axis

Neural networks usually process more than one example at a time.

If we have 32 RGB images, each 224 by 224 pixels, a batch might have a shape such as:

(32, 3, 224, 224)

Here the axes mean:

batch × channels × height × width

Nothing magical happened when we went from three axes to four. We just needed one more label: which image in the batch?

Language models do something similar. A simplified activation tensor might be described by:

(batch, sequence, hidden)

If its shape is:

(2, 512, 4096)

we can read that as two sequences, up to 512 token positions, with 4,096 values representing each position at that stage of the model.

That is already much closer to what an LLM is actually moving through memory than the vague phrase “the AI processes text.”

The weights from the last article are tensors too

In AI Foundations #3, William explained that a layer can contain millions of learned weights. Those weights are not normally stored as millions of separately named variables.

They are packed into tensors.

A weight matrix with shape:

(4096, 4096)

contains:

4096 × 4096 = 16,777,216 parameters

So when we say a model has billions of parameters, we are talking about huge collections of numerical values organized into tensors of different shapes.

This connects three ideas that originally looked separate to me:

parameters are the learned numbers; tensors are how many of those numbers are organized; layers perform operations using those tensors.

That is a much more useful mental model than treating “tensor” as another AI buzzword.

A tensor has more than a shape

Shape tells us how the values are organized logically, but a real tensor also has a data type, usually called its dtype.

For example, values might use FP32, FP16, BF16, integers, or quantized representations.

That matters because the same shape can require very different amounts of memory.

Take our (4096, 4096) weight tensor with 16,777,216 values.

If every value occupies 4 bytes, the raw values take about 64 MiB. At 2 bytes each, about 32 MiB. At an idealized 4 bits per value, the payload would be about 8 MiB before accounting for quantization metadata and other overhead.

The shape did not change. The number of parameters did not change. The representation of each value changed.

This is why the tensor idea connects directly to quantization and local AI. A model file is not just “7 billion numbers.” Those numbers have shapes and data types, and those choices help determine how much storage and memory the model needs and what hardware can compute with it efficiently.

Shape and memory layout are not the same thing

This part surprised me.

A tensor can look like a neat multidimensional box to us while its actual data is stored in a one-dimensional region of memory. PyTorch describes a regular tensor using underlying storage plus information such as dtype, shape, stride, and offset.

Stride tells the program how far it has to move through that storage to advance along each axis.

You do not need stride arithmetic yet to understand the important lesson: the shape we reason about and the bytes the computer moves are related, but they are not the same thing.

That distinction later matters for operations such as transpose, reshape, contiguous copies, and GPU performance.

It is also one reason tensor libraries exist. We want to write something like “multiply these matrices” rather than manually calculate where millions of individual numbers live in memory.

Why GPUs fit this world so well

A neural network repeatedly performs mathematical operations over large tensors. Instead of asking a CPU to handle every number as an unrelated little task, accelerators can perform enormous amounts of structured numerical work in parallel.

PyTorch tensors can live on CPUs or accelerators such as GPUs. Moving a tensor to a GPU does not change the basic mathematical idea. It changes where its data lives and where operations on it are executed.

That gives me a more concrete way to interpret phrases like “the model is loaded into VRAM.” A large part of what is being placed there is tensor data: weights, activations, caches, and other numerical state needed for computation.

A tensor is not automatically an AI thing

This is another useful boundary.

Tensors are general numerical data structures. PyTorch’s own examples use them for ordinary numeric computation; a tensor does not somehow contain intelligence because it belongs to a neural-network library.

The AI comes from what the model has learned and from the sequence of operations performed on these values.

A spreadsheet full of numbers is not a financial model until formulas and assumptions give those numbers relationships. In roughly the same spirit, a tensor is structured numerical data; the neural-network architecture tells the system what to do with it.

The mental picture I am keeping

I now think of a tensor as having three questions attached to it:

1. What are the numbers? The actual values.
2. What is the shape? How the values are organized into axes.
3. What is the dtype? How each value is represented.

For systems work I would eventually add device, stride, layout, and storage. But those first three are enough to make a lot of beginner AI diagrams less mysterious.

When you see something like:

x: [2, 512, 4096], dtype=bf16

you no longer have to read it as programmer hieroglyphics. It says there is a three-axis block of numbers with shape 2 × 512 × 4096, and each element uses the BF16 data type.

That is the bridge I wanted before moving on to the next question: what actually happens to these tensors when data moves forward through a neural network?

That is the forward pass, and it is where the static pieces we have built so far finally start moving.