Tensors and their dimensions and ranks and shapes used to drive me crazy. But I think I finally understand it. This is an article I wish I could have found.

So a tensor, at the end of the day, is just a generalization of a matrix. Or rather - a matrix is a 2-dimensional tensor. Let’s look at a couple of tensors and learn from examples.

Tensors are immutable - just like Python strings. You can’t update a tensor, but you can create a new one.

From the Tensorflow docs, here’s some vocabulary surrounding tensors:

  1. Shape: number of elements of each of the dimensions of a tensor.
  2. Rank: Number of tensor dimensions. Scalar is rank 0, vector is rank 1, matrix is rank 2.
  3. Axis or dimension: A particular dimension of a tensor (we will see examples)
  4. Size: the total number of items in the tensor (or the product shape vector, whatever that means)

Let’s first look at a complicated example:

>>> torch.zeros(1,2,3,4,5)

tensor([[[[[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]],

      [[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]],

      [[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]]],


     [[[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]],

      [[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]],

      [[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]]]]])

Okay. So it shouldn’t be too hard to infer what is happening here. All the command is saying is to make a tensor of zeros, and we pass in the shape of the tensor. Let’s verify that is indeed what we are doing.

>>> a = torch.zeros(1,1,23,42,5)
>>> a.shape
torch.Size([1, 1, 23, 42, 5])

So the natural question that follows, is what does it mean for a tensor to have a shape of $(x, y, z)$? If you scroll up to the earlier definition of shape - it is the number of elements of each dimension. Let’s work on simple assumptions, and assume that when we have a tensor with a shape of $(x,y,z)$, what that means is that the tensors dimension 0 has $x$ elements in it, dimension 1 has $y$ elements in it, and dimension 2 has $z$ elements in it. Let’s try this out.

>>> a = torch.randn(3, 5, 4)
>>> a
tensor([[[ 2.3943e-01,  7.6971e-01, -4.5301e-01,  6.4772e-01],
        [-9.9622e-01, -6.5626e-01, -5.7584e-01,  3.2929e-01],
        [ 2.5115e-01,  1.7819e-01, -6.6255e-01, -1.0225e+00],
        [ 1.1163e+00, -1.2572e-01, -1.4620e+00,  8.7245e-01],
        [ 1.2521e-01,  1.1502e+00,  2.4332e-01,  6.8732e-01]],

        [[-6.5957e-01, -1.2786e-01,  7.3562e-01, -1.4286e+00],
        [-1.0472e-01,  8.7401e-01,  8.6666e-01, -2.9067e-01],
        [-3.5717e-01,  2.6646e+00,  7.3378e-01, -9.8718e-01],
        [ 2.4657e-01,  1.3094e-01, -1.2468e-01,  4.2728e-01],
        [ 4.4265e-02, -8.1253e-01, -1.3276e+00, -1.2146e+00]],

        [[-1.7902e-02, -8.0276e-01, -1.7175e-01, -1.5387e+00],
        [-2.7683e-01, -1.2516e+00,  1.3630e-03,  2.2188e-02],
        [ 1.5717e+00,  1.0675e+00,  2.2269e-01,  5.4317e-01],
        [ 2.7328e-01,  5.1839e-01, -2.1065e+00, -1.4524e+00],
        [ 1.2250e-01, -1.1339e+00, -6.8501e-01,  1.0978e+00]]])

The outer array of our tensor a with shape (3, 5, 4) has 3 elements in it. The length of a is 3.

Now each individual element of a has 5 different elements, this would explain the second number in our shape tuple. And finally, each of these elements have 4 elements - which explains the last number, 4. You can think of the last number of the shape tuple to be the number of elements in each final subunit. Think of the elements of each consecutive axes of a tensor to just be the elements that are one level of abstraction lower. So the elements of axis 2 in our toy example are the ones at the lowest level. Every other axis contains arrays, the last axis will be numbers/floats. The shape is just the length of each axis.

What happens if, say, the elements of a don’t have the same number of elements? What happens to the shape then? That’s not a possibility. You have to have each ‘subunit’ having the same shape (and therefore the same number of elements). Thinking in terms of matrices spanning a vector space, we have that each row of a matrix corresponds to one point in that space. A matrix is a collection of points or vectors of the same lengths.