Vectors and matrices

1. Intuition

• A scalar is a single number: a learning rate, a loss value.
• A vector is an ordered list: a house described as [bedrooms, thousand sq ft] = [2, 1.5]. Its length is its dimension.
• A matrix is a grid of numbers, rows by columns. A dataset of 1,000 houses with 2 features is a $1000 \times 2$ matrix: one row per house.
• A tensor is the same idea with more axes. A batch of 32 color images of 64×64 pixels is a tensor of shape $[32, 3, 64, 64]$.

The dot product of two equal-length vectors multiplies matching entries and adds them up. If $x$ is a house and $w$ holds a price per unit of each feature, then $x \cdot w$ is a predicted price. Geometrically, the dot product measures how much two vectors point the same way: large and positive if aligned, zero if perpendicular, negative if opposed.

Matrix multiplication is many dot products at once: each entry of $AB$ is a row of $A$ dotted with a column of $B$.

2. Visualization

What to notice:

• Adding vectors places them tip to tail.
• The dot product $a \cdot c = 5$ equals $\lVert a \rVert$ (3.16) times the length of $c$'s shadow on $a$ (1.58). If $c$ were perpendicular to $a$, the shadow and the dot product would be 0.

3. The math

Symbols

Dot product

$\theta$ is the angle between the vectors. Dividing by both lengths gives cosine similarity, $\cos\theta$, used everywhere in NLP and retrieval.

Matrix product

If $A$ is $[m, n]$ and $B$ is $[n, p]$, then $C = AB$ is $[m, p]$ and

The inner dimensions ($n$) must match and disappear; the outer dimensions ($m$, $p$) remain. Matrix multiplication is not commutative: $AB \neq BA$ in general, and often $BA$ isn't even defined.

Worked example 1: a prediction as a dot product

Features $x = [2, 1.5]$ (bedrooms, thousand sq ft), weights $w = [30, 120]$ (thousand dollars per unit), bias $b = 50$.

Predicted price: 290 thousand dollars. If the true price is 310, the residual is $310 - 290 = 20$ and the squared error is $400$ (in thousand-dollars squared).

Worked example 2: matrix product with shapes

Shape check first: $[3, 2] \times [2, 3] \to [3, 3]$. Then, for example, $C_{13}$ = row 1 of $A$ dotted with column 3 of $B$ $= 1 \times 2 + 2 \times 1 = 4$, and $C_{33} = 5 \times 2 + 6 \times 1 = 16$.

Worked example 3: cosine similarity

$a = [3, 1]$, $c = [1, 2]$: $a \cdot c = 3 + 2 = 5$, $\lVert a \rVert = \sqrt{10} = 3.162$, $\lVert c \rVert = \sqrt{5} = 2.236$. $\cos\theta = 5 / (3.162 \times 2.236) = 5 / 7.071 = 0.7071$, so the angle is 45°.

4. Implementation

You will build dot products, matrix products, and a rotation from plain Python lists, then check every answer against NumPy. Writing the loops once makes the shape rules stick.

Build it from scratch

A dot product multiplies matching entries and adds them up. The assert is the shape check that NumPy does for you; failing loudly here beats silently dropping entries, which is what zip would do on its own.

def dot(a, b):
    assert len(a) == len(b), f"shape mismatch: {len(a)} vs {len(b)}"
    return sum(ai * bi for ai, bi in zip(a, b))

def norm(a):
    return math.sqrt(dot(a, a))

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))

Predict: cosine([3, 1], [6, 2]) versus cosine([3, 1], [1, 2]). Which is 1.0, and why does scaling a vector not change the result?

Projection splits a vector into "the part along a" and "the rest". It is the geometric meaning of the dot product from the figure: scale a by how much of c points its way.

def project(c, a):
    """The part of c that points along a: (c·a / a·a) a."""
    s = dot(c, a) / dot(a, a)
    return [s * ai for ai in a]

Predict: what is project([1, 2], [3, 1])? Work it out by hand: c·a = 5 and a·a = 10.

A matrix product is a grid of dot products: entry (i, j) is row i of A dotted with column j of B. Transposing B turns its columns into rows so dot can reuse them. The assert is the inner-dimension rule from section 3.

def shape(M):
    return (len(M), len(M[0]))

def transpose(M):
    return [list(col) for col in zip(*M)]

def matmul(A, B):
    (n, k), (k2, m) = shape(A), shape(B)
    assert k == k2, f"inner dims differ: {shape(A)} @ {shape(B)}"
    Bt = transpose(B)                                   # columns of B as rows
    return [[dot(row, col) for col in Bt] for row in A]  # entry (i, j) = row i · column j

Predict: matmul(A, A) for the [3, 2] matrix A from worked example 2. Does it run? What about matmul(transpose(A), A), and what shape comes out?

A matrix is also a function that moves points. A rotation matrix turns every vector by the same angle, and multiplying two transformation matrices composes them.

def rotation(theta):
    c, s = math.cos(theta), math.sin(theta)
    return [[c, -s], [s, c]]

def apply(M, v):
    return [dot(row, v) for row in M]

Predict: where does apply(matmul(R, R), [1, 0]) land when R = rotation(math.pi / 2)?

Use the library

NumPy runs the same operations in compiled code, and @ is the matrix product. Everything above becomes one line each.

import numpy as np

x = np.array([2, 1.5])
w = np.array([30, 120])
print(x @ w + 50)                  # 290.0  (@ is the matrix/dot product)

X = np.array([[2, 1.5], [3, 2.0], [1, 0.8]])   # [3 houses, 2 features]
print(X @ w + 50)                  # [290. 380. 176.]  one prediction per row: [3,2] @ [2] -> [3]

A = np.array([[1, 2], [3, 4], [5, 6]])
B = np.array([[1, 0, 2], [0, 1, 1]])
print((A @ B).shape)               # (3, 3)
print(A * A)                       # element-wise, NOT a matrix product

What NumPy adds over the scratch version: speed (BLAS kernels, see Python toolkit), broadcasting so X @ w + 50 adds 50 to every row, and batched products over leading dimensions ([batch, n, k] @ [batch, k, m]). What it hides: A * A is legal and silently does the wrong thing if you meant A @ A.

Run it

From guide/:

uv run code/00-foundations/vectors_scratch.py

a·c = 5,  |a| = 3.1623,  cos θ = 0.7071
projection of c on a = [1.5, 0.5]
house price x·w + b = 290.0
A (3, 2) @ B (2, 3) = (3, 3): [[1, 2, 4], [3, 4, 10], [5, 6, 16]]
A @ A -> inner dims differ: (3, 2) @ (3, 2)
rotate (1, 0) by 90°: [0.0, 1.0]
two 90° rotations = one 180°: [-1.0, 0.0]
all checks match NumPy

Full script: code/00-foundations/vectors_scratch.py. The module's figures come from code/00-foundations/foundations.py.

5. Engineering

The shape habit. Write shapes next to every line: X @ W # [B, d_in] @ [d_in, d_out] -> [B, d_out]. Most deep-learning bugs are shape bugs, and the worst ones don't crash: broadcasting silently turns a [B] minus [B, 1] into a [B, B] matrix.

Vectorize. One matrix product over a batch is thousands of times faster than a Python loop of dot products, because it runs in optimized BLAS or GPU kernels. Batching is also why GPUs make deep learning practical.

Cost. A product of $[m, n]$ and $[n, p]$ takes $m \times n \times p$ multiply-adds. Counting these tells you where a model spends its compute.

Watch out

Failure modes. Confusing * (element-wise) with @ (matrix product); transposing to silence a shape error instead of understanding it; broadcasting a [n] vector against [n, 1] and getting [n, n]; integer arrays truncating results.

Common mistakes

• Assuming $AB = BA$.
• Forgetting that a dot product needs equal lengths.
• Reading a matrix as columns-per-example when the code uses rows-per-example (frameworks use rows).

6. Knowledge check

Practice exercise. Compute $[1, -2, 3] \cdot [4, 0, -1]$, then the shape of $X W$ where $X$ is $[64, 10]$ and $W$ is $[10, 3]$.

Implementation challenge. Write matmul(A, B) with three nested Python loops, check it against A @ B on random matrices, then time both for $200 \times 200$ inputs.

Summary

• Vectors describe one example; matrices stack examples or transform vectors; tensors add more axes.
• The dot product is a weighted sum and a measure of alignment; cosine similarity normalizes it.
• $[m, n] \times [n, p] \to [m, p]$: inner dimensions must match and disappear.
• Write shapes next to every operation and vectorize instead of looping.

Next: Calculus for ML

Related: Forward pass · Self-attention

Parts of this lesson are adapted from AI Engineering from Scratch by Rohit Ghumare (MIT license, notice): Linear Algebra Intuition, Vectors, Matrices & Operations, Matrix Transformations.

Interview angle