Matrices and TransformationsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Matrices and Transformations

A matrix is more than a grid of numbers — it's a function that transforms vectors. Every layer in a neural network is a matrix multiplication followed by a nonlinearity, so matrices are the building blocks of the hypothesis space — the set of functions a model can express. Understanding what matrices do geometrically makes deep learning intuitive.

Real-world examples: Weight matrices in neural networks, image rotations and projections in computer graphics, data whitening and covariance in statistics, change-of-basis in PCA.

What is a matrix?

A matrix A∈Rm×n\mathbf{A} \in \mathbb{R}^{m \times n} has mm rows and nn columns. It defines a linear map from Rn\mathbb{R}^n to Rm\mathbb{R}^m:

y=Ax,x∈Rn,  y∈Rm\mathbf{y} = \mathbf{A}\mathbf{x}, \quad \mathbf{x} \in \mathbb{R}^n,\; \mathbf{y} \in \mathbb{R}^m

Each row of A\mathbf{A} is a dot product with x\mathbf{x}:

yi=ai⊤x=∑j=1nAijxjy_i = \mathbf{a}_i^\top \mathbf{x} = \sum_{j=1}^n A_{ij} x_j

Neural network layer: a layer with weight matrix W∈Rdout×din\mathbf{W} \in \mathbb{R}^{d_\text{out} \times d_\text{in}} computes Wx+b\mathbf{W}\mathbf{x} + \mathbf{b} — a matrix-vector product.

Geometric intuitions

Think of a 2×22\times 2 matrix as a transformation of the 2D plane. Common transformations:

MatrixEffect
[2002]\begin{bmatrix}2&0\\0&2\end{bmatrix}Scale (stretch by 2)
[0−110]\begin{bmatrix}0&-1\\1&0\end{bmatrix}Rotate 90°
[1000]\begin{bmatrix}1&0\\0&0\end{bmatrix}Project onto x-axis
[1101]\begin{bmatrix}1&1\\0&1\end{bmatrix}Shear

Column perspective

The columns of A\mathbf{A} tell you where the standard basis vectors land. Column 1 is where e1\mathbf{e}_1 goes, column 2 where e2\mathbf{e}_2 goes. The whole transformation is determined by these two destinations.

Matrix multiplication

For A∈Rm×k\mathbf{A} \in \mathbb{R}^{m\times k} and B∈Rk×n\mathbf{B} \in \mathbb{R}^{k\times n}:

(AB)ij=∑r=1kAirBrj(\mathbf{AB})_{ij} = \sum_{r=1}^k A_{ir} B_{rj}

The (i,j)(i,j) entry is the dot product of row ii of A\mathbf{A} with column jj of B\mathbf{B}.

Worked example

Compose a 90° rotation R=[0−110]\mathbf{R} = \begin{bmatrix}0&-1\\1&0\end{bmatrix} with a scale S=[2003]\mathbf{S} = \begin{bmatrix}2&0\\0&3\end{bmatrix}. Each entry of RS\mathbf{RS} is row of R\mathbf{R} · column of S\mathbf{S}:

RS=[0⋅2+(−1)⋅00⋅0+(−1)⋅31⋅2+0⋅01⋅0+0⋅3]=[0−320]\mathbf{RS} = \begin{bmatrix} 0\cdot2 + (-1)\cdot0 & 0\cdot0 + (-1)\cdot3 \\ 1\cdot2 + 0\cdot0 & 1\cdot0 + 0\cdot3 \end{bmatrix} = \begin{bmatrix} 0 & -3 \\ 2 & 0 \end{bmatrix}

Read the columns: e1=[1,0]\mathbf{e}_1=[1,0] is first scaled to [2,0][2,0] then rotated to [0,2][0,2] — exactly column 1 of RS\mathbf{RS}. The product is "do S\mathbf{S}, then R\mathbf{R}."

Key properties:

  • Not commutative: AB≠BA\mathbf{AB} \neq \mathbf{BA} in general
  • Associative: (AB)C=A(BC)(\mathbf{AB})\mathbf{C} = \mathbf{A}(\mathbf{BC})
  • Distributive: A(B+C)=AB+AC\mathbf{A}(\mathbf{B}+\mathbf{C}) = \mathbf{AB} + \mathbf{AC}

Composing transformations: A(Bx)=(AB)x\mathbf{A}(\mathbf{Bx})= (\mathbf{AB})\mathbf{x}. A deep network is a composition of many such maps.

Python
import numpy as np

R = np.array([[0, -1], [1, 0]])  # rotate 90°
S = np.array([[2, 0], [0, 3]])   # scale: x by 2, y by 3

RS = R @ S                        # same composition as above: do S, then R
e1 = np.array([1.0, 0.0])

print(RS @ e1)       # [0. 2.] — e1 is scaled to [2,0], then rotated to [0,2]
print(R @ (S @ e1))  # [0. 2.] — associativity: same result

Transpose and symmetry

The transpose A⊤\mathbf{A}^\top flips rows and columns: (A⊤)ij=Aji(\mathbf{A}^\top)_{ij} = A_{ji}.

A matrix is symmetric if A=A⊤\mathbf{A} = \mathbf{A}^\top. Symmetric matrices arise naturally: covariance matrices and Gram matrices (X⊤X\mathbf{X}^\top \mathbf{X}) are always symmetric.

Useful identities:

(AB)⊤=B⊤A⊤,(Ax)⊤=x⊤A⊤(\mathbf{AB})^\top = \mathbf{B}^\top \mathbf{A}^\top, \qquad (\mathbf{Ax})^\top = \mathbf{x}^\top \mathbf{A}^\top

Rank and invertibility

The rank of A\mathbf{A} is the dimension of its column space — the number of linearly independent columns.

  • Full rank: rank(A)=min⁡(m,n)\text{rank}(\mathbf{A}) = \min(m, n). For square n×nn\times n matrices, full rank ↔ invertible.
  • Rank deficient: some columns are redundant (linearly dependent).

The determinant

The determinant measures the signed factor by which a transformation scales area (volume in higher dimensions). For a 2×22\times2 matrix:

det⁡[abcd]=ad−bc\det\begin{bmatrix}a&b\\c&d\end{bmatrix} = ad - bc

Geometrically, the unit square (area 1) becomes a parallelogram of area ∣det⁡∣|\det|. So:

  • ∣det⁡∣>1|\det| > 1 expands area, ∣det⁡∣<1|\det| < 1 shrinks it.
  • det⁡<0\det < 0 means the transformation flips orientation (a reflection).
  • det⁡=0\det = 0 means the square collapses to a line (area 0) — the map squashes 2D down to 1D, throwing away a dimension. You can't undo that, which is exactly why det⁡=0⇔\det = 0 \Leftrightarrow not invertible.

A matrix transforms space

îĵ
det (area scale)1.00

Each matrix column shows where a basis vector lands: î → (a, c), ĵ → (b, d). The determinant is the area-scaling factor — negative means space was flipped, zero means it was squashed to a line.

The interactive figure shows the determinant updating as you drag the matrix columns. A square matrix A\mathbf{A} is invertible iff det⁡(A)≠0\det(\mathbf{A}) \neq 0, and then A−1\mathbf{A}^{-1} satisfies AA−1=I\mathbf{A}\mathbf{A}^{-1} = \mathbf{I}.

When matrices are not invertible

Rank-deficient matrices map multiple inputs to the same output — information is lost. In ML, this shows up as multicollinear features (OLS fails) or as underdetermined systems (more unknowns than equations).

The OLS closed form w^=(X⊤X)−1X⊤y\hat{\mathbf{w}} = (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbf{y} requires X⊤X\mathbf{X}^\top \mathbf{X} to be invertible — hence why highly correlated features cause problems.

Special matrix types in ML

TypePropertyWhere it appears
Identity I\mathbf{I}IA=A\mathbf{IA} = \mathbf{A}Residual connections (adding x\mathbf{x} back)
DiagonalOff-diagonal zerosEfficient batch normalization
OrthogonalQ⊤Q=I\mathbf{Q}^\top \mathbf{Q} = \mathbf{I}QR decomposition, rotations
Positive semi-definitex⊤Ax≥0\mathbf{x}^\top \mathbf{A}\mathbf{x} \geq 0 (never flips a vector past 90°)Covariance matrices

Exercise

A = [[1,2],[3,4]] and x = [1, 0]. What is Ax?

Need a hint?

Each output row is the dot product of a row of A with x.

Exercise

A 3×3 matrix has rank 2. What can you conclude?

Need a hint?

Rank = dimension of the column space.

Exercise

For matrices A and B, which identity is always true?

Need a hint?

Think about the transpose of a product.

Common mistakes

  1. Assuming matrix multiplication is commutative: AB≠BA\mathbf{AB} \neq \mathbf{BA} in general. Order matters: "rotate then scale" and "scale then rotate" produce different results. Only diagonal matrices (and scalar multiples of the identity) commute with everything.
  2. Confusing rank with the number of columns: the rank is the number of linearly independent columns, which can be much smaller. A 10×1010\times 10 matrix of all ones has rank 1 despite having 10 columns.
  3. Forgetting that non-square matrices can still have left- or right-inverses: a tall matrix (m>nm > n, full column rank) has a left-inverse (A⊤A)−1A⊤(\mathbf{A}^\top\mathbf{A})^{-1}\mathbf{A}^\top; a wide matrix (m<nm < n, full row rank) has a right-inverse. Only square full-rank matrices have both.

Key takeaways

  • A matrix transforms vectors linearly — intuit it as rotation, scaling, projection.
  • Matrix multiplication chains transformations; a deep network is a long chain.
  • Rank measures how much information a transformation preserves.
  • Covariance and Gram matrices (X⊤X\mathbf{X}^\top\mathbf{X}) are always symmetric and positive semi-definite.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Related
Matrices and TransformationsVectors and SpacesEigenvalues and Eigenvect…Principal Component Analy…Layers & the Forward Pass