Matrices and Transformations
A matrix is more than a grid of numbers — it's a function that transforms vectors. Every layer in a neural network is a matrix multiplication followed by a nonlinearity, so matrices are the building blocks of the hypothesis space — the set of functions a model can express. Understanding what matrices do geometrically makes deep learning intuitive.
Real-world examples: Weight matrices in neural networks, image rotations and projections in computer graphics, data whitening and covariance in statistics, change-of-basis in PCA.
What is a matrix?
A matrix has rows and columns. It defines a linear map from to :
Each row of is a dot product with :
Neural network layer: a layer with weight matrix computes — a matrix-vector product.
Geometric intuitions
Think of a matrix as a transformation of the 2D plane. Common transformations:
| Matrix | Effect |
|---|---|
| Scale (stretch by 2) | |
| Rotate 90° | |
| Project onto x-axis | |
| Shear |
Column perspective
The columns of tell you where the standard basis vectors land. Column 1 is where goes, column 2 where goes. The whole transformation is determined by these two destinations.
Matrix multiplication
For and :
The entry is the dot product of row of with column of .
Worked example
Compose a 90° rotation with a scale . Each entry of is row of · column of :
Read the columns: is first scaled to then rotated to — exactly column 1 of . The product is "do , then ."
Key properties:
- Not commutative: in general
- Associative:
- Distributive:
Composing transformations: . A deep network is a composition of many such maps.
import numpy as np
R = np.array([[0, -1], [1, 0]]) # rotate 90°
S = np.array([[2, 0], [0, 3]]) # scale: x by 2, y by 3
RS = R @ S # same composition as above: do S, then R
e1 = np.array([1.0, 0.0])
print(RS @ e1) # [0. 2.] — e1 is scaled to [2,0], then rotated to [0,2]
print(R @ (S @ e1)) # [0. 2.] — associativity: same result
Transpose and symmetry
The transpose flips rows and columns: .
A matrix is symmetric if . Symmetric matrices arise naturally: covariance matrices and Gram matrices () are always symmetric.
Useful identities:
Rank and invertibility
The rank of is the dimension of its column space — the number of linearly independent columns.
- Full rank: . For square matrices, full rank ↔ invertible.
- Rank deficient: some columns are redundant (linearly dependent).
The determinant
The determinant measures the signed factor by which a transformation scales area (volume in higher dimensions). For a matrix:
Geometrically, the unit square (area 1) becomes a parallelogram of area . So:
- expands area, shrinks it.
- means the transformation flips orientation (a reflection).
- means the square collapses to a line (area 0) — the map squashes 2D down to 1D, throwing away a dimension. You can't undo that, which is exactly why not invertible.
A matrix transforms space
Each matrix column shows where a basis vector lands: î → (a, c), ĵ → (b, d). The determinant is the area-scaling factor — negative means space was flipped, zero means it was squashed to a line.
The interactive figure shows the determinant updating as you drag the matrix columns. A square matrix is invertible iff , and then satisfies .
When matrices are not invertible
Rank-deficient matrices map multiple inputs to the same output — information is lost. In ML, this shows up as multicollinear features (OLS fails) or as underdetermined systems (more unknowns than equations).
The OLS closed form requires to be invertible — hence why highly correlated features cause problems.
Special matrix types in ML
| Type | Property | Where it appears |
|---|---|---|
| Identity | Residual connections (adding back) | |
| Diagonal | Off-diagonal zeros | Efficient batch normalization |
| Orthogonal | QR decomposition, rotations | |
| Positive semi-definite | (never flips a vector past 90°) | Covariance matrices |
Exercise
A = [[1,2],[3,4]] and x = [1, 0]. What is Ax?
Need a hint?
Each output row is the dot product of a row of A with x.
Exercise
A 3×3 matrix has rank 2. What can you conclude?
Need a hint?
Rank = dimension of the column space.
Exercise
For matrices A and B, which identity is always true?
Need a hint?
Think about the transpose of a product.
Common mistakes
- Assuming matrix multiplication is commutative: in general. Order matters: "rotate then scale" and "scale then rotate" produce different results. Only diagonal matrices (and scalar multiples of the identity) commute with everything.
- Confusing rank with the number of columns: the rank is the number of linearly independent columns, which can be much smaller. A matrix of all ones has rank 1 despite having 10 columns.
- Forgetting that non-square matrices can still have left- or right-inverses: a tall matrix (, full column rank) has a left-inverse ; a wide matrix (, full row rank) has a right-inverse. Only square full-rank matrices have both.
Key takeaways
- A matrix transforms vectors linearly — intuit it as rotation, scaling, projection.
- Matrix multiplication chains transformations; a deep network is a long chain.
- Rank measures how much information a transformation preserves.
- Covariance and Gram matrices () are always symmetric and positive semi-definite.
Related concepts
- Eigenvalues and Eigenvectors: special directions that matrices don't rotate
- Vectors and Spaces: the objects matrices transform
- PCA & Dimensionality Reduction: eigendecomposition of the covariance matrix