Vectors and Spaces
Every piece of data in machine learning (an image, a sentence, a medical record) is eventually represented as a vector: a list of numbers with direction and magnitude. That representation is the data stage of every ML project; and the dot product you build on it is, quite literally, the whole of a linear model — the simplest hypothesis space there is. Almost everything you learn later is a change to one slot of that loop, so it pays to start at the two slots linear algebra owns.
What is a vector?
A vector is an ordered list of real numbers:
Geometrically, you can think of it as an arrow from the origin to a point in -dimensional space. In ML, is often the number of features.
Real-world example: A house with area 1500 sq ft, 3 bedrooms, and price $300k is the vector . (Here we pack everything — even the price — into one vector; in supervised learning you'll usually keep the target separate from the features .)
Core operations
Addition and scaling
These two operations are the building blocks of all linear algebra, and of every algorithm in this course.
The Euclidean norm
The length (norm) of a vector measures how far it points from the origin:
A unit vector has norm 1. You can normalize any vector: .
import numpy as np
v = np.array([3.0, 4.0])
norm = np.linalg.norm(v) # 5.0
unit = v / norm # [0.6, 0.8]
print(f"‖v‖ = {norm}")
print(f"unit vector: {unit}")
The dot product
The dot product is the most fundamental operation in ML:
where is the angle between the vectors.
Sign tells direction, magnitude tells similarity
When the vectors point roughly the same way; when it equals zero they are perpendicular; when it is negative they diverge. Dividing by both norms gives cosine similarity — a pure angle measure independent of length.
What the dot product measures
| Meaning | |
|---|---|
| Vectors point in roughly the same direction | |
| Vectors are orthogonal (perpendicular) | |
| Vectors point in roughly opposite directions |
Worked example
, :
u = np.array([2.0, 1.0])
v = np.array([1.0, 3.0])
dot = np.dot(u, v) # 5.0
cos_theta = dot / (np.linalg.norm(u) * np.linalg.norm(v)) # ≈ 0.707
angle_deg = np.degrees(np.arccos(cos_theta)) # ≈ 45°
Vectors and the dot product
The dot product a·b = |a||b|cos θ. It's zero when the vectors are perpendicular (θ = 90°), positive when they point similarly, negative when opposed. The faint segment is b projected onto a.
Projection: the dot product's geometric job
The dot product answers "how much of points along ?" That quantity is the projection of onto — the shadow casts on the line through . Start from and divide by :
The denominator rescales by 's length so the result lands exactly on the line through . Drag the vectors above to watch the projection segment grow and shrink with the angle.
Vector spaces and basis
A vector space is a set of vectors that is closed under addition and scalar multiplication. is the most common one in ML.
A basis is a set of linearly independent vectors (defined in the next section) that span the space. In , the standard basis is:
Every vector is a linear combination of basis vectors: .
Orthogonal bases are special
When basis vectors are mutually orthogonal (all pairwise dot products = 0), projection onto any basis vector is simply the dot product — no cross-term contamination. This makes orthogonal bases computationally clean and geometrically intuitive.
Linear independence and span
Vectors are linearly independent if no one of them can be written as a combination of the others. Equivalently, implies all .
The span of a set is all vectors you can reach by linear combination — it describes the "reachable" subspace.
Worked example: dependent vs independent
Take and . Notice , so with non-zero coefficients. They are dependent. Their span is just the line through ; no combination ever leaves it, so they cannot describe a 2D plane.
Now take and . Setting gives the system:
From the first equation . Substituting into the second: , so and therefore . The only solution is the trivial one, so and are independent and their span is all of — they form a basis.
Computing rank with row reduction
Stack the vectors as columns of a matrix, then apply Gaussian elimination (row reduction): swap rows, scale rows, and subtract multiples of one row from another to reach row echelon form. Count the non-zero rows — that count is the rank. Full rank (rank = number of columns) ⇒ all vectors are independent.
Example: → subtract 2×row1 from row2 → . One non-zero row → rank 1 → vectors are dependent (). ✓
Exercise
What is the dot product of u = [3, 1] and v = [2, 4]?
Need a hint?
Multiply component-wise and sum: u₁v₁ + u₂v₂.
Exercise
Two vectors are orthogonal if their dot product equals what?
Need a hint?
Think about what cos(90°) equals.
Exercise
What is the Euclidean norm of v = [3, 4]?
Need a hint?
Use the Pythagorean theorem: ‖v‖ = √(v₁² + v₂²).
Common mistakes
- Confusing the dot product with the cross product: the dot product is a scalar; the cross product is a vector (and only defined in ). Also, cosine similarity requires normalizing both vectors first — skipping normalization makes it length-dependent, not angle-dependent.
- Assuming all vector spaces are Euclidean: is not the only vector space. Function spaces (e.g. continuous functions on ) and probability distributions (under appropriate operations) are also vector spaces. ML uses many non-Euclidean spaces.
- Confusing a basis with just any spanning set: a basis must be both linearly independent and spanning. A spanning set with redundant vectors is not a basis; removing the redundancy (e.g. via row reduction) is needed to obtain one.
Key takeaways
- A vector is a list of numbers representing a point/direction in space.
- The dot product is the core of every linear model — it measures similarity and projects one vector onto another.
- Orthogonal vectors have zero dot product; orthogonal bases simplify projection.
- Norm measures length; normalizing a vector gives direction without magnitude.
- In project-loop terms: vectors are the data representation, and the dot product is the linear hypothesis space — two of the six slots, before you've trained anything.
Connection to ML
The vector operations in this lesson appear at the core of almost every ML algorithm:
- Linear models: is a single dot product between the weight vector and the input. The dot product is the prediction.
- Neural network neurons: each neuron computes a dot product before applying a nonlinearity. A fully-connected layer applies this for every row of the weight matrix simultaneously.
- Cosine similarity: used in word embeddings and recommendation systems to measure how similar two vectors are independent of their length: .
- Least-squares regression: the OLS solution projects the target vector onto the column space of the design matrix . The projection formula derived above is exactly what computes.
- PCA: finds the orthogonal basis of maximum variance. Projecting data onto principal components is a sequence of dot products with those basis vectors.
- Rank deficiency: if a feature matrix is rank-deficient, two features are linearly dependent. The model gains no new information from the second feature, and OLS becomes unstable.
Related concepts
- Matrices and Transformations: how vectors transform
- Eigenvalues and Eigenvectors: special directions in linear transformations
- What is a Neuron?: the dot product in action