Vectors and SpacesOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Vectors and Spaces

Every piece of data in machine learning (an image, a sentence, a medical record) is eventually represented as a vector: a list of numbers with direction and magnitude. That representation is the data stage of every ML project; and the dot product you build on it is, quite literally, the whole of a linear model — the simplest hypothesis space there is. Almost everything you learn later is a change to one slot of that loop, so it pays to start at the two slots linear algebra owns.

Deep dive
The ML Project Loop
Includes runnable notebook

What is a vector?

A vector v∈Rn\mathbf{v} \in \mathbb{R}^n is an ordered list of nn real numbers:

v=[v1v2⋮vn]\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}

Geometrically, you can think of it as an arrow from the origin to a point in nn-dimensional space. In ML, nn is often the number of features.

Real-world example: A house with area 1500 sq ft, 3 bedrooms, and price $300k is the vector x=[1500,3,300000]⊤\mathbf{x} = [1500, 3, 300000]^\top. (Here we pack everything — even the price — into one vector; in supervised learning you'll usually keep the target yy separate from the features x\mathbf{x}.)

Core operations

Addition and scaling

u+v=[u1+v1u2+v2],cv=[cv1cv2]\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_1 + v_1 \\ u_2 + v_2 \end{bmatrix}, \qquad c\mathbf{v} = \begin{bmatrix} cv_1 \\ cv_2 \end{bmatrix}

These two operations are the building blocks of all linear algebra, and of every algorithm in this course.

The Euclidean norm

The length (norm) of a vector measures how far it points from the origin:

∥v∥=v12+v22+⋯+vn2=v⊤v\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\mathbf{v}^\top \mathbf{v}}

A unit vector has norm 1. You can normalize any vector: v^=v/∥v∥\hat{\mathbf{v}} = \mathbf{v} / \|\mathbf{v}\|.

Python
import numpy as np

v = np.array([3.0, 4.0])
norm = np.linalg.norm(v)       # 5.0
unit = v / norm                 # [0.6, 0.8]
print(f"‖v‖ = {norm}")
print(f"unit vector: {unit}")

The dot product

The dot product is the most fundamental operation in ML:

u⋅v=u⊤v=∑i=1nuivi=∥u∥∥v∥cos⁡θ\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^\top \mathbf{v} = \sum_{i=1}^n u_i v_i = \|\mathbf{u}\|\|\mathbf{v}\|\cos\theta

where θ\theta is the angle between the vectors.

Sign tells direction, magnitude tells similarity

When u⋅v>0\mathbf{u}\cdot\mathbf{v} > 0 the vectors point roughly the same way; when it equals zero they are perpendicular; when it is negative they diverge. Dividing by both norms gives cosine similarity cos⁡θ=u^⋅v^\cos\theta = \hat{\mathbf{u}}\cdot\hat{\mathbf{v}} — a pure angle measure independent of length.

What the dot product measures

u⋅v\mathbf{u} \cdot \mathbf{v}Meaning
>0> 0Vectors point in roughly the same direction
=0= 0Vectors are orthogonal (perpendicular)
<0< 0Vectors point in roughly opposite directions

Worked example

u=[2,1]\mathbf{u} = [2, 1], v=[1,3]\mathbf{v} = [1, 3]:

u⋅v=2×1+1×3=5\mathbf{u} \cdot \mathbf{v} = 2 \times 1 + 1 \times 3 = 5 cos⁡θ=55⋅10=550≈0.707  ⟹  θ≈45°\cos\theta = \frac{5}{\sqrt{5} \cdot \sqrt{10}} = \frac{5}{\sqrt{50}} \approx 0.707 \implies \theta \approx 45°
Python
u = np.array([2.0, 1.0])
v = np.array([1.0, 3.0])

dot = np.dot(u, v)              # 5.0
cos_theta = dot / (np.linalg.norm(u) * np.linalg.norm(v))  # ≈ 0.707
angle_deg = np.degrees(np.arccos(cos_theta))                # ≈ 45°

Vectors and the dot product

aba+b
a · b5.50
|a|3.16
|b|2.69
angle θ50°

The dot product a·b = |a||b|cos θ. It's zero when the vectors are perpendicular (θ = 90°), positive when they point similarly, negative when opposed. The faint segment is b projected onto a.

Projection: the dot product's geometric job

The dot product answers "how much of b\mathbf{b} points along a\mathbf{a}?" That quantity is the projection of b\mathbf{b} onto a\mathbf{a} — the shadow b\mathbf{b} casts on the line through a\mathbf{a}. Start from a⋅b=∥a∥∥b∥cos⁡θ\mathbf{a}\cdot\mathbf{b} = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta and divide by ∥a∥\|\mathbf{a}\|:

scalar proj=∥b∥cos⁡θ=a⋅b∥a∥,vector projab=a⋅ba⋅a⏟how many copies of a a\text{scalar proj} = \|\mathbf{b}\|\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\|\mathbf{a}\|}, \qquad \text{vector proj}_{\mathbf{a}}\mathbf{b} = \underbrace{\frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{a}\cdot\mathbf{a}}}_{\text{how many copies of }\mathbf{a}}\,\mathbf{a}

The denominator a⋅a=∥a∥2\mathbf{a}\cdot\mathbf{a} = \|\mathbf{a}\|^2 rescales by a\mathbf{a}'s length so the result lands exactly on the line through a\mathbf{a}. Drag the vectors above to watch the projection segment grow and shrink with the angle.

Vector spaces and basis

A vector space is a set of vectors that is closed under addition and scalar multiplication. Rn\mathbb{R}^n is the most common one in ML.

A basis is a set of linearly independent vectors (defined in the next section) that span the space. In R2\mathbb{R}^2, the standard basis is:

e1=[10],e2=[01]\mathbf{e}_1 = \begin{bmatrix}1\\0\end{bmatrix}, \quad \mathbf{e}_2 = \begin{bmatrix}0\\1\end{bmatrix}

Every vector is a linear combination of basis vectors: v=v1e1+v2e2\mathbf{v} = v_1\mathbf{e}_1 + v_2\mathbf{e}_2.

Orthogonal bases are special

When basis vectors are mutually orthogonal (all pairwise dot products = 0), projection onto any basis vector is simply the dot product — no cross-term contamination. This makes orthogonal bases computationally clean and geometrically intuitive.

Linear independence and span

Vectors v1,…,vk\mathbf{v}_1, \ldots, \mathbf{v}_k are linearly independent if no one of them can be written as a combination of the others. Equivalently, ∑icivi=0\sum_i c_i \mathbf{v}_i = \mathbf{0} implies all ci=0c_i = 0.

The span of a set is all vectors you can reach by linear combination — it describes the "reachable" subspace.

Worked example: dependent vs independent

Take a=[1,2]\mathbf{a} = [1, 2] and b=[2,4]\mathbf{b} = [2, 4]. Notice b=2a\mathbf{b} = 2\mathbf{a}, so 2a−b=02\mathbf{a} - \mathbf{b} = \mathbf{0} with non-zero coefficients. They are dependent. Their span is just the line through a\mathbf{a}; no combination ever leaves it, so they cannot describe a 2D plane.

Now take a=[1,2]\mathbf{a} = [1, 2] and c=[2,1]\mathbf{c} = [2, 1]. Setting c1a+c2c=0c_1\mathbf{a} + c_2\mathbf{c} = \mathbf{0} gives the system:

c1+2c2=0(top row),2c1+c2=0(bottom row)c_1 + 2c_2 = 0 \quad \text{(top row)}, \qquad 2c_1 + c_2 = 0 \quad \text{(bottom row)}

From the first equation c1=−2c2c_1 = -2c_2. Substituting into the second: 2(−2c2)+c2=−3c2=02(-2c_2) + c_2 = -3c_2 = 0, so c2=0c_2 = 0 and therefore c1=0c_1 = 0. The only solution is the trivial one, so a\mathbf{a} and c\mathbf{c} are independent and their span is all of R2\mathbb{R}^2 — they form a basis.

Computing rank with row reduction

Stack the vectors as columns of a matrix, then apply Gaussian elimination (row reduction): swap rows, scale rows, and subtract multiples of one row from another to reach row echelon form. Count the non-zero rows — that count is the rank. Full rank (rank = number of columns) ⇒ all vectors are independent.

Example: [1224]\begin{bmatrix}1&2\\2&4\end{bmatrix} → subtract 2×row1 from row2 → [1200]\begin{bmatrix}1&2\\0&0\end{bmatrix}. One non-zero row → rank 1 → vectors are dependent (b=2a\mathbf{b} = 2\mathbf{a}). ✓

Exercise

What is the dot product of u = [3, 1] and v = [2, 4]?

Need a hint?

Multiply component-wise and sum: u₁v₁ + u₂v₂.

Exercise

Two vectors are orthogonal if their dot product equals what?

Need a hint?

Think about what cos(90°) equals.

Exercise

What is the Euclidean norm of v = [3, 4]?

Need a hint?

Use the Pythagorean theorem: ‖v‖ = √(v₁² + v₂²).

Common mistakes

  1. Confusing the dot product with the cross product: the dot product u⋅v\mathbf{u}\cdot\mathbf{v} is a scalar; the cross product is a vector (and only defined in R3\mathbb{R}^3). Also, cosine similarity requires normalizing both vectors first — skipping normalization makes it length-dependent, not angle-dependent.
  2. Assuming all vector spaces are Euclidean: Rn\mathbb{R}^n is not the only vector space. Function spaces (e.g. continuous functions on [0,1][0,1]) and probability distributions (under appropriate operations) are also vector spaces. ML uses many non-Euclidean spaces.
  3. Confusing a basis with just any spanning set: a basis must be both linearly independent and spanning. A spanning set with redundant vectors is not a basis; removing the redundancy (e.g. via row reduction) is needed to obtain one.

Key takeaways

  • A vector is a list of numbers representing a point/direction in space.
  • The dot product w⊤x\mathbf{w}^\top \mathbf{x} is the core of every linear model — it measures similarity and projects one vector onto another.
  • Orthogonal vectors have zero dot product; orthogonal bases simplify projection.
  • Norm measures length; normalizing a vector gives direction without magnitude.
  • In project-loop terms: vectors are the data representation, and the dot product is the linear hypothesis space — two of the six slots, before you've trained anything.

Connection to ML

The vector operations in this lesson appear at the core of almost every ML algorithm:

  • Linear models: f(x)=w⊤x+bf(\mathbf{x}) = \mathbf{w}^\top\mathbf{x} + b is a single dot product between the weight vector and the input. The dot product is the prediction.
  • Neural network neurons: each neuron computes a dot product w⊤x\mathbf{w}^\top\mathbf{x} before applying a nonlinearity. A fully-connected layer applies this for every row of the weight matrix simultaneously.
  • Cosine similarity: used in word embeddings and recommendation systems to measure how similar two vectors are independent of their length: sim(u,v)=u^⋅v^\text{sim}(\mathbf{u}, \mathbf{v}) = \hat{\mathbf{u}}\cdot\hat{\mathbf{v}}.
  • Least-squares regression: the OLS solution projects the target vector y\mathbf{y} onto the column space of the design matrix X\mathbf{X}. The projection formula derived above is exactly what (X⊤X)−1X⊤y(\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y} computes.
  • PCA: finds the orthogonal basis of maximum variance. Projecting data onto principal components is a sequence of dot products with those basis vectors.
  • Rank deficiency: if a feature matrix is rank-deficient, two features are linearly dependent. The model gains no new information from the second feature, and OLS becomes unstable.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

RelatedGo deeper
Vectors and SpacesMatrices and Transformati…Eigenvalues and Eigenvect…What is a Neuron?The ML Project Loop