Layers & the Forward Pass
A single neuron computes one weighted sum. A layer is just many neurons looking at the same inputs in parallel, and the moment you stack neurons this way, the math collapses into one clean matrix multiplication. Running data through layer after layer is called the forward pass, and it is the only thing a trained network ever does at prediction time.
Real-world examples: Every model inference you've ever triggered (a photo tagged, a sentence autocompleted, a fraud alert raised) is one forward pass through stacked layers.
From one neuron to a layer
Recall that a single neuron computes . Now put neurons side by side, each with its own weight vector and bias, all reading the same input . Stack the weight vectors as the rows of a matrix and the biases into a vector :
Each row of is one neuron; each entry of is that neuron's activation. The activation is applied element-wise. A multi-layer network is then nothing but this rule applied repeatedly:
where is the output activation — sigmoid for binary classification, softmax for multi-class, or identity (linear) for regression.
The shape rule
A layer mapping inputs to outputs has and — that's parameters. Memorize "rows = outputs, columns = inputs" and you will never mis-shape a matrix again.
A fully worked 2→2→1 forward pass
Let's push concrete numbers through a tiny network: 2 inputs, one hidden layer of 2 ReLU neurons, and a sigmoid output. The parameters:
Step 1: hidden pre-activations. Each row of dots with , plus its bias:
So .
Step 2: apply ReLU element-wise. The second neuron's pre-activation is negative, so ReLU clamps it to zero:
The second neuron is inactive for this input — it contributes nothing downstream. For a different input it might switch on; that input-dependent on/off pattern is how ReLU networks build piecewise-linear functions.
Step 3: output layer. The output neuron weights the hidden activations and applies sigmoid:
The network predicts class 1 with probability . That's the entire forward pass — two matrix-vector products and two activations.
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))
W1 = np.array([[2.0, -1.0], [0.5, 1.0]]) # (2 hidden, 2 inputs)
b1 = np.array([0.5, -3.0])
W2 = np.array([[2.0, 3.0]]) # (1 output, 2 hidden)
b2 = np.array([1.0])
x = np.array([1.0, 2.0])
z1 = W1 @ x + b1 # [ 0.5 -0.5]
h1 = np.maximum(0, z1) # [ 0.5 0. ] ← ReLU clamps neuron 2
z2 = W2 @ h1 + b2 # [2.]
y = sigmoid(z2) # [0.8808]
print("z1 = {} h1 = {} z2 = {} y = {}".format(z1, h1, z2, y.round(4)))
Batches are free
Replace the vector with a matrix holding examples as rows and the same code processes the whole batch at once: . This is why GPUs make deep learning fast — a forward pass is just a few big matrix multiplications.
Why depth composes features
Each layer doesn't see the raw input — it sees the previous layer's activations. Layer 1 might learn simple detectors ("is bigger than ?"); layer 2 combines those detectors into more abstract ones ("are both edge detectors on?"). Mathematically, the network is a composition of functions:
The non-linearity between every pair of matrices is essential: without it, the product collapses into one matrix and the whole network is a single linear map, no matter how deep. With it, each layer can bend the representation, so a few stacked layers carve decision boundaries no single layer could.
Watch this happen below. Drag the inputs and follow the activations layer by layer — edge color shows weight sign, neuron brightness shows the activation flowing through:
A feed-forward network, live
Edges: brand = positive weight, rose = negative; thicker = larger magnitude. Neuron brightness shows its activation value. The output neuron is linear — change inputs, activation, width, and depth and watch the forward pass recompute.
Try this with the viz: switch to ReLU and move the sliders until some hidden neurons read exactly — those neurons are clamped off, and the output is momentarily insensitive to them. Then add a second hidden layer and notice how the same input change ripples differently through the deeper network.
How wide should a hidden layer be?
Width is capacity. With hidden ReLU neurons, a 2-input network can carve the plane into regions using up to "fold lines" — each hidden neuron contributes one. Two neurons can solve XOR; a wiggly spiral boundary may need dozens. The parameter count grows quickly though: a network has
The practical trade-off:
| Choice | Pros | Cons |
|---|---|---|
| Wider | More capacity per layer, easy to optimize | Parameters grow fast, features stay "shallow" |
| Deeper | Composes features hierarchically, parameter-efficient | Harder to train (vanishing gradients) without tricks |
Width is not a substitute for depth
The universal approximation theorem says one hidden layer can fit anything, but possibly only with exponentially many neurons. Depth re-uses intermediate features, so deep-and-narrow usually beats shallow-and-huge at equal parameter count. In practice: start with 1–2 hidden layers of modest width and grow only when underfitting.
Exercise
A hidden layer computes z = Wx + b = [-1.5, 2.0]. After applying ReLU element-wise, what is the activation vector h?
Need a hint?
ReLU(z) = max(0, z), applied to each entry separately.
Exercise
A network has 2 inputs, one hidden layer of 4 neurons, and 1 output. How many parameters (weights + biases) does it have in total?
Need a hint?
Each layer mapping n inputs to m outputs has m×n weights plus m biases. Sum both layers.
Exercise
Why does a deep network often need far fewer parameters than a single wide hidden layer to fit the same function?
Need a hint?
Think about what later layers receive as input.
Common mistakes
- Mis-shaped matrices: remember rows = outputs, columns = inputs;
W @ xfails (or silently broadcasts wrong) otherwise. - Applying the hidden activation to the output layer: a regression output squashed through ReLU can never be negative; a classification output needs sigmoid/softmax, not tanh.
- Forgetting that ReLU zeros are persistent: if a neuron's pre-activation is negative for every training input, it is "dead" and its weights stop receiving gradient.
Key takeaways
- A layer is neurons sharing an input: with — parameters.
- The forward pass is alternating matrix multiplications and element-wise activations, layer by layer.
- The non-linearity between layers is what stops the stack collapsing into a single linear map.
- Depth composes features: later layers build abstractions out of earlier ones, which is more parameter-efficient than going wide.
- Width adds capacity within a layer; each hidden ReLU neuron adds one "fold" to the function the network can express.
Related concepts
- What is a Neuron?: the unit each row of represents
- Matrices & Transformations: what does geometrically
- Chain Rule & Backpropagation: how gradients flow backwards through this same composition