Activation Functions: From Sigmoid to GELU
An activation function is the non-linearity applied to a neuron's pre-activation . Without it, stacking layers is pointless: the composition of linear maps is still linear, so a 100-layer network would collapse to a single matrix multiply. The activation is what lets depth buy expressive power, and the choice of activation governs how gradients flow, which is the difference between a network that trains and one that stalls.
The job of an activation
Two properties matter for training a deep network by gradient descent:
- Non-linearity — so composing layers produces genuinely new functions (curved decision boundaries, not just hyperplanes).
- A well-behaved derivative: backprop multiplies derivatives layer by layer. If each factor is much smaller than 1, the product shrinks toward zero through depth (the vanishing gradient); if much larger, it explodes. The activation's derivative is one factor in every one of those products.
This is why "use a non-linearity" is necessary but not sufficient — sigmoid is non-linear yet trains deep nets poorly, because its derivative saturates.
Saturation and the vanishing gradient
A function saturates when its output flattens for large , driving its derivative toward 0. Sigmoid has derivative , which peaks at just 0.25 (at ) and is near 0 once . Stack ten sigmoid layers and the gradient reaching the first layer is scaled by at most — the early layers barely learn.
Sigmoid is also not zero-centered
Sigmoid outputs are all positive (in ), so every weight feeding into a downstream neuron receives a gradient of the same sign on a given example. That forces inefficient zig-zag updates. Tanh fixes the centering (range ) but still saturates — its derivative peaks at 1 but vanishes in the tails just like sigmoid.
ReLU and its family
ReLU is the modern default for hidden layers. Its derivative is exactly 1 for (it does not shrink gradients in the active region) and it costs a single comparison. That combination is why deep nets became trainable at scale.
Its weakness is the dead/dying ReLU: a neuron whose pre-activation is negative for every input outputs 0 with derivative 0, so it receives no gradient and can never recover. A bad initialization or too-large a learning rate can kill a large fraction of units permanently.
The variants patch this leak by giving negative inputs a small, nonzero slope:
| Function | Formula ( branch) | Negative-side gradient | Note |
|---|---|---|---|
| Leaky ReLU | (e.g. ) | constant | never fully dies |
| PReLU | , learned | learned per-channel | extra parameters |
| ELU | smooth, in tail | pushes mean activation toward 0 |
Smooth modern activations: GELU and Swish
Transformers and recent vision models favor smooth, non-monotonic activations that empirically train better than ReLU at scale:
- GELU (Gaussian Error Linear Unit): , where is the standard-normal CDF. It weights the input by the probability it is retained — a "soft" gate. The default in BERT, GPT, and ViT.
- Swish / SiLU: . Closely related to GELU; self-gated and smooth. Used in EfficientNet and many LLMs (often as the gate in SwiGLU feed-forward blocks).
Both are smooth everywhere (no kink at 0) and allow small negative outputs, which helps gradient flow without the dead-unit problem.
Softmax: a special case for outputs
Softmax is not a hidden-layer activation; it is the output layer for multi-class classification. It maps a vector of logits to a probability distribution:
It couples all outputs (they sum to 1) and pairs with cross-entropy loss, whose combined gradient simplifies to the clean . For numerical stability, subtract from every logit before exponentiating. This is exact (softmax is shift-invariant) and prevents overflow.
Derivatives at a glance
| Activation | Max of | ||
|---|---|---|---|
| Sigmoid | 0.25 | ||
| Tanh | 1.0 | ||
| ReLU | 1.0 | ||
| Leaky ReLU | if else | 1.0 | |
| ELU | if else | if else | 1.0 |
The pattern that matters: sigmoid's ceiling vs ReLU's is the heart of why the field moved away from saturating activations for hidden layers.
A practical chooser
- Hidden layers, default: ReLU. Cheap, well-understood, strong baseline.
- ReLU units dying: switch to Leaky ReLU / ELU, or lower the learning rate and fix initialization.
- Transformers / large models: GELU (or SwiGLU in the feed-forward block).
- Binary classification output: sigmoid (one logit → probability).
- Multi-class output: softmax over the logits.
- Regression output: usually no activation (linear) on the final layer.
Initialization is tuned to the activation
He initialization is designed for ReLU (variance ); Xavier/Glorot for tanh/sigmoid (). Pairing the wrong init with an activation reintroduces vanishing or exploding signals on the very first forward pass, which is why activation choice and weight init are decided together.
Related concepts
- What Is a Neuron: where the activation sits in the single-neuron computation
- Weight Initialization: He vs Xavier, chosen to match the activation's gradient behavior
- Softmax & Cross-Entropy: the output-layer activation and its loss, derived in full
- Vanishing Gradient Analysis: the saturation problem traced through a deep network