CNN Visualization & Adversarial Attacks
A trained CNN achieves 95% accuracy, but what did it actually learn? And why does adding invisible pixel noise make it misclassify a cat as a guacamole with 99% confidence? These two questions (interpretability and robustness) define modern CNN analysis.
Why visualization matters
Neural networks are often called "black boxes," but their internals are inspectable. Visualization methods answer questions like:
- Which pixels drove this prediction?
- What does neuron 47 in layer 3 respond to?
- Are the learned features sensible or are they spurious correlations?
These answers are essential for debugging, regulatory compliance ("why did the model deny this loan?"), and catching dataset biases before deployment.
1. Saliency maps
The simplest visualization: compute the gradient of the predicted class score with respect to the input pixels.
Large means pixel had a large influence on the output. Display the magnitude as a heatmap overlaid on the image.
Algorithm:
- Forward pass: compute the score for the predicted class
- Backward pass: compute (gradient with respect to the input)
- Take the absolute value (or max over channels for color images)
- Display as a heatmap
Guided Backpropagation
Standard saliency can be noisy. Guided Backpropagation modifies the backward pass to only propagate positive gradients through ReLU layers, producing sharper and more interpretable maps. It zeros out gradients wherever the activation OR the gradient is negative.
Limitation: Saliency maps show pixel-level sensitivity, not semantic regions. Two pixels that are equally sensitive may have totally different roles. Also, gradients saturate in deep networks, making the maps noisy.
2. Grad-CAM
Gradient-weighted Class Activation Mapping (Grad-CAM) fixes the resolution problem by working at the last convolutional layer, where feature maps still encode spatial structure.
Key insight: the last conv layer feature maps tell you where the network is looking; the gradients flowing into that layer tell you which maps matter for the target class.
Algorithm:
- Forward pass; extract feature maps from the last conv layer
- Backprop: compute gradient of class score with respect to each feature map:
- Global average pool these gradients to get importance weights:
- Weighted combination:
- Upsample to input resolution and overlay on the image
The ReLU keeps only the features that increase the class score. The result is a coarse heatmap highlighting the discriminative image regions.
Grad-CAM vs saliency
Saliency is high-resolution but noisy; Grad-CAM is coarse (14×14 for a 224×224 image) but semantically meaningful. Grad-CAM++ extends this to handle multiple instances of the same object. In practice, Grad-CAM is the most commonly used diagnostic tool in CNN interpretability.
3. Activation maximization
Instead of asking "what does the network respond to in this image?", ask: "what image would maximally activate this neuron?"
Algorithm:
- Fix the network weights
- Start from random noise
- Iteratively optimize: (gradient ascent on activation)
- Add regularization (L2 on pixel values, total variation smoothness) to keep the image realistic
The result is a synthetic image that represents the neuron's "template." Early layers produce oriented edges and color blobs; deeper layers produce increasingly complex textures and object parts.
4. t-SNE of CNN feature codes
Run your entire dataset through the network, extract the penultimate layer's activations (the embedding before the classifier), and apply t-SNE to reduce to 2D. If the network has learned good representations, images that look similar should cluster nearby.
This visualization reveals:
- Whether the network separates classes cleanly in feature space
- Which classes are confused with each other
- Whether the network has learned semantic clusters (all "cats" together regardless of breed)
5. Adversarial examples
Here is something striking: you can fool a state-of-the-art CNN with changes invisible to humans.
Take an image classified correctly as "panda." Add a carefully crafted noise pattern — each pixel changes by at most 0.007 out of 255 — and the CNN now classifies it as "gibbon" with 99.3% confidence. The image looks identical to humans.
This reveals that CNNs learn a very different representation of images than humans do. They are sensitive to high-frequency pixel patterns that humans ignore.
6. Fast Gradient Sign Method (FGSM)
FGSM (Goodfellow et al., 2014) is the simplest adversarial attack. The idea: instead of optimizing the prediction (gradient descent on the loss), take one step in the direction that maximizes the loss.
- — original clean image
- — true label
- — cross-entropy loss
- — perturbation magnitude (e.g., 8/255 for imperceptible change)
The function makes every pixel move the maximum allowed amount in the loss-increasing direction. This is a white-box attack (requires access to the model's gradients).
Why it works: CNNs are highly sensitive to many dimensions simultaneously. Even if each pixel changes by a tiny amount, the sum of changes across thousands of pixels can completely flip the prediction.
For targeted attack (force output to class ):
Minimize loss toward the target class instead of maximizing loss away from the true class.
The attack, on a model small enough to check
Why the attack is about dimension, not images
A logistic regression trained in your browser on 300 synthetic 12×12 images (400 full-batch steps, 100% train accuracy). Left to right: the clean image, sign(∇ₓ loss) — which for a linear model is just sign(w) — and the adversarial image x + ε·sign(w). Everything below is measured from that fit.
For a linear model the whole attack is exactly solvable, which is why Goodfellow's original paper argued from one. The perturbation moves the logit by
and the panel checks that measured shift against the prediction on every step. So the budget needed to flip a decision is .
On this 144-pixel model that comes out at 43/255 — the canonical flips nothing. That is the honest result, and it is the one worth having: a low-dimensional model is not meaningfully vulnerable to an imperceptible attack. Adversarial fragility does not come from images, or from deep networks, or from nonlinearity. It comes from the denominator.
grows roughly linearly with the number of inputs at fixed mean weight, so drag the dimension slider: at the same model shape has and — a thousandth of a pixel level. Each pixel still moves by an imperceptible ; there are simply 150,000 of them, all pushing the same way. That is the entire phenomenon.
7. PGD: iterative FGSM
Projected Gradient Descent (PGD) extends FGSM to steps with a smaller per-step size , projecting back to the -ball after each step:
where is the allowed perturbation set. PGD finds stronger adversarial examples than FGSM because it refines the perturbation over multiple steps. It is considered the "standard" attack for evaluating robustness.
8. Defenses
Adversarial training: the most reliable defense: include adversarial examples in the training set. During each training iteration, generate adversarial examples from the current model and train on both clean and adversarial data.
This is a minimax problem: the inner max finds the worst-case perturbation; the outer min trains the model to resist it. Models trained this way are called adversarially trained and are much more robust — at some cost to clean accuracy.
Other defenses:
- Input preprocessing: Gaussian blurring, JPEG compression, or feature squeezing to destroy adversarial structure before it reaches the model
- Randomized smoothing: certifiably robust predictions via Monte Carlo noise injection
- Certified defenses: prove mathematically that no attack within an ball can fool the model; computationally expensive but provides hard guarantees
Arms race dynamics
Adversarial defense is an ongoing arms race. Many proposed defenses have been broken by adaptive attacks (attacks that know the defense). Adversarial training with PGD is currently the most empirically reliable approach, but no method achieves both full robustness and full accuracy.
Poisoning vs. evasion attacks
| Attack type | When | Requires |
|---|---|---|
| Poisoning | During training | Access to training data |
| Evasion | At test time | Access to model (or its outputs) |
FGSM and PGD are evasion attacks. Poisoning attacks (e.g., "backdoor attacks" that trigger misclassification on images with a specific watermark) require compromising the training pipeline.
Exercise
Grad-CAM computes importance weights for each feature map in the last conv layer. How are these weights calculated?
Need a hint?
It uses gradients flowing back from the class score to the feature maps.
Exercise
A saliency map highlights which pixels most influence the predicted class. What gradient is computed to create it?
Need a hint?
We want to know how the output changes as each input pixel changes.
Exercise
FGSM creates an adversarial example as x_adv = x + ε·sign(∇ₓ J(θ, x, y)). What does taking the sign() of the gradient achieve?
Need a hint?
Think about the worst-case perturbation under an ℓ∞ constraint.
Key takeaways
- Saliency maps show pixel-level sensitivity via input gradients; Grad-CAM gives coarser but more semantically meaningful spatial attention maps using the last conv layer.
- Activation maximization synthesizes the "ideal" input for a neuron; t-SNE of features reveals how the network clusters images in embedding space.
- Adversarial examples are imperceptible perturbations that fool a CNN — they exist because CNNs are sensitive to high-dimensional linear structure humans ignore.
- FGSM generates adversarial examples in one gradient step; PGD iterates for stronger attacks.
- Adversarial training (training on adversarial examples) is the most robust defense, at the cost of some clean accuracy.
Related concepts
- The Convolution Operation: the feature maps that Grad-CAM visualizes
- Chain Rule & Backpropagation: the input-gradient computation behind saliency maps and FGSM
- Generative Adversarial Networks: adversarial dynamics in a generative setting