Deep Learning for Time Series
ARIMA is powerful, but it rests on two assumptions that many real-world series violate: linearity and univariate structure. A factory's energy consumption depends non-linearly on temperature, day-of-week, and production volume simultaneously. A stock's next-day price is influenced by thousands of correlated assets. Deep learning drops both assumptions, learning complex, multivariate, non-linear forecasting functions directly from data.
Three architectural families have proved most useful for time series:
- LSTMs (and GRUs): handle long-range dependencies through gated memory
- Temporal Convolutional Networks (TCNs): parallelizable dilated convolutions with large receptive fields
- Transformer-based models: attention over arbitrary positions; excel at very long sequences
All three share the same input representation: a sliding window that converts the raw sequence into a supervised learning problem.
The sliding window approach
Every deep forecasting model ingests a fixed-length window of past observations and outputs one or more future values. This reframes time series as standard supervised learning.
Formally:
- Input window of length :
- Forecast horizon :
Concrete example with , , series :
| Input window | Target |
|---|---|
Each row is one training sample. For (multi-step), the target becomes a vector. The dataset has samples, so longer windows reduce the number of training examples.
import numpy as np
def sliding_windows(series: np.ndarray, W: int, H: int):
"""
Convert a 1-D time series into (X, y) arrays for supervised learning.
X shape: (n_samples, W)
y shape: (n_samples, H)
"""
X, y = [], []
n = len(series)
for i in range(n - W - H + 1):
X.append(series[i : i + W])
y.append(series[i + W : i + W + H])
return np.array(X), np.array(y)
# Example
series = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=float)
X, y = sliding_windows(series, W=5, H=1)
print("X:\n", X)
# [[1. 2. 3. 4. 5.]
# [2. 3. 4. 5. 6.]
# [3. 4. 5. 6. 7.]]
print("y:\n", y)
# [[6.] [7.] [8.]]
For multivariate series with features at each time step, becomes a matrix of shape and the same window logic applies independently across features.
LSTM for time series
LSTMs are covered in depth in the LSTM & GRU lesson. Here we focus on the forecasting-specific design choices.
Many-to-one (point forecast): feed the window through the LSTM, take the last hidden state , and map it through a linear layer to predict future values. Simple and effective for short horizons.
Sequence-to-sequence (multi-step): encode the input window, then decode step-by-step. Teacher forcing during training passes ground truth as decoder input; at inference, predictions are fed back.
Many-to-one vs many-to-many
For most industrial forecasting tasks (hourly demand for the next 24 hours, daily sales for the next 7 days), a many-to-one LSTM that outputs all steps from the final hidden state is simpler to implement and often just as accurate as a full seq2seq model. Reserve seq2seq for very long horizons () where the decoder's additional context becomes critical.
import torch
import torch.nn as nn
class LSTMForecaster(nn.Module):
"""
Many-to-one LSTM forecaster.
Input: (batch, W, n_features)
Output: (batch, H)
"""
def __init__(self, n_features: int, hidden_size: int, n_layers: int, horizon: int,
dropout: float = 0.1):
super().__init__()
self.lstm = nn.LSTM(
input_size=n_features,
hidden_size=hidden_size,
num_layers=n_layers,
batch_first=True,
dropout=dropout if n_layers > 1 else 0.0,
)
self.head = nn.Linear(hidden_size, horizon)
def forward(self, x: torch.Tensor) -> torch.Tensor:
# x: (batch, W, n_features)
output, (h_n, _) = self.lstm(x)
# h_n: (n_layers, batch, hidden_size) — take the last layer
last_hidden = h_n[-1] # (batch, hidden_size)
return self.head(last_hidden) # (batch, H)
# Quick smoke test
model = LSTMForecaster(n_features=1, hidden_size=64, n_layers=2, horizon=12)
x_batch = torch.randn(32, 24, 1) # batch=32, window=24, univariate
print(model(x_batch).shape) # torch.Size([32, 12])
Temporal Convolutional Networks (TCN)
TCNs replace recurrence with dilated causal convolutions — a design that is fully parallelizable (unlike LSTMs, which process steps sequentially) and achieves large receptive fields through exponentially increasing dilation factors.
Causal convolution: at time , the output depends only on — never on the future. This is enforced by left-padding the input.
Dilation: a convolution with kernel size and dilation skips positions between kernel elements. At layer with dilation , a single kernel spans positions. Stacking layers with doubling dilation gives a total receptive field:
With and (9 layers), the receptive field reaches time steps — far more than a practical LSTM can handle stably. (With two stacked conv layers per residual block, as in the original Bai et al. TCN, this roughly doubles again.)
Key advantages over LSTM:
- Fully parallelizable: compute all timesteps simultaneously during training
- Stable gradients: no vanishing gradient through long sequences (residual connections help too)
- Smaller memory footprint: no hidden state to store and transport
class TCNBlock(nn.Module):
"""
One residual TCN block: two dilated causal conv layers + residual skip.
"""
def __init__(self, in_channels: int, out_channels: int, kernel_size: int,
dilation: int, dropout: float = 0.1):
super().__init__()
pad = (kernel_size - 1) * dilation # left-only padding
self.net = nn.Sequential(
nn.Conv1d(in_channels, out_channels, kernel_size,
padding=pad, dilation=dilation),
nn.Chomp1d(pad), # remove right-side padding
nn.ReLU(),
nn.Dropout(dropout),
nn.Conv1d(out_channels, out_channels, kernel_size,
padding=pad, dilation=dilation),
nn.Chomp1d(pad),
nn.ReLU(),
nn.Dropout(dropout),
)
self.skip = (nn.Conv1d(in_channels, out_channels, 1)
if in_channels != out_channels else nn.Identity())
def forward(self, x):
return nn.functional.relu(self.net(x) + self.skip(x))
class Chomp1d(nn.Module):
def __init__(self, chomp_size):
super().__init__()
self.chomp_size = chomp_size
def forward(self, x):
return x[:, :, :-self.chomp_size].contiguous()
Transformer-based models
Vanilla Transformers (as in Attention & Transformers) apply full self-attention, whose cost scales quadratically with sequence length — impractical for time series with thousands of steps. Several specialised architectures address this:
Informer (Zhou et al. 2021): introduces ProbSparse attention that selects the most informative query-key pairs, reducing complexity from to . Designed specifically for long-sequence forecasting.
PatchTST (Nie et al. 2023): divides the time series into non-overlapping patches (similar to ViT image patches) and applies standard Transformer attention over the patch tokens. Reduces the number of attention computations dramatically while preserving local temporal patterns within each patch.
Temporal Fusion Transformer (TFT) (Lim et al. 2021): combines multi-head attention with gated residual networks and variable selection networks. The variable selection mechanism provides built-in interpretability: for each forecast, it identifies which input features (past values, known covariates) contributed most.
ARIMA Forecast with Confidence Intervals
Observed data (left) and multi-step ahead forecast (right). The dark band is the 80% prediction interval; the lighter band is 95%. Intervals widen as the forecast horizon increases because uncertainty compounds.
Evaluation metrics
A model is only as good as the metric you use to judge it. Different metrics emphasise different failure modes:
| Metric | Formula | When to use |
|---|---|---|
| MAE | Robust to outliers; same units as the series | |
| RMSE | Penalises large errors more; differentiable | |
| MAPE | Scale-independent; undefined when | |
| sMAPE | Bounded ; symmetric |
MAPE pitfalls
MAPE is asymmetric: a 50% over-prediction and a 50% under-prediction yield different MAPE values. It also breaks whenever the actual value is zero or near zero — common in demand forecasting (a product not ordered in a particular week). sMAPE mitigates the asymmetry but introduces its own quirks. When in doubt, report both MAE and RMSE.
Walk-forward validation
Standard k-fold cross-validation randomly shuffles data, creating training samples that may include observations from the future relative to their validation point — a form of data leakage. Walk-forward (or expanding-window) validation respects the temporal order: train on , validate on , then expand the window and repeat. The reported error is the average across all validation windows.
Classical vs deep learning decision guide
Choosing between ARIMA and deep learning is a practical engineering decision, not a philosophical one. The table below summarises the key trade-offs:
| Scenario | Recommendation |
|---|---|
| Univariate, approximately linear, short series (< 500 points) | ARIMA |
| Multiple seasonality, interpretable forecasts required | SARIMA / Prophet |
| Multivariate inputs with complex non-linear interactions | LSTM / TCN |
| Very long-range dependencies (> 1000 lag steps) | Transformer-based (Informer, PatchTST) |
| Prediction intervals / probabilistic forecasts | ARIMA (exact) or quantile regression |
| Hundreds of related series (retail demand, sensor banks) | Global LSTM / TCN trained across all series |
| Real-time inference with tight latency budget | TCN (parallelizable, no sequential state) |
Python implementation
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset
# ── 0. Helpers ─────────────────────────────────────────────────────────────────
def sliding_windows(series, W, H):
X, y = [], []
for i in range(len(series) - W - H + 1):
X.append(series[i:i+W])
y.append(series[i+W:i+W+H])
return np.array(X, dtype=np.float32), np.array(y, dtype=np.float32)
def mae(y_true, y_pred):
return np.mean(np.abs(y_true - y_pred))
def rmse(y_true, y_pred):
return np.sqrt(np.mean((y_true - y_pred)**2))
# ── 1. Generate synthetic series ──────────────────────────────────────────────
np.random.seed(0)
T = 500
t = np.arange(T)
series = (0.05 * t
+ 5 * np.sin(2 * np.pi * t / 52) # annual seasonality
+ np.random.normal(0, 0.8, T))
# ── 2. Build sliding windows ──────────────────────────────────────────────────
W, H = 52, 4 # 52-week input window, 4-week forecast horizon
X, y = sliding_windows(series, W, H)
# ── 3. Train / val / test split (chronological) ───────────────────────────────
n = len(X)
n_train = int(0.7 * n)
n_val = int(0.15 * n)
X_train, y_train = X[:n_train], y[:n_train]
X_val, y_val = X[n_train:n_train+n_val], y[n_train:n_train+n_val]
X_test, y_test = X[n_train+n_val:], y[n_train+n_val:]
# ── 4. PyTorch datasets ───────────────────────────────────────────────────────
def to_loader(X, y, batch_size=32, shuffle=False):
ds = TensorDataset(torch.from_numpy(X).unsqueeze(-1), # (N, W, 1)
torch.from_numpy(y))
return DataLoader(ds, batch_size=batch_size, shuffle=shuffle)
train_dl = to_loader(X_train, y_train, shuffle=True)
val_dl = to_loader(X_val, y_val)
test_dl = to_loader(X_test, y_test)
# ── 5. Model, optimizer, loss ─────────────────────────────────────────────────
class LSTMForecaster(nn.Module):
def __init__(self, n_features, hidden_size, n_layers, horizon, dropout=0.1):
super().__init__()
self.lstm = nn.LSTM(n_features, hidden_size, n_layers,
batch_first=True,
dropout=dropout if n_layers > 1 else 0.0)
self.head = nn.Linear(hidden_size, horizon)
def forward(self, x):
_, (h_n, _) = self.lstm(x)
return self.head(h_n[-1])
model = LSTMForecaster(1, 64, 2, H)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()
# ── 6. Training loop ──────────────────────────────────────────────────────────
best_val_loss = float("inf")
for epoch in range(50):
model.train()
for xb, yb in train_dl:
optimizer.zero_grad()
loss = criterion(model(xb), yb)
loss.backward()
optimizer.step()
model.eval()
with torch.no_grad():
val_losses = [criterion(model(xb), yb).item() for xb, yb in val_dl]
val_loss = np.mean(val_losses)
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(model.state_dict(), "best_lstm.pt")
if (epoch + 1) % 10 == 0:
print(f"Epoch {epoch+1:3d} | val MSE = {val_loss:.4f}")
# ── 7. Evaluate on test set ───────────────────────────────────────────────────
model.load_state_dict(torch.load("best_lstm.pt"))
model.eval()
preds, targets = [], []
with torch.no_grad():
for xb, yb in test_dl:
preds.append(model(xb).numpy())
targets.append(yb.numpy())
preds = np.vstack(preds)
targets = np.vstack(targets)
print(f"Test MAE: {mae(targets, preds):.4f}")
print(f"Test RMSE: {rmse(targets, preds):.4f}")
# ── 8. Walk-forward cross-validation (manual expanding window) ────────────────
initial_size = 200 # minimum training set size
step = 10 # expand by 10 observations per fold
fold_errors = []
for fold_end in range(initial_size + W + H, n + 1, step):
X_cv_train = X[:fold_end - H - W]
y_cv_train = y[:fold_end - H - W]
X_cv_val = X[fold_end - H - W : fold_end - H - W + 1]
y_cv_val = y[fold_end - H - W : fold_end - H - W + 1]
if len(X_cv_val) == 0:
continue
# (In practice: refit model here; we use a naive last-value baseline)
naive_pred = X_cv_val[:, -1:].repeat(H, axis=1) # repeat last obs
fold_errors.append(mae(y_cv_val, naive_pred))
print(f"Walk-forward CV MAE (naive baseline): {np.mean(fold_errors):.4f}")
Exercise
You have 5 years of daily stock prices. Why can't you use standard k-fold cross-validation to evaluate a forecasting model?
Need a hint?
What happens when k-fold assigns data from day 1200 to the training fold and day 1000 to the validation fold?
Exercise
From the series [3, 5, 7, 9, 11, 13, 15, 17], how many training samples does a sliding window with W=4 (input length) and H=1 (forecast horizon) produce?
Need a hint?
Count how many complete (window, target) pairs fit in the series.
Key takeaways
- Sliding windows turn any time series into a supervised learning dataset. The window length controls how far back the model looks; the horizon controls how far ahead it predicts.
- LSTMs model sequential dependencies through gated memory and are the standard starting point for non-linear univariate and multivariate forecasting.
- TCNs use dilated causal convolutions to achieve large receptive fields with full parallelism — often faster to train than LSTMs with competitive accuracy.
- Transformer-based models (Informer, PatchTST, TFT) handle very long-range dependencies and multivariate inputs but require more data and careful tuning.
- Walk-forward validation is mandatory for honest model comparison — random splits leak future information into the training set.
Related concepts
- LSTM & GRU: the gated memory architecture that underpins LSTM forecasting
- Attention & Transformers: self-attention mechanism used in Informer and PatchTST
- ARIMA Models: the classical baseline that deep learning must beat
- Recurrent Neural Networks: the simpler predecessor to LSTM