ARIMA Models
The ARIMA family is the dominant classical approach to univariate time series forecasting. It is principled, interpretable, well-understood statistically, and often surprisingly competitive with more complex models on short, clean series. Understanding ARIMA also builds the vocabulary (stationarity, autoregression, moving averages, differencing) that underpins every modern forecasting framework.
The forecasting thread
This is the second stop for the site's running daily-demand problem from the project-loop hub. The linear-regression baseline treated it as a tabular regression; ARIMA instead models the series as a series — differencing away the trend, an AR term for the weekly echo, an MA term for the shocks. Same problem, a hypothesis space built for temporal structure, and it will still be blindsided by the day-400 regime shift, which is why lesson 04 and streaming ML exist.
ARIMA stands for AutoRegressive Integrated Moving Average. The three letters each correspond to one component of the model, and together they capture the most common structure found in real-world economic, scientific, and engineering time series.
The three components
AR(p): Autoregressive
An AR(p) model expresses the current value as a linear combination of its most recent values plus white noise:
where is a constant, are the autoregressive coefficients, and is independent white noise. The model is stable (stationary) when the roots of the characteristic polynomial all lie outside the unit circle.
Intuition: An AR(1) with says "tomorrow is 80% of today plus some noise." The series has a long memory — shocks decay geometrically but never vanish.
MA(q): Moving Average
An MA(q) model expresses the current value as the mean plus a linear combination of past forecast errors:
where is the series mean and are the moving-average coefficients. Unlike AR, an MA model has finite memory: a shock only affects directly if . Beyond lag , its influence is exactly zero.
Intuition: An MA(1) with says "today's value is partly determined by yesterday's forecast error." If we over-predicted yesterday, we correct partly today.
ARIMA(p, d, q)
The integrated part deals with non-stationarity. If the raw series is non-stationary but becomes stationary after differencing times, we model the -times-differenced series as an ARMA(p, q):
The model is written ARIMA(p, d, q) where is the AR order, is the number of differences (usually 0, 1, or 2), and is the MA order.
Choosing d: the Augmented Dickey-Fuller test
How many differences do you need? You don't guess — you test. The Augmented Dickey-Fuller (ADF) test checks whether a series has a unit root: a root of the AR characteristic polynomial sitting exactly on the unit circle, which is the precise mathematical signature of a random-walk-style non-stationary series. Its null hypothesis is "the series has a unit root"; a p-value below 0.05 rejects that null and declares the series stationary. The recipe is simple: run the test, and if it fails to reject, difference once more and re-test — the number of differences needed to reach stationarity is exactly your .
The memory fingerprint
AR and MA have opposite ACF/PACF signatures — a critical fact for model identification:
- AR(p): ACF tails off exponentially; PACF cuts off sharply after lag
- MA(q): ACF cuts off sharply after lag ; PACF tails off exponentially
- ARMA(p,q): both ACF and PACF tail off gradually — neither cuts off cleanly
Reading ACF and PACF for model selection
Before fitting any model, inspect the ACF and PACF of the stationary (differenced) series. The pattern of significant spikes points directly to the model structure:
| Pattern | Suggests |
|---|---|
| ACF: exponential decay or damped oscillation; PACF: cuts off after lag | AR() |
| ACF: cuts off after lag ; PACF: exponential decay or damped oscillation | MA() |
| Both ACF and PACF tail off gradually | ARMA(,) |
| No significant spikes in either plot | White noise: no model needed |
| Significant spikes at every lag, slow decay | Non-stationary: difference again |
| Spikes at lags | Seasonal structure: use SARIMA |
Autocorrelation Function (ACF)
Bars show autocorrelation at each lag. Dashed lines are the ±1.96/√T significance bounds — bars outside them are statistically significant. AR(1) shows exponential decay; MA(1) cuts off after lag 1; white noise stays within bounds throughout.
That reading is one stage of the Box-Jenkins procedure — ADF test → selection → ACF/PACF reading → model fitting → residual diagnostics → forecast. Walk it end to end below on a series simulated from a known ARIMA(1,1,1), so you can check whether the procedure recovers the model it was generated from. Every statistic is computed live; the candidate grid is clickable, and your choice carries into the diagnostics and the forecast, so fit the wrong model on purpose and watch the residual check catch it.
Box–Jenkins, step by step
You cannot fit this series as it stands
200 days of the running daily-demand series. It wanders: the level at the end has nothing to do with the level at the start, and there is no mean it returns to. ARIMA’s AR and MA parts are defined only for a stationary series, so none of the machinery applies yet.
Fitting anyway is the classic failure. Regress a wandering series on its own lag and you get a coefficient pinned near 1 with a standard error that is not what it claims to be — the textbook spurious regression. The model looks superb in-sample and forecasts nothing.
The autocorrelation at lag 1 is 0.98 and still 0.46 at lag 18 — barely decaying. That is a unit root, not memory.
series
200 daily observations · non-stationary in levels (ACF at lag 1 = 0.98).
The Box–Jenkins procedure applied to a simulated daily-demand series: test for a unit root, difference, read the correlograms, fit a shortlist, check the residuals, then forecast. Every statistic is computed in the browser — the ADF regression, the Durbin–Levinson PACF, a Hannan–Rissanen fit per candidate, Ljung–Box with a real chi-square tail, and ψ-weight prediction intervals. The series was generated from a known ARIMA(1,1,1), so you can check whether the procedure recovers it.
Worked example: ARIMA(1,1,1)
Consider 36 monthly observations with a gentle upward trend. After ADF testing, we determine is sufficient. On the differenced series, the PACF has one significant spike (suggesting AR(1)) and the ACF has one significant spike (suggesting MA(1)), leading to ARIMA(1,1,1).
The fitted model on the differenced series :
Suppose maximum likelihood estimation yields , , .
Computing a 6-step ahead forecast. Let be the last observed time point. The one-step forecast is:
For , future error terms have expectation zero, so the forecasts collapse toward a linear trend. The forecast confidence interval widens with the horizon:
| Horizon | Point forecast | 95% half-width |
|---|---|---|
| 1 | ||
| 2 | ||
| 3 | ||
| 6 |
The growing interval reflects genuine uncertainty: the further ahead we forecast, the less the current data constrains the outcome.
Model selection: AIC and BIC
When multiple candidate models fit the stationary series adequately, choose among them using an information criterion that balances fit against complexity:
where is the maximised log-likelihood, is the number of free parameters (including the constant and ), and is the series length. Lower is better for both.
BIC applies a heavier penalty versus AIC's . For , BIC penalises each extra parameter by versus AIC's — making BIC prefer simpler models, especially in short series.
Auto-ARIMA
Searching over all combinations manually is tedious. The pmdarima library's auto_arima() function automates the Box-Jenkins search: it steps through candidate orders, fits each by MLE, and returns the winner by AIC or BIC. Use it as a strong baseline, but always inspect its choice with ACF/PACF plots to build intuition.
Seasonal ARIMA (SARIMA)
Many real-world series have both a trend component and a seasonal component with known period (e.g. for monthly data with yearly seasonality, for daily data with weekly seasonality).
SARIMA adds seasonal autoregressive and seasonal moving-average terms at multiples of the seasonal lag :
where is the backshift operator (), and are the seasonal AR and MA polynomials of orders and , and is the number of seasonal differences .
SARIMA notation
The upper-case and subscript characterise the seasonal part; the lower-case characterise the non-seasonal part. For the classic airline passengers dataset, the benchmark model is SARIMA — known as the "airline model". A single nonseasonal MA term plus a single seasonal MA term captures most of the structure with just two parameters.
Python implementation
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
from statsmodels.tsa.stattools import adfuller
# ── 1. Simulate an ARIMA(1,1,1) series for demonstration ─────────────────────
np.random.seed(42)
T = 100
eps = np.random.normal(0, 1, T)
# Build ARMA(1,1) on differences, then integrate once
w = np.zeros(T)
for t in range(1, T):
w[t] = 0.45 * w[t-1] + eps[t] - 0.30 * eps[t-1]
# Integrate: y_t = y_{t-1} + w_t (d=1)
y = np.cumsum(w) + 50.0 # start around 50
dates = pd.date_range("2015-01", periods=T, freq="MS")
series = pd.Series(y, index=dates)
# ── 2. Confirm non-stationarity on levels, stationarity on differences ────────
adf_levels = adfuller(y, autolag="AIC")
adf_diff = adfuller(np.diff(y), autolag="AIC")
print(f"ADF levels p = {adf_levels[1]:.4f} (> 0.05 → non-stationary)")
print(f"ADF diff p = {adf_diff[1]:.4f} (< 0.05 → stationary)")
# ── 3. Plot ACF / PACF of differenced series ──────────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
plot_acf( np.diff(y), lags=20, ax=axes[0], title="ACF of Δy_t")
plot_pacf(np.diff(y), lags=20, ax=axes[1], title="PACF of Δy_t")
plt.tight_layout()
plt.show()
# ── 4. Fit ARIMA(1,1,1) ───────────────────────────────────────────────────────
model = ARIMA(series, order=(1, 1, 1))
result = model.fit()
print(result.summary())
# ── 5. Forecast 12 steps ahead with confidence intervals ─────────────────────
forecast = result.get_forecast(steps=12)
fc_mean = forecast.predicted_mean
fc_ci = forecast.conf_int(alpha=0.05)
plt.figure(figsize=(12, 5))
plt.plot(series, label="Observed", color="steelblue")
plt.plot(fc_mean, label="Forecast", color="darkorange", linestyle="--")
plt.fill_between(fc_ci.index, fc_ci.iloc[:, 0], fc_ci.iloc[:, 1],
alpha=0.25, color="darkorange", label="95% CI")
plt.title("ARIMA(1,1,1) — 12-step forecast")
plt.legend()
plt.tight_layout()
plt.show()
# ── 6. AIC/BIC comparison across candidate models ────────────────────────────
candidates = [(1,1,0), (0,1,1), (1,1,1), (2,1,0), (0,1,2)]
rows = []
for order in candidates:
try:
m = ARIMA(series, order=order).fit()
rows.append({"order": str(order), "AIC": round(m.aic, 2), "BIC": round(m.bic, 2)})
except Exception:
pass
print(pd.DataFrame(rows).sort_values("AIC").to_string(index=False))
# ── 7. Seasonal ARIMA via pmdarima (auto_arima) ───────────────────────────────
# pip install pmdarima
from pmdarima import auto_arima
auto_model = auto_arima(
series,
seasonal=True,
m=12, # monthly seasonality
information_criterion="aic",
stepwise=True,
trace=True,
)
print(auto_model.summary())
Exercise
A stationary series has an ACF that cuts off sharply after lag 2 (near-zero for lags 3, 4, 5, …) while the PACF decays gradually. What ARIMA order does this suggest?
Need a hint?
MA processes have finite ACF; AR processes have finite PACF.
Exercise
An ARIMA(1,1,0) model with φ̂ = 0.6 and σ̂ = 5 produces a 1-step ahead forecast of 120. Approximately how wide is the 95% prediction interval at horizon h = 1?
Need a hint?
At h=1 the prediction interval is roughly ±1.96σ.
Key takeaways
- AR(p) models the current value as a weighted sum of the last values; MA(q) as a weighted sum of the last forecast errors. Together they form ARMA(p,q).
- The integrated step (the in ARIMA) applies first-differencing to remove non-stationarity before fitting the ARMA part.
- ACF and PACF plots of the stationary series are the primary tools for choosing and : AR → PACF cuts off; MA → ACF cuts off; ARMA → both tail off.
- AIC/BIC rank competing models; BIC penalises complexity more heavily and favours simpler models in short series.
- SARIMA extends ARIMA with seasonal AR/MA terms at multiples of the seasonal period — essential for series like monthly retail sales or daily temperature.
Related concepts
- Time Series Fundamentals: stationarity, differencing, and the ACF/PACF tools used here
- Deep Learning for Time Series: when ARIMA is not expressive enough
- Linear Regression: AR coefficients are estimated by a variant of OLS