ARIMA ModelsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

ARIMA Models

The ARIMA family is the dominant classical approach to univariate time series forecasting. It is principled, interpretable, well-understood statistically, and often surprisingly competitive with more complex models on short, clean series. Understanding ARIMA also builds the vocabulary (stationarity, autoregression, moving averages, differencing) that underpins every modern forecasting framework.

The forecasting thread

This is the second stop for the site's running daily-demand problem from the project-loop hub. The linear-regression baseline treated it as a tabular regression; ARIMA instead models the series as a series — differencing away the trend, an AR term for the weekly echo, an MA term for the shocks. Same problem, a hypothesis space built for temporal structure, and it will still be blindsided by the day-400 regime shift, which is why lesson 04 and streaming ML exist.

ARIMA stands for AutoRegressive Integrated Moving Average. The three letters each correspond to one component of the model, and together they capture the most common structure found in real-world economic, scientific, and engineering time series.

The three components

AR(p): Autoregressive

An AR(p) model expresses the current value as a linear combination of its pp most recent values plus white noise:

yt=c+ϕ1yt−1+ϕ2yt−2+⋯+ϕpyt−p+εty_t = c + \phi_1 y_{t-1} + \phi_2 y_{t-2} + \cdots + \phi_p y_{t-p} + \varepsilon_t

where cc is a constant, ϕ1,…,ϕp\phi_1, \ldots, \phi_p are the autoregressive coefficients, and εt∼N(0,σ2)\varepsilon_t \sim \mathcal{N}(0, \sigma^2) is independent white noise. The model is stable (stationary) when the roots of the characteristic polynomial 1−ϕ1z−⋯−ϕpzp=01 - \phi_1 z - \cdots - \phi_p z^p = 0 all lie outside the unit circle.

Intuition: An AR(1) with ϕ1=0.8\phi_1 = 0.8 says "tomorrow is 80% of today plus some noise." The series has a long memory — shocks decay geometrically but never vanish.

MA(q): Moving Average

An MA(q) model expresses the current value as the mean plus a linear combination of qq past forecast errors:

yt=μ+εt+θ1εt−1+θ2εt−2+⋯+θqεt−qy_t = \mu + \varepsilon_t + \theta_1\varepsilon_{t-1} + \theta_2\varepsilon_{t-2} + \cdots + \theta_q\varepsilon_{t-q}

where μ\mu is the series mean and θ1,…,θq\theta_1, \ldots, \theta_q are the moving-average coefficients. Unlike AR, an MA model has finite memory: a shock εt−k\varepsilon_{t-k} only affects yty_t directly if k≤qk \le q. Beyond lag qq, its influence is exactly zero.

Intuition: An MA(1) with θ1=0.6\theta_1 = 0.6 says "today's value is partly determined by yesterday's forecast error." If we over-predicted yesterday, we correct partly today.

ARIMA(p, d, q)

The integrated part deals with non-stationarity. If the raw series yty_t is non-stationary but becomes stationary after differencing dd times, we model the dd-times-differenced series as an ARMA(p, q):

Δdyt⏟wt=c+∑i=1pϕiwt−i+εt+∑j=1qθjεt−j\underbrace{\Delta^d y_t}_{w_t} = c + \sum_{i=1}^p \phi_i w_{t-i} + \varepsilon_t + \sum_{j=1}^q \theta_j \varepsilon_{t-j}

The model is written ARIMA(p, d, q) where pp is the AR order, dd is the number of differences (usually 0, 1, or 2), and qq is the MA order.

Choosing d: the Augmented Dickey-Fuller test

How many differences do you need? You don't guess — you test. The Augmented Dickey-Fuller (ADF) test checks whether a series has a unit root: a root of the AR characteristic polynomial sitting exactly on the unit circle, which is the precise mathematical signature of a random-walk-style non-stationary series. Its null hypothesis is "the series has a unit root"; a p-value below 0.05 rejects that null and declares the series stationary. The recipe is simple: run the test, and if it fails to reject, difference once more and re-test — the number of differences needed to reach stationarity is exactly your dd.

Deep dive
The Augmented Dickey-Fuller Test
Includes runnable notebook

The memory fingerprint

AR and MA have opposite ACF/PACF signatures — a critical fact for model identification:

  • AR(p): ACF tails off exponentially; PACF cuts off sharply after lag pp
  • MA(q): ACF cuts off sharply after lag qq; PACF tails off exponentially
  • ARMA(p,q): both ACF and PACF tail off gradually — neither cuts off cleanly

Reading ACF and PACF for model selection

Before fitting any model, inspect the ACF and PACF of the stationary (differenced) series. The pattern of significant spikes points directly to the model structure:

PatternSuggests
ACF: exponential decay or damped oscillation; PACF: cuts off after lag ppAR(pp)
ACF: cuts off after lag qq; PACF: exponential decay or damped oscillationMA(qq)
Both ACF and PACF tail off graduallyARMA(pp,qq)
No significant spikes in either plotWhite noise: no model needed
Significant spikes at every lag, slow decayNon-stationary: difference again
Spikes at lags s,2s,3s,…s, 2s, 3s, \ldotsSeasonal structure: use SARIMA

Autocorrelation Function (ACF)

sig. bound±0.179
1357911131516Lag-1.0-0.50.00.51.0significant +significant −

Bars show autocorrelation at each lag. Dashed lines are the ±1.96/√T significance bounds — bars outside them are statistically significant. AR(1) shows exponential decay; MA(1) cuts off after lag 1; white noise stays within bounds throughout.

That reading is one stage of the Box-Jenkins procedure — ADF test → dd selection → ACF/PACF reading → model fitting → residual diagnostics → forecast. Walk it end to end below on a series simulated from a known ARIMA(1,1,1), so you can check whether the procedure recovers the model it was generated from. Every statistic is computed live; the candidate grid is clickable, and your choice carries into the diagnostics and the forecast, so fit the wrong model on purpose and watch the residual check catch it.

Box–Jenkins, step by step

Identification · which model?step 1 of 6
You cannot fit this series as it stands

200 days of the running daily-demand series. It wanders: the level at the end has nothing to do with the level at the start, and there is no mean it returns to. ARIMA’s AR and MA parts are defined only for a stationary series, so none of the machinery applies yet.

Fitting anyway is the classic failure. Regress a wandering series on its own lag and you get a coefficient pinned near 1 with a standard error that is not what it claims to be — the textbook spurious regression. The model looks superb in-sample and forecasts nothing.

The autocorrelation at lag 1 is 0.98 and still 0.46 at lag 18 — barely decaying. That is a unit root, not memory.

The series200 days · levels
050100150199205308411daily demand, units · 200 daysno fixed mean · no fixed variance
What the procedure has established so far
series

200 daily observations · non-stationary in levels (ACF at lag 1 = 0.98).

daily demand

The Box–Jenkins procedure applied to a simulated daily-demand series: test for a unit root, difference, read the correlograms, fit a shortlist, check the residuals, then forecast. Every statistic is computed in the browser — the ADF regression, the Durbin–Levinson PACF, a Hannan–Rissanen fit per candidate, Ljung–Box with a real chi-square tail, and ψ-weight prediction intervals. The series was generated from a known ARIMA(1,1,1), so you can check whether the procedure recovers it.

Deep dive
ARIMA Order Selection: Box-Jenkins Methodology
Includes runnable notebook

Worked example: ARIMA(1,1,1)

Consider 36 monthly observations with a gentle upward trend. After ADF testing, we determine d=1d = 1 is sufficient. On the differenced series, the PACF has one significant spike (suggesting AR(1)) and the ACF has one significant spike (suggesting MA(1)), leading to ARIMA(1,1,1).

The fitted model on the differenced series wt=Δytw_t = \Delta y_t:

wt=c+ϕ^ wt−1+εt+θ^ εt−1w_t = c + \hat{\phi}\, w_{t-1} + \varepsilon_t + \hat{\theta}\, \varepsilon_{t-1}

Suppose maximum likelihood estimation yields ϕ^=0.45\hat{\phi} = 0.45, θ^=−0.30\hat{\theta} = -0.30, σ^=2.1\hat{\sigma} = 2.1.

Computing a 6-step ahead forecast. Let TT be the last observed time point. The one-step forecast is:

y^T+1∣T=yT+c+ϕ^ wT+θ^ εT\hat{y}_{T+1|T} = y_T + c + \hat{\phi}\, w_T + \hat{\theta}\, \varepsilon_T

For h>1h > 1, future error terms εT+j\varepsilon_{T+j} have expectation zero, so the forecasts collapse toward a linear trend. The forecast confidence interval widens with the horizon:

95% PI at horizon h:y^T+h∣T±1.96 σ^1+(h−1)ϕ^2\text{95\% PI at horizon } h: \quad \hat{y}_{T+h|T} \pm 1.96\,\hat{\sigma}\sqrt{1 + (h-1)\hat{\phi}^2}
Horizon hhPoint forecast95% half-width
1y^T+1\hat{y}_{T+1}1.96×2.1=4.121.96 \times 2.1 = 4.12
2y^T+2\hat{y}_{T+2}1.96×2.1×1+0.20=4.521.96 \times 2.1 \times \sqrt{1 + 0.20} = 4.52
3y^T+3\hat{y}_{T+3}1.96×2.1×1+0.41=4.901.96 \times 2.1 \times \sqrt{1 + 0.41} = 4.90
6y^T+6\hat{y}_{T+6}1.96×2.1×1+1.01=5.841.96 \times 2.1 \times \sqrt{1 + 1.01} = 5.84

The growing interval reflects genuine uncertainty: the further ahead we forecast, the less the current data constrains the outcome.

Model selection: AIC and BIC

When multiple candidate models fit the stationary series adequately, choose among them using an information criterion that balances fit against complexity:

AIC=−2ln⁡(L^)+2k\text{AIC} = -2\ln(\hat{\mathcal{L}}) + 2k BIC=−2ln⁡(L^)+kln⁡(T)\text{BIC} = -2\ln(\hat{\mathcal{L}}) + k\ln(T)

where L^\hat{\mathcal{L}} is the maximised log-likelihood, kk is the number of free parameters (including the constant and σ^2\hat{\sigma}^2), and TT is the series length. Lower is better for both.

BIC applies a heavier penalty kln⁡Tk \ln T versus AIC's 2k2k. For T=100T = 100, BIC penalises each extra parameter by ln⁡(100)≈4.6\ln(100) \approx 4.6 versus AIC's 2.02.0 — making BIC prefer simpler models, especially in short series.

Auto-ARIMA

Searching over all (p,d,q)(p, d, q) combinations manually is tedious. The pmdarima library's auto_arima() function automates the Box-Jenkins search: it steps through candidate orders, fits each by MLE, and returns the winner by AIC or BIC. Use it as a strong baseline, but always inspect its choice with ACF/PACF plots to build intuition.

Seasonal ARIMA (SARIMA)

Many real-world series have both a trend component and a seasonal component with known period mm (e.g. m=12m = 12 for monthly data with yearly seasonality, m=7m = 7 for daily data with weekly seasonality).

SARIMA(p,d,q)(P,D,Q)m(p, d, q)(P, D, Q)_m adds seasonal autoregressive and seasonal moving-average terms at multiples of the seasonal lag mm:

ΦP(Bm) ϕp(B) ΔmD Δd yt=c+ΘQ(Bm) θq(B) εt\Phi_P(B^m)\,\phi_p(B)\,\Delta^D_m\,\Delta^d\,y_t = c + \Theta_Q(B^m)\,\theta_q(B)\,\varepsilon_t

where BB is the backshift operator (Byt=yt−1B y_t = y_{t-1}), ΦP\Phi_P and ΘQ\Theta_Q are the seasonal AR and MA polynomials of orders PP and QQ, and DD is the number of seasonal differences Δmyt=yt−yt−m\Delta_m y_t = y_t - y_{t-m}.

SARIMA notation

The upper-case (P,D,Q)(P, D, Q) and subscript mm characterise the seasonal part; the lower-case (p,d,q)(p, d, q) characterise the non-seasonal part. For the classic airline passengers dataset, the benchmark model is SARIMA(0,1,1)(0,1,1)12(0,1,1)(0,1,1)_{12} — known as the "airline model". A single nonseasonal MA term plus a single seasonal MA term captures most of the structure with just two parameters.

Python implementation

Python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.graphics.tsaplots import plot_acf, plot_pacf
from statsmodels.tsa.stattools import adfuller

# ── 1. Simulate an ARIMA(1,1,1) series for demonstration ─────────────────────
np.random.seed(42)
T = 100
eps = np.random.normal(0, 1, T)

# Build ARMA(1,1) on differences, then integrate once
w = np.zeros(T)
for t in range(1, T):
    w[t] = 0.45 * w[t-1] + eps[t] - 0.30 * eps[t-1]

# Integrate: y_t = y_{t-1} + w_t  (d=1)
y = np.cumsum(w) + 50.0       # start around 50

dates = pd.date_range("2015-01", periods=T, freq="MS")
series = pd.Series(y, index=dates)

# ── 2. Confirm non-stationarity on levels, stationarity on differences ────────
adf_levels = adfuller(y, autolag="AIC")
adf_diff   = adfuller(np.diff(y), autolag="AIC")
print(f"ADF levels  p = {adf_levels[1]:.4f}  (> 0.05 → non-stationary)")
print(f"ADF diff    p = {adf_diff[1]:.4f}   (< 0.05 → stationary)")

# ── 3. Plot ACF / PACF of differenced series ──────────────────────────────────
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
plot_acf( np.diff(y), lags=20, ax=axes[0], title="ACF of Δy_t")
plot_pacf(np.diff(y), lags=20, ax=axes[1], title="PACF of Δy_t")
plt.tight_layout()
plt.show()

# ── 4. Fit ARIMA(1,1,1) ───────────────────────────────────────────────────────
model  = ARIMA(series, order=(1, 1, 1))
result = model.fit()
print(result.summary())

# ── 5. Forecast 12 steps ahead with confidence intervals ─────────────────────
forecast = result.get_forecast(steps=12)
fc_mean  = forecast.predicted_mean
fc_ci    = forecast.conf_int(alpha=0.05)

plt.figure(figsize=(12, 5))
plt.plot(series,         label="Observed",         color="steelblue")
plt.plot(fc_mean,        label="Forecast",          color="darkorange", linestyle="--")
plt.fill_between(fc_ci.index, fc_ci.iloc[:, 0], fc_ci.iloc[:, 1],
                 alpha=0.25, color="darkorange", label="95% CI")
plt.title("ARIMA(1,1,1) — 12-step forecast")
plt.legend()
plt.tight_layout()
plt.show()

# ── 6. AIC/BIC comparison across candidate models ────────────────────────────
candidates = [(1,1,0), (0,1,1), (1,1,1), (2,1,0), (0,1,2)]
rows = []
for order in candidates:
    try:
        m = ARIMA(series, order=order).fit()
        rows.append({"order": str(order), "AIC": round(m.aic, 2), "BIC": round(m.bic, 2)})
    except Exception:
        pass

print(pd.DataFrame(rows).sort_values("AIC").to_string(index=False))

# ── 7. Seasonal ARIMA via pmdarima (auto_arima) ───────────────────────────────
# pip install pmdarima
from pmdarima import auto_arima

auto_model = auto_arima(
    series,
    seasonal=True,
    m=12,                  # monthly seasonality
    information_criterion="aic",
    stepwise=True,
    trace=True,
)
print(auto_model.summary())

Exercise

A stationary series has an ACF that cuts off sharply after lag 2 (near-zero for lags 3, 4, 5, …) while the PACF decays gradually. What ARIMA order does this suggest?

Need a hint?

MA processes have finite ACF; AR processes have finite PACF.

Exercise

An ARIMA(1,1,0) model with φ̂ = 0.6 and σ̂ = 5 produces a 1-step ahead forecast of 120. Approximately how wide is the 95% prediction interval at horizon h = 1?

Need a hint?

At h=1 the prediction interval is roughly ±1.96σ.

Key takeaways

  • AR(p) models the current value as a weighted sum of the last pp values; MA(q) as a weighted sum of the last qq forecast errors. Together they form ARMA(p,q).
  • The integrated step (the dd in ARIMA) applies first-differencing to remove non-stationarity before fitting the ARMA part.
  • ACF and PACF plots of the stationary series are the primary tools for choosing pp and qq: AR → PACF cuts off; MA → ACF cuts off; ARMA → both tail off.
  • AIC/BIC rank competing models; BIC penalises complexity more heavily and favours simpler models in short series.
  • SARIMA extends ARIMA with seasonal AR/MA terms at multiples of the seasonal period mm — essential for series like monthly retail sales or daily temperature.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
ARIMA ModelsRecurrent Neural NetworksLinear & Logistic Regress…Recurrent Neural NetworksTime Series FundamentalsLinear RegressionDeep Learning for Time Se…Demand Forecasting in Pro…The ML Project LoopThe Augmented Dickey-Full…ARIMA Order Selection: Bo…ACF & PACF: Reading the C…