Multi-Agent Reinforcement LearningOpen in Colab

Multi-Agent Reinforcement Learning (MARL)

Advanced material

A specialized extension of single-agent RL. It assumes familiarity with policy gradients, actor-critic methods, and the basics of game theory. Read Policy Gradient & Actor-Critic first.

Single-agent RL assumes one agent in a stationary environment. Multi-Agent RL studies settings where multiple agents learn simultaneously — autonomous vehicles negotiating an intersection, bots in a strategy game, market-making agents, or robot swarms. The presence of other learning agents breaks the core assumptions of single-agent RL.

The Markov game (stochastic game)

MARL generalizes the MDP to a Markov game with NN agents:

⟨S, {Ai}i=1N, T, {Ri}i=1N, γ⟩\langle S,\, \{A_i\}_{i=1}^N,\, T,\, \{R_i\}_{i=1}^N,\, \gamma \rangle
  • Each agent ii has its own action set AiA_i and reward function RiR_i
  • The transition T(s′∣s,a1,…,aN)T(s' \mid s, a_1, \ldots, a_N) depends on the joint action of all agents
  • Each agent learns a policy πi(ai∣s)\pi_i(a_i \mid s) to maximize its own expected return

Three regimes

RegimeReward structureExample
CooperativeAll agents share one reward R1=⋯=RNR_1 = \cdots = R_NRobot team assembling a product
CompetitiveZero-sum: ∑iRi=0\sum_i R_i = 0Two-player Go, poker
MixedGeneral-sum, partial alignmentAutonomous driving (avoid collisions but reach own goal)

The non-stationarity problem

This is the central challenge of MARL. From any single agent's perspective, the environment includes the other agents' policies. As those agents learn and change their policies, the effective transition and reward dynamics change over time — the environment is non-stationary.

This breaks the Markov assumption that single-agent RL relies on:

  • Q-learning convergence guarantees no longer hold (the optimal Q-function is a moving target)
  • Experience replay becomes problematic — old transitions were generated by outdated opponent policies, so replaying them teaches the agent to respond to behavior that no longer occurs

Solution concepts: beyond "optimal"

In single-agent RL the goal is the optimal policy. With multiple agents, "optimal" is ill-defined — your best policy depends on what others do. MARL borrows equilibrium concepts from game theory:

  • Nash Equilibrium: a joint policy where no agent can improve by unilaterally changing its own policy
  • Correlated Equilibrium: agents coordinate through a shared signal
  • Minimax / maximin: for zero-sum games, play the policy that maximizes worst-case return

Centralized Training, Decentralized Execution (CTDE)

The dominant modern paradigm. During training, a central critic has access to the global state and all agents' actions; during execution, each agent acts using only its own local observations.

This resolves non-stationarity during training (the centralized critic sees everyone's actions, so the environment looks stationary to it) while keeping execution decentralized (practical for deployment).

AlgorithmIdea
MADDPGMulti-agent DDPG: each agent has a decentralized actor and a centralized critic that takes all agents' observations and actions
QMIXCooperative value factorization: a mixing network combines per-agent Q-values into a joint Q-value, constrained to be monotonic so decentralized argmax matches centralized argmax
MAPPOMulti-agent PPO with a centralized value function; surprisingly strong on cooperative benchmarks
Self-playFor competitive games, the agent trains against copies of itself (AlphaGo, AlphaStar, OpenAI Five)

Emergent phenomena

MARL produces behaviors that don't arise in single-agent settings:

  • Communication: agents can learn to emit signals that coordinate behavior, even when communication isn't explicitly rewarded
  • Specialization / role assignment: in cooperative teams, agents differentiate into complementary roles
  • Arms races: in competitive settings, agents iteratively discover and counter each other's strategies (OpenAI's hide-and-seek showed tool use emerging from competitive pressure)

Credit assignment

In cooperative MARL with a shared reward, a hard question is: which agent's action caused the team reward? Counterfactual multi-agent (COMA) policy gradients address this by computing a counterfactual baseline — comparing the actual reward to what would have happened if a single agent had acted differently, holding others fixed.

When to use MARL

  • The problem genuinely involves multiple decision-makers with their own objectives
  • Centralized control is infeasible (communication limits, scale, privacy)
  • You need robustness to adaptive opponents (self-play produces strong, general strategies)

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Referenced by
Multi-Agent Reinforcement LearningPolicy Gradient & Actor-C…Markov Decision Processes