9 The Calculus of Decisions: Policy Gradient Methods
This chapter shows why policy gradient methods are needed when reinforcement learning problems move beyond small, discrete action sets and into continuous, high-dimensional control. Using data center cooling as the running example, it contrasts value-based methods with direct policy optimization, emphasizing that discretizing many continuous controls quickly becomes impossible and that some tasks benefit from stochastic policies rather than deterministic action choice. Policy gradients solve this by learning a parameterized policy directly and updating it to improve expected reward, making them a natural fit for coordinated control problems with complex interactions.
The mathematical core is the policy gradient theorem, which allows the gradient of expected return to be computed without knowing the environment dynamics. The chapter develops REINFORCE as the simplest Monte Carlo policy gradient method: sample trajectories, compute return-to-go, and increase the likelihood of actions that led to better outcomes while decreasing the likelihood of worse ones. It then addresses REINFORCE’s high variance with a learned baseline, turning returns into advantages so the update reflects performance relative to expectation rather than raw reward scale. Entropy regularization is added to keep the policy exploratory early in training and prevent premature collapse to a narrow strategy.
To handle the multi-zone structure of the cooling task, the chapter introduces attention and the Transformer as a way to model relationships among zones. Self-attention lets each zone condition its decision on other relevant zones, while multi-head attention captures different interaction patterns in parallel; positional encoding preserves spatial identity. The resulting Transformer actor-critic learns Gaussian policies for continuous fan and valve controls, uses a critic to estimate state value, and trains with normalized advantages, gradient clipping, and on-policy rollouts. In the case study, this architecture learns a better cooling strategy than a hand-crafted controller, improving return while managing the trade-off between energy use and thermal safety.
Value-Based vs. Policy-Based Reinforcement Learning.
The Policy Gradient Objective.
Mathematical proof of Policy Gradient Theorem.
The REINFORCE algorithm core steps.
Variance Reduction with Baselines.
The Attention Mechanism — Query, Key, Value.
Multi head attention architecture
Transformer architecture.
Cooling optimization problem.
REINFORCE Training Progress — Data Center Cooling
RL Policy vs. Heuristic Controller — Box Plot Comparison
Summary
- Value-based methods like DQN require discrete, enumerable action spaces. When actions are continuous (fan speeds, valve openings, motor torques) or high-dimensional (multiple agents or zones), the action space cannot be discretized without losing precision or creating combinatorial explosion.
- Policy gradient methods optimize the policy directly, representing it as a parameterized probability distribution over actions. This naturally handles continuous and high-dimensional action spaces without discretization.
- The policy gradient theorem provides a way to compute ∇θJ(θ) without knowing environment dynamics. The gradient is: E[Σₜ ∇θ log πθ(aₜ|sₜ) R(τ)]. The log-derivative trick eliminates the unknown transition probabilities from the gradient.
- The score function ∇θ log πθ(a|s) points in the direction that makes action a more likely. The return R(τ) scales this direction: positive returns reinforce actions, negative returns suppress them. This is the core mechanism of REINFORCE.
- REINFORCE collects complete trajectories, computes returns, and updates the policy to make high-return trajectories more probable. It requires no replay buffer, no target network, and no bootstrapping—just sampling and gradient ascent.
- Gaussian policies for continuous control output a mean μ and standard deviation σ per action dimension. Actions are sampled from Normal(μ, σ), and the standard deviation naturally adapts to control exploration without an external schedule.
- Vanilla REINFORCE suffers from high variance because the return R(τ) can have arbitrary scale, causing all actions in a trajectory to be reinforced or suppressed regardless of their individual contribution.
- Subtracting a baseline b(s) from the return reduces variance without introducing bias. The optimal baseline is the state-value function V(s), giving the advantage A = R - V(s), which measures how much better or worse an action was compared to expectation.
- The actor-critic architecture uses two components: an actor (policy network) updated by the advantage-weighted policy gradient, and a critic (value network) trained to predict expected returns. The critic provides the baseline that stabilizes the actor's learning.
- Entropy regularization adds a bonus for maintaining policy stochasticity, preventing premature convergence to deterministic behavior before adequate exploration.
- The attention mechanism computes weighted combinations of inputs using Query-Key-Value projections. Each element queries all others, and the resulting attention weights reflect pairwise relevance. This enables dynamic, context-dependent information routing.
- Multi-head attention runs multiple independent attention computations in parallel, each learning different relationship patterns (thermal coupling, workload correlation, pressure dynamics). Concatenating the heads captures multiple types of inter-zone interactions simultaneously.
- The Transformer encoder stacks layers of multi-head self-attention and feedforward networks with residual connections and layer normalization. Each layer enables longer-range interactions, so two layers can capture indirect zone-to-zone effects.
- Positional encoding adds sinusoidal identity signals to each zone, allowing the Transformer to learn spatial relationships (zone adjacency) alongside state similarity.
- In data center case study, the Transformer-based REINFORCE agent achieved a 53.4% improvement over a hand-tuned PI controller, learning to trade slightly higher energy cost for dramatically fewer temperature violations—a trade-off directly aligned with the reward function's priorities.
- Policy gradient methods are the foundation for modern RL algorithms (PPO, SAC, A3C) and are essential whenever actions are continuous, stochastic policies are desirable, or multi-entity coordination is required. Understanding REINFORCE deeply prepares you for these more advanced methods in subsequent chapters.
Applied Reinforcement Learning ebook for free