Overview

9 The Calculus of Decisions: Policy Gradient Methods

This chapter shows why policy gradient methods are needed when reinforcement learning problems move beyond small, discrete action sets and into continuous, high-dimensional control. Using data center cooling as the running example, it contrasts value-based methods with direct policy optimization, emphasizing that discretizing many continuous controls quickly becomes impossible and that some tasks benefit from stochastic policies rather than deterministic action choice. Policy gradients solve this by learning a parameterized policy directly and updating it to improve expected reward, making them a natural fit for coordinated control problems with complex interactions.

The mathematical core is the policy gradient theorem, which allows the gradient of expected return to be computed without knowing the environment dynamics. The chapter develops REINFORCE as the simplest Monte Carlo policy gradient method: sample trajectories, compute return-to-go, and increase the likelihood of actions that led to better outcomes while decreasing the likelihood of worse ones. It then addresses REINFORCE’s high variance with a learned baseline, turning returns into advantages so the update reflects performance relative to expectation rather than raw reward scale. Entropy regularization is added to keep the policy exploratory early in training and prevent premature collapse to a narrow strategy.

To handle the multi-zone structure of the cooling task, the chapter introduces attention and the Transformer as a way to model relationships among zones. Self-attention lets each zone condition its decision on other relevant zones, while multi-head attention captures different interaction patterns in parallel; positional encoding preserves spatial identity. The resulting Transformer actor-critic learns Gaussian policies for continuous fan and valve controls, uses a critic to estimate state value, and trains with normalized advantages, gradient clipping, and on-policy rollouts. In the case study, this architecture learns a better cooling strategy than a hand-crafted controller, improving return while managing the trade-off between energy use and thermal safety.

Value-Based vs. Policy-Based Reinforcement Learning.
Value-Based vs. Policy-Based Reinforcement Learning.
The Policy Gradient Objective.
The Policy Gradient Objective.
Mathematical proof of Policy Gradient Theorem.
Mathematical proof of Policy Gradient Theorem.
The REINFORCE algorithm core steps.
The REINFORCE algorithm core steps.
Variance Reduction with Baselines.
Variance Reduction with Baselines.
The Attention Mechanism — Query, Key, Value.
The Attention Mechanism — Query, Key, Value.
Multi head attention architecture
Multi head attention architecture
Transformer architecture.
Transformer architecture.
Cooling optimization problem.
Cooling optimization problem.
REINFORCE Training Progress — Data Center Cooling
REINFORCE Training Progress — Data Center Cooling
RL Policy vs. Heuristic Controller — Box Plot Comparison
RL Policy vs. Heuristic Controller — Box Plot Comparison

Summary

  • Value-based methods like DQN require discrete, enumerable action spaces. When actions are continuous (fan speeds, valve openings, motor torques) or high-dimensional (multiple agents or zones), the action space cannot be discretized without losing precision or creating combinatorial explosion.
  • Policy gradient methods optimize the policy directly, representing it as a parameterized probability distribution over actions. This naturally handles continuous and high-dimensional action spaces without discretization.
  • The policy gradient theorem provides a way to compute ∇θJ(θ) without knowing environment dynamics. The gradient is: E[Σₜ ∇θ log πθ(aₜ|sₜ) R(τ)]. The log-derivative trick eliminates the unknown transition probabilities from the gradient.
  • The score function ∇θ log πθ(a|s) points in the direction that makes action a more likely. The return R(τ) scales this direction: positive returns reinforce actions, negative returns suppress them. This is the core mechanism of REINFORCE.
  • REINFORCE collects complete trajectories, computes returns, and updates the policy to make high-return trajectories more probable. It requires no replay buffer, no target network, and no bootstrapping—just sampling and gradient ascent.
  • Gaussian policies for continuous control output a mean μ and standard deviation σ per action dimension. Actions are sampled from Normal(μ, σ), and the standard deviation naturally adapts to control exploration without an external schedule.
  • Vanilla REINFORCE suffers from high variance because the return R(τ) can have arbitrary scale, causing all actions in a trajectory to be reinforced or suppressed regardless of their individual contribution.
  • Subtracting a baseline b(s) from the return reduces variance without introducing bias. The optimal baseline is the state-value function V(s), giving the advantage A = R - V(s), which measures how much better or worse an action was compared to expectation.
  • The actor-critic architecture uses two components: an actor (policy network) updated by the advantage-weighted policy gradient, and a critic (value network) trained to predict expected returns. The critic provides the baseline that stabilizes the actor's learning.
  • Entropy regularization adds a bonus for maintaining policy stochasticity, preventing premature convergence to deterministic behavior before adequate exploration.
  • The attention mechanism computes weighted combinations of inputs using Query-Key-Value projections. Each element queries all others, and the resulting attention weights reflect pairwise relevance. This enables dynamic, context-dependent information routing.
  • Multi-head attention runs multiple independent attention computations in parallel, each learning different relationship patterns (thermal coupling, workload correlation, pressure dynamics). Concatenating the heads captures multiple types of inter-zone interactions simultaneously.
  • The Transformer encoder stacks layers of multi-head self-attention and feedforward networks with residual connections and layer normalization. Each layer enables longer-range interactions, so two layers can capture indirect zone-to-zone effects.
  • Positional encoding adds sinusoidal identity signals to each zone, allowing the Transformer to learn spatial relationships (zone adjacency) alongside state similarity.
  • In data center case study, the Transformer-based REINFORCE agent achieved a 53.4% improvement over a hand-tuned PI controller, learning to trade slightly higher energy cost for dramatically fewer temperature violations—a trade-off directly aligned with the reward function's priorities.
  • Policy gradient methods are the foundation for modern RL algorithms (PPO, SAC, A3C) and are essential whenever actions are continuous, stochastic policies are desirable, or multi-entity coordination is required. Understanding REINFORCE deeply prepares you for these more advanced methods in subsequent chapters.

FAQ

What is policy gradient methods in reinforcement learning?Policy gradient methods are reinforcement learning algorithms that learn a parameterized policy directly and optimize it with gradient ascent to maximize expected cumulative reward. Instead of first estimating action values and then deriving a policy, they update the policy itself end-to-end.
Why are policy gradient methods preferred over value-based methods for continuous action spaces?Value-based methods like DQN require discrete actions, so continuous controls must be heavily discretized, which leads to a combinatorial explosion. Policy gradients can output continuous actions directly, making them a better fit for problems like fan speed or valve opening control.
What is the main idea behind the policy gradient theorem?The policy gradient theorem shows that the gradient of expected return can be written in terms of the policy’s log-probability gradient, without needing to differentiate through the environment dynamics. This allows learning from sampled trajectories even when the transition model is unknown.
What does the REINFORCE algorithm do?REINFORCE is a Monte Carlo policy gradient algorithm that samples complete trajectories, computes discounted return-to-go for each time step, and updates the policy to increase the probability of actions that led to higher returns and decrease the probability of actions that led to lower returns.
Why does REINFORCE use return-to-go G instead of the full trajectory return?Return-to-go only credits an action for rewards that happen after that action is taken. Using the full trajectory return would incorrectly assign earlier rewards to later actions, adding noise and increasing gradient variance.
What is a baseline in policy gradient methods, and why is it useful?A baseline is a state-dependent value subtracted from the return, usually V(s). It does not change the expected gradient, but it reduces variance by centering the learning signal around zero, making good actions positive and bad actions negative relative to expectation.
What is advantage, and how is it used in actor-critic methods?Advantage is defined as A(s, a) = G - V(s). It measures how much better or worse an action performed compared to what the critic expected. The actor updates its policy using advantage-weighted gradients, while the critic learns to predict state values from observed returns.
Why is entropy regularization added to policy gradient training?Entropy regularization encourages the policy to stay exploratory and avoid collapsing too early into a near-deterministic strategy. It helps prevent premature convergence and keeps the agent sampling a useful range of actions during training.
How does attention help in the data center cooling problem?Attention lets each cooling zone focus on other zones that are most relevant to its decision, such as thermally coupled neighbors or zones with similar workload patterns. This enables relational reasoning and coordinated control across the whole data center.
Why use a Transformer-based policy network in this chapter’s case study?A Transformer can model interactions among multiple zones through self-attention, while positional encoding preserves zone identity and spatial structure. This makes it well suited for learning coordinated cooling policies in a high-dimensional, coupled environment.

pro $24.99 per month

  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose one free eBook per month to keep
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime

lite $19.99 per month

  • access to all Manning books, including MEAPs!

team

5, 10 or 20 seats+ for your team - learn more


Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Applied Reinforcement Learning Applied Reinforcement Learning ebook for free
Close
choose your plan

team

monthly
annual
$49.99
$499.99
only $41.67 per month
  • five seats for your team
  • access to all Manning books, MEAPs, liveVideos, liveProjects, and audiobooks!
  • choose another free product every time you renew
  • choose twelve free products per year
  • exclusive 50% discount on all purchases
  • renews monthly, pause or cancel renewal anytime
  • renews annually, pause or cancel renewal anytime
  • Applied Reinforcement Learning Applied Reinforcement Learning ebook for free