8 Deep Q-Networks for High-dimensional Data

The chapter explains how reinforcement learning moves beyond tabular methods when the state space is too large, continuous, or high-dimensional to store values explicitly. Using a radiotherapy treatment planning problem as a motivating example, it shows why agents need function approximation: instead of memorizing one value per state-action pair, they learn a parameterized model that generalizes across many similar situations. Linear approximation is introduced as the simplest step in this direction, with temporal-difference learning providing the update rule that turns experience into improved value estimates.

It then builds the deep learning foundation needed for Deep Q-Networks, describing neural networks as layered function approximators that learn useful representations directly from raw inputs. The text emphasizes the forward pass, nonlinear activations such as ReLU, loss computation, and backpropagation as the mechanism for assigning credit and blame across many parameters. This sets up the central idea of DQN: combine Q-learning with a neural network so the agent can estimate Q-values in complex environments, while still learning through prediction error and gradient-based updates.

The chapter’s applied section translates DQN into a treatment-optimization system with a realistic environment, compact state representation, discrete action choices, stochastic dose delivery, tumor dynamics, and a carefully shaped reward function that balances tumor control against organ safety. It highlights the two major stabilizers of DQN, experience replay and a separate target network, and extends the method with Double DQN to reduce overestimation bias. A full training loop with curriculum learning demonstrates how the agent gradually improves as task difficulty increases, and the chapter closes by showing that DQN is broadly useful for sequential decision problems with delayed consequences, while also noting its limits in continuous-action or highly sample-expensive settings.

Two level of abstractions in modern reinforcement learning.

Learning as parameter adjustment: fitting a linear model by turning the θ knobs

Linear function approximation with temporal-difference learning

End-to-end learning in deep neural networks

The building block of deep networks: neurons and activation functions

Deep Q-Networks learning systems

Radiotherapy treatment planning as a sequential decision-making problem

Learning dynamics and outcomes of DQN-based treatment optimization

• Tabular RL cannot scale; function approximation replaces lookup tables with parameterized models that generalize across states
• Function approximation replaces tabular Q-tables with parameterized models Q̂(s,a;θ) that generalize across states. Use linear models when you can engineer interpretable features; switch to deep networks for raw inputs like images or sensor data.
• Neural networks stack layers of linear transformations (Wx + b) and nonlinear activations (ReLU, tanh) to learn hierarchical features. Stack 2-4 hidden layers for most RL problems; add more only for high-dimensional inputs like images (use CNNs) or sequences (use LSTMs).
• Backpropagation computes gradients via chain rule ∂L/∂w = (∂L/∂y)(∂y/∂w), flowing derivatives backward through layers to update parameters.
• Naive deep Q-learning diverges from moving targets (bootstrap values depend on changing Q-values) and correlated sequential data. Experience replay and target networks are essential stabilization techniques, not optional enhancements.
• Experience replay stores transitions (s, a, r, s') in buffers and samples random mini-batches, breaking temporal correlations. Set buffer size to 10K-100K for simple problems, 1M+ for complex domains; start training after accumulating at least 10× batch_size transitions.
• Target networks freeze parameters θ⁻, updated every C steps (typically 1K-10K), stabilizing TD targets. Increase C if training is unstable (loss oscillates); decrease C if learning stalls (Q-values change slowly).
• Double DQN decouples action selection from evaluation: Q(s', argmax_a Q(s',a;θ); θ⁻), reducing overestimation bias. Implement by selecting actions with online network but evaluating Q-values with target network.
• ε-greedy exploration balances random (ε) and greedy (1-ε) actions, decaying ε from 1.0 to 0.01-0.1 during training. Decay over first 50-80% of training; use ε=0.05 for final deployment to maintain minimal exploration.
• State representations must be Markovian (contain all predictive information), compact (minimal redundancy), and informative. Test by checking if adding previous timesteps improves performance; if yes, current state isn't Markovian enough.
• Reward shaping provides dense intermediate feedback at each step instead of sparse terminal rewards, accelerating learning 5-10x. Add progress rewards (distance to goal) and constraint penalties (safety violations) but monitor for reward hacking.
• Curriculum learning starts with simplified environments and gradually increases difficulty as agent improves. Increase difficulty (noise, obstacles, episode length) when success rate exceeds 70% for 100+ episodes.
• DQN handles discrete actions (10-10K) efficiently by outputting one Q-value per action and selecting argmax. For continuous actions or spaces exceeding 100K options, use actor-critic methods (PPO, SAC, DDPG) instead.
• DQN requires millions of environment interactions, suitable only when simulations exist or interactions are cheap. For expensive real-world interactions (robotics, clinical trials), use offline RL, model-based RL, or imitation learning.
• Agents optimize the reward function you specify, not your intentions. Validate reward design by checking if optimal reward-seeking behavior matches desired task behavior; redesign if agent finds unintended shortcuts.
• Critical hyperparameters: learning rate (1e-4 to 1e-3), batch size (32-128), γ (0.9-0.99), buffer size (10K-1M), target update frequency (1K-10K). Start with published values from similar domains; tune learning rate first, then other parameters.
• Debug on toy environments (CartPole, FrozenLake) before scaling to target problems. Verify Q-values increase, success rate improves, and exploration decays; if it fails on simple tasks, fix implementation before adding complexity.
• Neural network policies are black boxes with limited interpretability. For regulated or safety-critical domains, add interpretability techniques (attention mechanisms, saliency maps, policy distillation) or use inherently interpretable methods.
• DQN excels at discrete action problems with high-dimensional states and sequential dependencies. Applications include supply chain optimization, energy grid control, dynamic pricing, and network routing.
• DQN fails for continuous actions, competitive multi-agent settings, or million-scale action spaces. Use PPO/SAC for continuous control, MADDPG/QMIX for multi-agent coordination, hierarchical methods for combinatorial actions.
• Rainbow DQN combines six improvements (prioritized replay, dueling networks, noisy nets, distributional RL, multi-step returns, Double DQN). Use when vanilla DQN plateaus; expect 2-5x better final performance on complex domains.

FAQ