10 Reinforcement learning
Reinforcement learning is presented as a psychology-inspired branch of machine learning in which an agent learns by interacting with an environment and optimizing cumulative reward over time. Unlike supervised and unsupervised learning, RL focuses on sequences of actions and delayed consequences, balancing immediate gains with long-term outcomes through exploration and exploitation. The chapter motivates RL with intuitive, real-world analogies, then frames it formally via the Markov Decision Process, clarifying how goals, states, actions, and rewards define problems where the objective is known but the best strategy must be discovered by trial and error.
To make RL concrete, the chapter builds a grid-based parking-lot simulator and walks through the full training lifecycle. It introduces Q-learning as a model-free method that stores action values in a Q-table indexed by state and action, updated with a Bellman-style rule using learning rate and discount factor hyperparameters while episodes propagate value back from goals. Practical choices—like how to represent state (e.g., adjacent-cell encoding to improve transfer), how to schedule random action selection, and how to measure progress (penalties avoided, average reward per action)—are emphasized alongside Q-table limitations, notably poor generalization to unseen states and bias toward short-horizon context when state is narrowly defined.
The chapter then transitions to deep reinforcement learning, where neural networks approximate Q-values and replace tabular models to generalize across large or continuous state spaces. It compares input encodings (compact scalar vs. one-hot), suggests sensible network sizes and ReLU-family activations, and explains target computation, loss, and backpropagation within the RL loop. It also contrasts model-free and model-based approaches and highlights representative use cases—robotics, recommendation systems, financial trading, and game playing—where scalable simulation and well-shaped reward functions enable agents to learn robust, long-term strategies.
How reinforcement learning fits into machine learning
Categorization of machine learning, deep learning, and reinforcement learning
Example of reinforcement learning: teaching a dog to sit by using food as a reward
An example of possible actions that have long-term consequences
The self-driving car in a parking lot problem
The Markov Decision Process for reinforcement learning
Agent actions in the parking-lot environment
Rewards due to specific events in the environment due to actions performed
A bad solution to the parking-lot problem
A good solution to the parking-lot problem
An example Q-table and states that it represents
A better example of a Q-table and states that it represents
Life cycle of a Q-learning reinforcement learning algorithm
An example initialized Q-table
Example Q-table update calculation for state 1
Example Q-table update calculation for state 2
Example Q-table update calculation for state 1 after several iterations
Referencing a Q-table to determine what action to take
Examples of model-based and model-free reinforcement learning
The difference between using a Q-table and ANN for the parking-lot problem
Example 8 input ANN for the parking-lot problem
Example 32 input ANN for the parking-lot problem
Summary of reinforcement learning
FAQ
What is reinforcement learning?
Reinforcement learning (RL) is a trial-and-error learning approach where an agent interacts with an environment, takes actions, and receives rewards or penalties. The agent’s objective is to learn a policy that maximizes cumulative (long-term) reward. It’s inspired by behavioral psychology: desired behaviors are reinforced with positive outcomes, while undesired behaviors are discouraged with negative outcomes.How does RL differ from supervised and unsupervised learning, and when should I use it?
- Supervised learning learns from labeled examples (inputs with correct outputs).- Unsupervised learning discovers structure in unlabeled data (e.g., clusters).
- Reinforcement learning learns from feedback signals (rewards/penalties) that result from actions taken in an environment.
Use RL when you know the goal and the allowed actions but not the optimal sequence of actions to reach that goal—especially in problems that require sequential decision-making and long-term planning.
What are rewards, penalties, and the discount factor in RL?
- Rewards/penalties: Scalar feedback after each action. Positive values encourage behaviors; negative values discourage them.- Cumulative reward: The sum of rewards over time is what the agent aims to maximize.
- Discount factor (gamma): Weighs future rewards relative to immediate ones. A higher discount values long-term outcomes, encouraging the agent to choose actions that may pay off later rather than only chasing instant gratification.
What types of problems are well-suited to reinforcement learning?
Problems where individual actions compound toward a larger goal and where the best strategy unfolds over sequences of decisions, for example:- Strategic planning and games (short-term sacrifices for long-term wins)
- Robotics and control (continuous decisions in dynamic environments)
- Process automation and navigation (chains of actions with delayed consequences)
What is the Markov Decision Process (MDP) and the RL training loop?
An MDP formalizes sequential decision-making with states, actions, rewards, transitions, and terminal conditions. A typical RL loop per episode is:1) Initialize/reset the environment.
2) Observe the current state.
3) Choose an action (via exploration or policy).
4) Apply the action, receive reward, observe next state.
5) Update the policy/value estimates.
6) Repeat until the goal is reached or the episode ends.
How do I design the environment and state/action/reward for RL?
- Simulator: Encodes the environment’s dynamics and enforces rules (state updates, boundaries, collisions).- Actions: All choices the agent can take at each step (e.g., move north/south/east/west).
- State: A compact representation of the situation (e.g., the agent’s surroundings). Careful state design affects generalization and sample efficiency.
- Reward function: Encourages progress toward the goal and discourages harmful outcomes. Crafting rewards is critical to avoid unintended behaviors.
What is Q-learning and what is a Q-table?
Q-learning is a model-free RL algorithm that learns the expected long-term return (Q-value) of taking an action in a given state. It stores these values in a Q-table indexed by (state, action). During training, the agent updates Q-values from experience and, at decision time, selects the action with the highest Q-value for the current state. Good state representations (e.g., focusing on relevant nearby features) help the Q-table transfer better across variations of the environment.How are Q-values updated, and what do the learning rate and discount factor do?
After taking an action and observing the reward and next state, the Q-value for the prior (state, action) is nudged toward “reward + discounted best future value.”- Learning rate (alpha): How strongly to incorporate the new estimate (higher = faster but noisier updates).
- Discount (gamma): How much to value future rewards when computing the target. Higher gamma promotes long-term planning.
How does the agent balance exploration vs. exploitation, and what are episodes?
- Exploration vs. exploitation: The agent occasionally takes random actions (exploration) to discover better strategies, and otherwise follows the current best-known action (exploitation). A common strategy is epsilon-greedy: with probability epsilon take a random action; otherwise pick the best-known action.- Episodes: One complete run from a start state to a terminal condition (goal reached, failure, or timeout). Many episodes are needed so value estimates can propagate backward from outcomes to earlier decisions.
How do I evaluate RL performance, and what are model-free, model-based, and deep RL?
- Evaluation: Use task-specific metrics and general ones like average reward per action or number of penalties per episode. Track progress over many episodes.- Model-free vs. model-based: Model-free (e.g., Q-learning) learns from trial-and-error without an internal world model. Model-based tries to learn or use a model of the environment to plan ahead.
- Deep RL: Replaces the Q-table with a neural network that predicts Q-values or actions from states, enabling generalization to unseen situations. Input encoding matters (e.g., scalar vs. one-hot features of nearby cells), and training uses backpropagation with targets computed from rewards and discounted future estimates.
Grokking AI Algorithms, Second Edition ebook for free
