Learning path
Next moduleModule 27: Training at scale
Module 26
Reinforcement learning
Learn from rewards instead of labels: MDPs and the Bellman equation, Q-learning and SARSA, deep Q-networks, policy gradients, and PPO, the algorithm behind RLHF.
- 1. MDPs, states, actions, and rewardsFrame a sequential decision problem as a Markov decision process, compute discounted returns and state values with the Bellman equation, and spot broken formulations before training.40 min
- 2. Q-learning and SARSALearn action values from experience with one-step temporal-difference updates, compute Q-learning and SARSA targets by hand, and explain why the two take different paths along a cliff.45 min
- 3. Deep Q-networks (DQN)Replace the Q-table with a neural network, and use experience replay, a target network, epsilon decay, and the Huber loss to keep bootstrapped training stable.45 min
- 4. Policy gradients and REINFORCEDerive the policy gradient with the log-derivative trick, implement REINFORCE with returns-to-go and a baseline, and recognize the same estimator inside LLM methods such as GRPO.45 min
- 5. Proximal policy optimization (PPO)Compute GAE advantages and PPO's clipped surrogate by hand, implement an actor-critic PPO loop that logs KL and clip fraction, and explain how PPO, its KL penalty, and GRPO are used to fine-tune LLMs.50 min
Supervised learning has the right answer for every input. Reinforcement learning only gets a score, often late, for a sequence of choices. This module builds the core algorithms from scratch on small grid worlds, so every number can be checked by hand. It is the groundwork for RLHF and the PPO-based and GRPO-style fine-tuning in Module 15: there a token is an action, a response is an episode, and a reward model or verifier hands out the score.
| Idea | What it learns | Where it shows up |
|---|---|---|
| MDP and Bellman equation | the shape of the problem; values as expected return | every RL method |
| Temporal-difference learning | action values from one step at a time | Q-learning, SARSA, DQN, critics |
| Function approximation | values for states too many to list | DQN |
| Policy gradient | the policy directly, from sampled returns | REINFORCE, GRPO |
| Trust region and clipping | stable policy updates | PPO, RLHF |
| # | Lesson | You will be able to |
|---|---|---|
| 1 | MDPs, states, actions, and rewards | frame a problem as an MDP, compute returns and values, and run policy evaluation and value iteration |
| 2 | Q-learning and SARSA | compute TD updates by hand and explain why on-policy and off-policy agents take different paths along a cliff |
| 3 | Deep Q-networks | train a Q-network with experience replay and a target network, and explain why each is needed |
| 4 | Policy gradients and REINFORCE | derive the policy gradient, use a baseline to cut variance, and connect it to LLM fine-tuning |
| 5 | Proximal policy optimization | compute the clipped objective and GAE, and explain how PPO and GRPO are used to fine-tune LLMs |
Interactive lab: Q-learning on a grid world