Hugging Face Deep RL Course Unit 2 Part 1: Introduction to Q-Learning Concepts

TL;DR

Hugging Face published the first installment of its Deep Reinforcement Learning class, covering the theory behind value‑based methods, the Bellman equation, and the distinction between Monte‑Carlo and Temporal‑Difference learning, which are essential foundations for the upcoming Q‑Learning tutorial.

What is Reinforcement Learning? – Core Idea

Reinforcement Learning (RL) trains an agent to make sequential decisions by interacting with an environment, receiving rewards as feedback, and aiming to maximize expected cumulative reward. The decision rule is called a policy (π), which maps each observed state to an action (or a probability distribution over actions). The ultimate objective is to discover an optimal policy π\* that yields the highest possible expected return.

Two broad families of RL methods exist:

  • Policy‑based methods: directly optimize the policy parameters.
  • Value‑based methods: learn a value function that estimates how good a state (or state‑action pair) is, then derive a policy from that function (often a greedy or ε‑greedy policy).

The article focuses on the value‑based side.

Two Types of Value‑Based Functions

State‑Value Function (V)

The state‑value function under a policy π is defined as:

( V^{\pi}(s) = \mathbb{E}{\pi}\big[ \sum{k=0}^{\infty} \gamma^{k} R_{t+k+1} \mid S_t = s \big] )

It gives the expected discounted return when the agent starts in state s and follows π thereafter.

Action‑Value Function (Q)

The action‑value function extends V to state‑action pairs:

( Q^{\pi}(s, a) = \mathbb{E}{\pi}\big[ \sum{k=0}^{\infty} \gamma^{k} R_{t+k+1} \mid S_t = s, A_t = a \big] )

It measures the expected return when the agent starts in state s, takes action a, and then follows π.

Both functions ultimately represent expected returns, but Q provides the granularity needed for greedy action selection in value‑based algorithms.

The Bellman Equation – Recursive Value Estimation

Computing V or Q by enumerating all possible future trajectories is infeasible. The Bellman equation offers a recursive alternative:

( V^{\pi}(s) = R_{t+1} + \gamma ; V^{\pi}(s') )

where (s') is the next state after taking the action prescribed by π. For Q‑values, the recursion becomes:

( Q^{\pi}(s, a) = R_{t+1} + \gamma ; \mathbb{E}{s'}\big[ \max{a'} Q^{\pi}(s', a') \big] )

In the article’s simplified examples, the discount factor (\gamma) is set to 1, so the Bellman update reduces to immediate reward + value of the next state. This recursion underpins both Monte‑Carlo and Temporal‑Difference learning.

Monte‑Carlo vs. Temporal‑Difference (TD) Learning

Both methods use experience to update value estimates, but they differ in when and how the update occurs.

Monte‑Carlo (MC) Learning

  • When? After an entire episode finishes.

  • How? Compute the return (G_t = \sum_{k=t}^{T} \gamma^{k-t} R_{k+1}) for each visited state and use it as a target:

    ( V(s) \leftarrow V(s) + \alpha \big[ G_t - V(s) \big] )

  • Pros: Uses the true discounted return, no bias from bootstrapping.

  • Cons: Requires complete episodes; high variance; slower to propagate information.

Temporal‑Difference (TD) Learning

  • When? After every single step.

  • How? Form a TD target using the immediate reward and the current estimate of the next state’s value:

    ( V(s) \leftarrow V(s) + \alpha \big[ R_{t+1} + \gamma V(s') - V(s) \big] )

  • Pros: Updates online, lower variance, faster learning.

  • Cons: Introduces bias because it relies on the current estimate of (V(s')) (bootstrapping).

The article illustrates both approaches with a simple grid‑world mouse example, showing step‑by‑step calculations of updates for a learning rate (\alpha = 0.1) and (\gamma = 1).

Summary of Key Takeaways

  • Value‑based RL learns a function (V or Q) that predicts expected returns; the policy is derived from this function (e.g., greedy or ε‑greedy).
  • State‑value (V) evaluates states; action‑value (Q) evaluates state‑action pairs, enabling direct action selection.
  • The Bellman equation provides a recursive formulation that replaces exhaustive summation with a simple update: immediate reward + discounted next‑state value.
  • Monte‑Carlo updates after full episodes using the true return; TD learning updates after each step using a bootstrapped estimate.
  • Mastering these concepts is essential before tackling Q‑Learning, the first deep RL algorithm that achieved human‑level performance on Atari games.

Next Steps in the Course

The article announces that the second part will cover Q‑Learning and include hands‑on implementations in two environments:

  1. FrozenLake‑v1 (non‑slippery) – navigate from start (S) to goal (G) while avoiding holes (H).
  2. Taxi‑v3 – learn to transport passengers between locations in a grid city.

Students are encouraged to test their understanding with a quiz (linked in the post) and to provide feedback via a Google Form.

Keep learning, stay awesome!

Sources