Hugging Face Introduction to Q-Learning

Q-Learning is an off-policy, value-based reinforcement learning (RL) method that uses a Temporal Difference (TD) approach to train an action-value function. It allows an agent to learn the optimal policy for an environment by iteratively updating a Q-table, which serves as a memory of the quality of specific actions taken in specific states.

Understanding the Q-Function and Q-Table

The Q-function is an action-value function that determines the value of being in a particular state and taking a specific action. The "Q" stands for "Quality."

Internally, the Q-function relies on a Q-table, a matrix where each cell corresponds to a state-action pair value. The agent uses this table as a cheat sheet: given a state and an action, the Q-function searches the Q-table to output the corresponding Q-value.

Training progresses as follows:

  1. Initialization: The Q-table is typically initialized with zeros, meaning the agent starts with no knowledge of the environment.
  2. Exploration: As the agent interacts with the environment, it updates the Q-table with better approximations of state-action values.
  3. Optimization: Once the Q-table is optimized, the agent possesses an optimal policy because it knows the best action to take for every possible state.

The Q-Learning Algorithm

Q-Learning follows a structured process to update its action-value function at each step rather than waiting until the end of an episode.

Action Selection: Epsilon-Greedy Strategy

To manage the exploration-exploitation trade-off, Q-Learning employs an Epsilon-Greedy Strategy:

  • Exploration: With probability $\epsilon$, the agent selects a random action to discover new state-action pairs.
  • Exploitation: With probability $1 - \epsilon$, the agent selects the action with the highest state-action value from the Q-table.

As training progresses, the value of $\epsilon$ is progressively reduced (decayed), shifting the agent's behavior from random exploration to the exploitation of learned knowledge.

The Update Mechanism

After performing an action $A_t$ and receiving a reward $R_{t+1}$ and a next state $S_{t+1}$, the agent updates the value of $Q(S_t, A_t)$ using a TD target. The TD target is composed of the immediate reward plus the discounted value of the best possible state-action pair in the next state (bootstrapping).

Off-Policy vs. On-Policy Learning

Q-Learning is categorized as an off-policy algorithm because it uses different policies for acting and updating:

  • Acting Policy: The Epsilon-Greedy policy is used to select the action the agent actually performs in the environment.
  • Updating Policy: A greedy policy (which always selects the action with the highest value) is used to calculate the TD target and update the Q-value.

In contrast, on-policy algorithms, such as Sarsa, use the same policy (e.g., Epsilon-Greedy) for both acting and updating.

Practical Application and Examples

To demonstrate Q-Learning, Hugging Face utilizes a maze example where a mouse must reach cheese while avoiding poison. The agent's learning is driven by a reward function:

  • +10: Reaching the big pile of cheese (Goal).
  • +1: Reaching small cheese.
  • 0: Moving to a state with no cheese.
  • -10: Hitting poison (Terminal state/Death).

Through iterative steps of taking random actions (due to high initial $\epsilon$) and updating the Q-table based on rewards, the agent eventually learns the optimal path (e.g., right, right, down).

Training Environments

For hands-on implementation, the guide suggests two primary environments:

  1. Frozen Lake v1: A non-slippery version where the agent must navigate from a start state (S) to a goal state (G) while avoiding holes (H).
  2. Autonomous Taxi: An agent that must learn to navigate a city to transport passengers from point A to point B.

Sources