Demystifying Q-Learning: From Bellman Equations to a Robot Finding Its Way Home
Author: Yongfeng Gu, Date: October 7, 2026
Reinforcement Learning (RL) doesn't have to be intimidating. In this post, we build Q-Learning from the ground up using a simple 6x6 grid game, and develop an intuition for every symbol along the way.
The Core Question¶
Imagine a robot placed at the top-left corner of a 6x6 grid. Its goal is to reach the bottom-right corner in as few steps as possible. The grid contains three types of cells besides the start and goal:
- Blank cells: nothing happens.
- Mine cells (M): the robot explodes and the episode ends.
- Lightning cells (L): the robot gets a speed boost (a small bonus reward).
col0 col1 col2 col3 col4 col5 row0 [ S ] [ ] [ L ] [ ] [ ] [ ] row1 [ ] [ M ] [ ] [ ] [ M ] [ ] row2 [ L ] [ ] [ ] [ L ] [ ] [ M ] row3 [ ] [ M ] [ ] [ ] [ ] [ ] row4 [ ] [ ] [ M ] [ L ] [ M ] [ ] row5 [ ] [ L ] [ ] [ ] [ ] [ G ]
The robot has no map. It doesn't know where the mines are. It must learn the optimal path purely through experience. This is the essence of Reinforcement Learning, and Q-Learning is one of the most elegant algorithms to solve it.
What Is Q(s, a)?¶
Before we get to the algorithm, we need to understand the central object in Q-Learning: the Q-function, written as Q(s, a).
Q stands for “Quality”. Q(s, a) answers a single question:
In state s, if I take action a, and then behave optimally forever after, how much total reward can I expect to accumulate?
Q(s, a) is not the immediate reward from taking action a. It is the cumulative discounted reward over the entire remaining trajectory. The immediate reward is just the first term; the rest comes from all future decisions.
Think of choosing a college major: its true “quality” includes not only satisfaction today, but career prospects, earning potential, and life satisfaction over decades. Q(s, a) captures the full long-term value, not just short-term gratification.
In our grid game:
- State s = the robot's position (row, col), giving us 36 possible states.
- Action a = one of {Up, Down, Left, Right}.
- Q(s, a) = the expected total reward starting from position s, moving in direction a, and then always making the best possible decisions afterward.
If Q((0,0), Down) = 72 and Q((0,0), Right) = 65, then moving Down from the start is the better long-term choice.
The Q-Table: A Lookup Table for Wisdom
In tabular Q-Learning, we store Q(s, a) in a simple 2D table called the Q-Table. Rows are states, columns are actions, and each cell holds a number. At the beginning, every cell is initialized to zero:
Up Down Left Right (0,0) [ 0 | 0 | 0 | 0 ] (0,1) [ 0 | 0 | 0 | 0 ] (0,2) [ 0 | 0 | 0 | 0 ] ... (5,5) [ 0 | 0 | 0 | 0 ]
As the robot explores and observes outcomes, it fills in the table. Once it converges, the optimal policy is trivially extracted: for each state, pick the action with the highest Q-value. The robot has learned the shortest safe path.
Up Down Left Right (0,0) [ ... | 72 | ... | 65 ] ← Down is best → go Down (1,0) [ ... | 76 | ... | 60 ] ← Down is best → go Down (2,0) [ ... | 80 | ... | 55 ] ← Down is best (lightning nearby!) (3,0) [ ... | 84 | ... | 50 ] (4,0) [ ... | 88 | ... | 45 ] (5,0) [ ... | - | ... | 90 ] ← Right is best → go Right (5,1) [ ... | - | ... | 92 ] ← Lightning boosts the value (5,2) [ ... | - | ... | 94 ] (5,3) [ ... | - | ... | 96 ] (5,4) [ ... | - | ... | 98 ] (5,5) [ 0 | 0 | 0 | 0 ] ← Goal, no decision needed
Where Does the Update Rule Come From? The Bellman Equation¶
The Q-Table doesn't fill itself. We need a learning rule, and that rule comes from one of the most fundamental ideas in dynamic programming: the Bellman Equation.
The value of a decision equals the immediate reward plus the discounted value of the next state under the best future action.
Q(s, a) = R(s, a) + γ · max₍a′₎ Q(s′, a′)
R(s, a)is the immediate reward after action a in state s.γ(gamma) is the discount factor (0 < γ < 1), controlling how much we care about future rewards.s′is the new state after taking action a.max₍a′₎ Q(s′, a′)is the best Q-value achievable from the new state.
The total future reward from (s, a) decomposes into what you get now and the discounted future Q-value. The decomposition is recursive, which makes it powerful.
With γ = 0.95, if moving Right from (5,4) leads to the goal with reward +100, then Q((5,4), Right) ≈ 100. One step earlier, Q((5,3), Right) ≈ 0.95 × 100 = 95. The value propagates backward from the goal, diminishing by γ at each step.
From Theory to Learning: The Q-Learning Update¶
The Bellman Equation tells us what Q-values should be if we know the environment perfectly. Our robot does not, so it learns by doing: after every step, it compares what happened with what it expected, then nudges the Q-value toward that experience.
Q(s, a) ← Q(s, a) + α · [ r + γ · max₍a′₎ Q(s′, a′) − Q(s, a) ]
The bracketed term is the TD (Temporal Difference) Error:
| Component | Meaning |
|---|---|
r + γ · max Q(s′, a′) | Target: what experience says Q(s, a) should be |
Q(s, a) | Old estimate: what we previously believed |
| Their difference | Surprise: how wrong we were |
α (learning rate) | How aggressively to adjust, typically 0.1–0.5 |
If the surprise is positive, we increase Q(s, a); if negative, we decrease it. Over thousands of iterations, Q-values converge to their true values.
Walking Through the Algorithm: Episode by Episode
Use α = 0.1, γ = 0.95, and ε = 0.3 for exploration.
Reward scheme:
- Each step: −1 (encourages shorter paths)
- Reaching goal G: +100
- Hitting mine M: −100 (episode ends)
- Landing on lightning L: +5
Episode 1: Blind Exploration
The Q-Table is all zeros. At (0,0), a random number 0.15 < ε makes the robot explore and pick Right. It reaches (0,1), a blank cell, and receives r = −1:
Q((0,0), Right) ← 0 + 0.1 × [−1 + 0.95 × 0 − 0] = −0.1
At (0,1), the robot next moves Down to (1,1), a mine. The episode ends:
Q((0,1), Down) ← 0 + 0.1 × [−100 + 0 − 0] = −10.0
The robot now slightly disfavors Right from the start and has learned a harsh lesson about moving Down from (0,1).
Episode 2: A Glimmer of Hope
Through ε-greedy exploration, the robot discovers (0,0) → (1,0) → (2,0), where (2,0) is lightning. Choosing Down at (1,0) gets r = +5:
Q((1,0), Down) ← 0 + 0.1 × [5 + 0.95 × 0 − 0] = +0.5
The first positive Q-value appears. If the robot eventually reaches the goal, the +100 reward slowly propagates backward through the trajectory.
Episode 3 and Beyond: Value Propagation
Episode N: Q((5,4), Right) learns ≈ 100 ("Right from here → goal!")
Episode N+1: Q((5,3), Right) learns ≈ 95 ("Right → (5,4) → goal")
Episode N+2: Q((5,2), Right) learns ≈ 90.25
Episode N+3: Q((5,1), Right) learns ≈ 85.7
After hundreds of episodes, the optimal policy emerges: the robot walks down the left boundary, collects lightning boosts, avoids mines, then walks right along the bottom to the goal.
col0 col1 col2 col3 col4 col5 row0 [ ↓ ] [ ] [ L ] [ ] [ ] [ ] row1 [ ↓ ] [ M ] [ ] [ ] [ M ] [ ] row2 [ ↓ ] [ ] [ ] [ L ] [ ] [ M ] row3 [ ↓ ] [ M ] [ ] [ ] [ ] [ ] row4 [ ↓ ] [ ] [ M ] [ L ] [ M ] [ ] row5 [ → ] [ → ] [ → ] [ → ] [ → ] [ G ]
Three Concepts That Make Q-Learning Work
Exploration (controlled by ε) sometimes ignores the Q-Table and tries a random action. This discovers new paths, lightning cells, and the goal.
Exploitation follows the action with the highest current Q-value. This leverages what the robot has already learned.
Trial and Error observes every outcome and updates the Q-Table. The trial is taking an action; the error is the gap between expectation and reality, namely the TD Error.
Exploration decides what to try, exploitation decides what to trust, and trial-and-error turns experience into knowledge.
The Complete Algorithm¶
Here is Q-Learning in pseudocode:
Initialize Q-Table Q(s, a) = 0 for all s, a
for each episode:
reset robot to start state s
while s is not the goal:
// Action selection: ε-greedy
if random() < ε:
a = random action // Exploration
else:
a = argmax₍a′₎ Q(s, a′) // Exploitation
// Execute action, observe outcome
take action a
observe reward r and new state s′
// Q-value update (Bellman-inspired)
target = r + γ × max₍a′₎ Q(s′, a′)
Q(s, a) ← Q(s, a) + α × (target − Q(s, a))
// Move to new state
s ← s′
// Optionally decay ε over time
ε ← ε × 0.995
Why Does This Matter for Real-World Problems?
The grid game is simple, but the principles scale. In ad bidding, the state might be remaining budget, time of day, and historical conversion rate. The action might be a bid multiplier; the reward, conversions; mines, budget exhaustion or CPA violations; lightning, a surge in traffic quality.
Q-Learning does not need a model of the environment or transition probabilities. It only needs to act, observe, and update. For problems with too many states for a table, Deep Q-Networks (DQN) replace the Q-Table with a neural network, but the core idea remains the same.
Summary¶
| Concept | What It Means |
|---|---|
| Q(s, a) | Expected cumulative future reward from action a, then optimal behavior |
| Q-Table | Lookup table storing Q(s, a) for every state-action pair |
| Bellman Equation | Q(s,a) = R(s,a) + γ · max Q(s′, a′) |
| TD Error | The gap between the observed target and the current Q estimate |
| Update Rule | Nudge Q toward reality using the TD Error |
| Exploration | Randomly try actions to discover information |
| Exploitation | Follow the best-known action |
| Trial and Error | The loop: act → observe → update |
Q-Learning is simultaneously simple to implement and profound in implication. Optimal behavior can emerge from curiosity (exploration), experience (trial and error), and a single recursive equation (Bellman). The robot does not know the map or plan ahead; it learns one step at a time until the shortest path reveals itself in the numbers.