Policy Gradient: Learning to Act by Learning What Works

Author: Yongfeng Gu, Date: October 12, 2026

Policy Gradient methods take a fundamentally different approach from Q-Learning and DQN. Instead of learning how good each action is before deriving a policy, they directly learn the policy itself: a neural network that outputs action probabilities and improves through trial and error.

The 6×6 Grid World

Our robot starts at (0,0) and must reach G at (5,5). Mines carry a -100 penalty, lightning cells offer +5, reaching G earns +100, every move costs -1, and γ = 0.99.

        col0   col1   col2   col3   col4   col5
row0  [ S  ] [    ] [    ] [    ] [    ] [    ]
row1  [    ] [ M  ] [    ] [    ] [ M  ] [    ]
row2  [    ] [    ] [    ] [ L  ] [    ] [ M  ]
row3  [    ] [ M  ] [    ] [    ] [    ] [    ]
row4  [    ] [    ] [ M  ] [ L  ] [ M  ] [    ]
row5  [    ] [    ] [    ] [    ] [    ] [ G  ]

From Value-Based to Policy-Based

Q-Learning derives a policy through π(s) = argmaxₐ Q(s, a) and separately uses ε-greedy exploration. Policy Gradient instead represents a stochastic policy directly:

π(a|s; θ) = P(action = a | state = s, parameters = θ)
J(θ) = E[total return]

• No argmax: actions are sampled from a probability distribution.
• No ε-greedy: stochasticity provides exploration.
• Direct optimization: gradient ascent maximizes J(θ).

The Policy Network Architecture

Input:  36-dimensional one-hot robot position
        ↓
        Linear(36, 128) + ReLU
        ↓
        Linear(128, 128) + ReLU
        ↓
        Linear(128, 4) → logits
        ↓
        Softmax → [π(Up|s), π(Down|s), π(Left|s), π(Right|s)]

Its shared parameters θ allow a useful pattern learned in one state to generalize to similar states.

The REINFORCE Algorithm

REINFORCE plays a complete episode, reinforces actions that produced good outcomes, and suppresses actions that produced bad ones.

J(θ) = Eτ[R(τ)]
∇θJ(θ) = Eτ[Σₜ ∇θ log π(aₜ|sₜ; θ) · Gₜ]
Gₜ = Σₖ₌ₜᵀ γᵏ⁻ᵗrₖ
L(θ) = -Σₜ₌₀ᵀ log π(aₜ|sₜ; θ) · Gₜ

Every step contributes to the loss. Positive return increases the probability of its chosen action; negative return decreases it.

Step-by-Step: How the Policy Learns

A trajectory that enters a mine has negative returns and lowers the probability of its decisions. A path that reaches lightning and then the goal has positive returns, reinforcing every action along that trajectory. After many episodes, the robot can learn to travel down the left column and then right along the bottom row.

(0,0) → Right → (0,1) → Down → (1,1) = MINE!
G₁ = -100; G₀ = -1 + 0.99 × (-100) ≈ -100

Gradient Computation and Backpropagation

For a selected action aₜ, the Softmax log-likelihood gradient is:

∂Lₜ/∂zⱼ = Gₜ × (π(j|sₜ) - 1)  if j = aₜ
∂Lₜ/∂zⱼ = Gₜ × π(j|sₜ)          if j ≠ aₜ

Backpropagation carries each contribution through all layers, while the final gradient sums contributions from every time step: ∂L/∂θ = Σₜ ∂Lₜ/∂θ.

Complete Algorithm Flow

for each episode:
    collect trajectory using current policy π
    compute discounted returns Gₜ backward
    loss = Σₜ -log π(aₜ|sₜ; θ) × Gₜ
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Phase	Output
Sampling	Complete trajectory.
Compute returns	[G₀, G₁, ..., Gₜ].
Compute loss	Scalar loss L.
Update parameters	Updated θ.

Comparison with Q-Learning and DQN

Dimension	Q-Learning / DQN	Policy Gradient
Learned quantity	Q(s, a): action values.	π(a|s): action probabilities.
Action selection	ε-greedy on Q-values.	Direct sampling from π.
Exploration	Explicit ε-greedy.	Automatic stochastic sampling.
Output	Q-values.	Probability distribution.
Suitable for	Small, discrete action spaces.	Discrete and continuous action spaces.

Connecting to Ad Bidding (RTA)

For auto-bidding, the state can include remaining budget, pace, time of day, and performance metrics; actions can adjust the bid multiplier; and reward can represent KPI achievement. A policy-gradient model can directly learn a bidding strategy while naturally exploring alternatives.

Conclusion

Policy Gradient changes the question from “How good is each action?” to “How likely should each action be?” REINFORCE follows three steps: play, evaluate, reinforce. It needs no value function, Bellman equation, or target network—only a policy network shaped by the consequences of its own actions.

This post is part of a series on RL fundamentals for ad bidding. Previous posts covered Q-Learning and DQN using the same 6×6 grid-world example.