Reinforcement Learning from Bellman Equations to Actor–Critic

The many names in reinforcement learning become easier to organize once three questions are kept separate: What information is a state? How is future return estimated? How is the policy improved? Bellman equations connect the first two questions; actor–critic methods connect value estimation to policy optimization. Neither connection creates a rigid on-policy/off-policy classification.

State sufficiency comes before the algorithm

A discounted Markov decision process can be written as

\mathcal{M}=\langle\mathcal{S},\mathcal{A},P,R,\rho_0,\gamma\rangle.

The Markov property says that, conditioned on the current state and action, the distribution of the next state and reward does not depend on an earlier history. It does not require a finite chain to be irreducible or aperiodic; those conditions are used for particular stationary-distribution and convergence results, not to define Markov dynamics.

If the agent receives an observation $o_t$ that omits relevant history, the learning input may be non-Markov even when the hidden environment state is Markov. This is state aliasing or partial observability, not automatically “environment instability.” Common responses are to augment the state with known sufficient statistics, maintain a belief state, or let a recurrent policy encode an action-observation history.

For a trajectory, define the discounted return from time $t$ as

G_t=\sum_{k=0}^{T-t-1}\gamma^k r_{t+k}.

The state value, action value, and advantage under policy $\pi$ are

V^\pi(s)=\mathbb{E}_\pi[G_t\mid s_t=s],\qquad Q^\pi(s,a)=\mathbb{E}_\pi[G_t\mid s_t=s,a_t=a],\qquad A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s).

These are definitions, not neural-network architectures.

Bellman equations turn long horizons into local targets

Conditioning on one transition gives the Bellman expectation equations:

V^\pi(s)=\mathbb{E}_{a\sim\pi,\,s'\sim P}\left[R(s,a,s')+\gamma V^\pi(s')\right],

Q^\pi(s,a)=\mathbb{E}_{s'\sim P,\,a'\sim\pi}\left[R(s,a,s')+\gamma Q^\pi(s',a')\right].

How these identities are used depends on what is available:

Dynamic programming computes expectations from a known model and sweeps through states.
Monte Carlo estimation uses sampled returns and waits for enough of the future to be observed.
Temporal-difference estimation bootstraps from a current value estimate after one or several transitions.

For a one-step state-value critic, the TD residual is

\delta_t=r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t).

Bootstrapping usually lowers target variance and enables online updates, but approximation error in the target can introduce bias and can propagate through repeated updates. Monte Carlo targets avoid that particular bootstrap error while often carrying higher sampling variance. “Monte Carlo is unbiased and TD is biased” is therefore a useful first intuition only after the objective, truncation, function approximation, and sampling distribution have been specified.

Policy gradients: causality and baselines

For a differentiable stochastic policy, the policy-gradient theorem supports an estimator of the form

\nabla_\theta J(\theta)=\mathbb{E}_{\pi_\theta}\left[\nabla_\theta\log\pi_\theta(a_t\mid s_t)\,Q^{\pi_\theta}(s_t,a_t)\right].

In an episodic trajectory, rewards that occurred before $a_t$ cannot have been caused by $a_t$ . Replacing the full trajectory return beside the score term with reward-to-go removes such zero-expectation terms without changing the expected gradient under the usual sampling assumptions. The reason is causality, not the informal rule that “fewer summed samples always means lower variance.”

A baseline $b(s_t)$ that does not depend on the sampled action can be subtracted because

\mathbb{E}_{a\sim\pi_\theta}\left[\nabla_\theta\log\pi_\theta(a\mid s)\,b(s)\right]=0.

A well-chosen baseline often reduces variance, but an arbitrary one need not. Taking $b(s)=V^\pi(s)$ yields the advantage. In implementations, the actor loss should not differentiate through the sampled advantage unless that coupling is intentional. A learned or bootstrapped critic can trade variance for bias through value approximation; the baseline identity itself does not create bias.

Actor–critic is an architecture, not a data regime

An actor–critic method has two roles:

the actor parameterizes and improves a policy;
the critic estimates a value, action value, advantage, or related return signal used by the actor.

This design can be on-policy or off-policy. A2C and PPO commonly use fresh on-policy rollouts and a state-value critic. DDPG and SAC are off-policy actor–critic methods that train action-value critics from replay. Conversely, DQN is off-policy and value-based but has no separately parameterized actor. The axes should remain separate:

Axis	Typical choices	Question answered
Data distribution	on-policy, off-policy	Which policy generated the training data?
Improvement mechanism	value greedy, stochastic policy gradient, deterministic policy gradient	How is the policy changed?
Return estimator	Monte Carlo, TD, multi-step, eligibility trace	How is future return estimated?
Function organization	value-only, policy-only, actor–critic	Which functions are represented explicitly?

Importance sampling can correct some distribution mismatch when the target policy is absolutely continuous with respect to the behavior policy, but long products of probability ratios can have severe variance. Replay-based off-policy actor–critic algorithms instead rely on Bellman targets, coverage of relevant state-action pairs, and algorithm-specific stabilization. They are not made correct simply by replacing $V$ with $Q$ .

GAE makes the estimator trade-off explicit

Generalized advantage estimation (GAE) exponentially combines TD residuals:

\hat A_t^{\mathrm{GAE}(\gamma,\lambda)}=\sum_{\ell=0}^{T-t-1}(\gamma\lambda)^\ell\delta_{t+\ell},\qquad \delta_t=r_t+\gamma V_\phi(s_{t+1})-V_\phi(s_t).

With $\lambda$ near zero, the estimate relies heavily on short-horizon bootstrapping. With $\lambda$ near one, it approaches a Monte Carlo-style reward-to-go adjusted by the baseline. The practical bias–variance behavior depends on critic error, horizon truncation, discounting, and data correlation; it cannot be summarized as “advantage estimates are always low variance and biased.”

PPO then uses these advantages in a clipped surrogate objective. Its probability ratio permits several bounded optimization passes over a newly collected batch, but PPO remains on-policy in the usual algorithmic sense. Reusing a current rollout for multiple epochs is not the same as learning indefinitely from an experience replay buffer.

A practical derivation-to-code workflow

Fix the objective

State whether the task optimizes finite-horizon, continuing discounted, or average reward. Discounting is part of the objective or estimator design; it should not be called “bias” without naming the undiscounted quantity being used as the reference.

Validate the state

Predict the next observation and reward from the proposed state-action pair. Large systematic residuals, improved predictions from longer histories, or policies that require hidden simulator variables are evidence that the state representation is insufficient.

Audit targets

Log reward-to-go, bootstrap value, TD residual, and advantage separately. Apply terminal masks to true terminal states and distinguish them from time-limit truncations. Normalize advantages only after masking invalid samples.

Audit policy shift

For on-policy methods, track approximate KL, clip fraction, and entropy. For off-policy methods, track replay age, behavior coverage, critic target scale, and overestimation. A falling critic loss alone does not establish a better policy.

Compare along one axis at a time

When an experiment changes the encoder, replay scheme, critic target, and policy objective together, its result cannot identify the useful component. Start with tabular or small deterministic checks of the Bellman target, then add approximation and stochasticity.

The unifying view is simple: Bellman equations define consistent value relationships; sampling schemes estimate them; the actor turns an estimated improvement direction into a new policy. On-policy versus off-policy, Monte Carlo versus TD, and value-based versus actor–critic describe different design axes and should not be collapsed into one taxonomy.