Multi-Reward Reinforcement Learning for LLM Agents: Comparing PPO, GRPO, DAPO, and GDPO

If you only ever train language models on toy math puzzles, reinforcement learning feels simple: did the model output 42? If yes, reward is 1. If no, reward is 0.

The moment you try to train an autonomous agent for real-world enterprise work, however, a single scalar reward is an absolute illusion.

In production, your agent has to juggle multiple competing, messy, non-commensurate priorities at the same time:

• The Main Mission (R₁): Did it actually solve the customer’s request? (e.g., execute the right SQL query, compile the circuit, return the right data payload).
• Execution Efficiency (R₂): Did it solve it elegantly in 3 tool calls, or did it run a wild 40-step loop that burned $4 in API tokens and spiked database CPU?
• Hard Guardrails & Constraints (R₃): Did it stay inside the sandbox? Did it strictly adhere to output JSON schemas, avoid mutating production tables, and preserve security invariants?

Here is the dirty secret of post-training: if you take these three scores and simply add them together into standard algorithms like PPO or vanilla GRPO, your training run will almost certainly tear itself apart. The loudest reward channel swallows the subtle ones, the agent learns to game the system, and up to a third of your expensive GPU batches end up generating zero gradients.

Below, we trace how policy gradient estimators evolved from PPO and GRPO to DAPO and GDPO, explain the mathematics of why “Sum-then-Normalize” is fundamentally broken, and share empirical data from our research platform (gft-studio) showing why decoupled normalization (GDPO) is the key to training well-behaved multi-objective agents.

1. The Evolution: PPO → GRPO → DAPO → GDPO

To understand why modern agent training looks the way it does, we have to look at the pain points that forced each algorithmic step forward:

1.1 PPO: The VRAM-Hungry Workhorse

Proximal Policy Optimization (Schulman et al., 2017) was the engine behind the original RLHF revolution. It uses an Actor-Critic architecture: the Actor generates the tokens, and a separate Critic network learns to predict the expected future reward from state s.

Why PPO Hurts in Practice:

1. The VRAM Double-Tax: If your policy is a 27B model, your Critic is usually another 27B model. You have to hold two massive models in GPU memory along with their optimizer states and activations. You end up needing twice as many H100s just to keep the critic alive.
2. Critic Drift on Multi-Reward Tasks: Trying to train a single critic head to predict a composite stew of task accuracy, latency penalties, and format compliance is notoriously unstable. The critic gets confused, advantages get noisy, and policy updates turn sluggish.

1.2 GRPO: Ditching the Critic Entirely

DeepSeekMath (2024) introduced Group Relative Policy Optimization (GRPO), and it felt like a breath of fresh air. GRPO tossed the Critic network into the recycling bin.

Instead of asking a neural net to predict a baseline, GRPO samples a group of G candidate completions{o₁, o₂, ..., o_G} for the same prompt, scores them all, and normalizes advantages against the group’s own mean and standard deviation:

Instant win: GPU memory needs dropped in half! But when applied to multi-objective environments, vanilla GRPO made an innocent-looking mathematical assumption called Sum-then-Normalize:

As we will see in a moment, this single line of math creates a devastating failure mode: Scale Dominance.

1.3 DAPO: Rescuing Dead Groups with Dynamic Sampling

When you run GRPO on difficult engineering problems, you quickly discover the curse of Dead Groups. If a coding task is tough and all 8 candidate rollouts in a group fail with a syntax error, every single rollout gets a reward of 0.

When all rewards are 0, the group standard deviation is 0. That means the advantage is 0 across the entire group! Your expensive GPU cluster just spent 30 seconds generating tokens, and the gradient update is completely empty. In hard tasks, 30% to 40% of all training steps can be dead groups.

DAPO introduced dynamic sampling: during rollout scoring, if a group has zero variance, it immediately discards the dead data and pulls fresh active prompts until every training batch contains real learning signal, cutting wasted GPU cycles to near zero.

1.4 GDPO: Decoupled Normalization (Normalize-then-Sum)

Introduced in 2026 (arXiv:2601.05242) and integrated into modern libraries like TRL 1.7, GDPO fixes the fundamental multi-reward flaw of GRPO. Instead of adding raw scores together and then normalizing, GDPO enforces Normalize-then-Sum:

1. Independent Channel Standardization: Every reward objective is normalized independently across the group first:
   A_{i,k} = (r_{i,k} - mean(r_k)) / (std(r_k) + ε)
   Now, whether a reward channel naturally varies between [0, 1.0] or between [0, 0.05], both channels have a mean of 0 and a variance of 1. The small metric can no longer be bullied by the large one.
2. Weighted Linear Assembly with Safety Clamping:
   A_i^{GDPO} = clip( ∑_{k=1}^K w_k · A_{i,k}, -c_{max}, +c_{max} )
   Clean, balanced advantages where secondary safety constraints actually matter.

2. The Mathematics of Scale Dominance: Why Simple Sums Fail

To see why Sum-then-Normalize fails, look at the variance of a sum of two independent reward signals:

In an agent task:

• Channel 1 (Task Success) is binary: r₁ ∈ {0, 1}. Its variance is roughly σ₁² ≈ 0.25.
• Channel 2 (Token Efficiency) is small: r₂ ∈ [0, 0.05]. Its variance is tiny: σ₂² ≈ 0.0006.

When you sum them up, Channel 1 accounts for 99.7% of the total variance. When GRPO divides by σ_R, Channel 2 is effectively multiplied by zero. The model quickly learns a toxic heuristic: “I can spam 50 unnecessary tool calls and burn huge token budgets, because the efficiency penalty is mathematically invisible to my gradients!”

3. Geometric Reward Collapse: When Winners Look Like Losers

Under vanilla GRPO, distinct trade-offs get mashed into the exact same scalar score:

• Candidate A: Perfect task solution (R₁ = 1.0), but horribly violates formatting rules (R₃ = 0.0). Total = 1.0.
• Candidate B: Flawless format and safe execution (R₃ = 0.2), but solved 80% of the core task (R₁ = 0.8). Total = 1.0.

To vanilla GRPO, both candidates look identical (A_A = A_B). The policy receives zero gradient to choose the clean, compliant solution over the broken, unsafe one. With GDPO’s decoupled normalization, Candidate B’s excellence on the constraint channel stands out with a strong positive sub-advantage, guiding the model toward the true Pareto frontier.

4. Real Telemetry from gft-studio

We tested all four algorithms on an identical multi-objective agent gym in gft-studio, evaluating task success (R₁), token efficiency (R₂), and constraint compliance (R₃) on a 27B model:

The Big Takeaways

• Vanilla GRPO Bleeds Safety Constraints: While it solved the primary task 51.2% of the time, its constraint violation rate was alarming: 29.4%. Because task reward dominated the sum, the agent routinely broke formatting and safety contracts to get the job done.
• DAPO Saves 34% of Wasted Compute: Over a third of vanilla GRPO groups had zero variance. DAPO’s dynamic prompt replenishment ensured that every batch drove meaningful parameter updates.
• GDPO Nails the Pareto Frontier: By standardizing each reward channel independently, constraint violations collapsed from 29.4% down to under 3.8%, while overall task success jumped to 63.4%.

5. The Practitioner’s Field Guide

When building reinforcement learning pipelines for real-world enterprise agents:

1. Never use joint summation for multi-reward environments. Always normalize channels independently first (Normalize-then-Sum).
2. Turn on dynamic group sampling (DAPO) if your task is hard. If your baseline model solves the problem less than 20% of the time, you will waste huge amounts of GPU compute on dead groups without it.
3. Clamp your combined advantages. When combining multiple normalized channels, an outlier rollout that spikes on two channels simultaneously can cause massive policy drift. Always apply a safety clamp (c_max ∈ [3.0, 5.0]) to protect training stability.