Multi-Reward RL, Part 2: Benchmarking GRPO, DAPO, and CISPO on Unseen Tasks

Follow-up: Part 3 scales the CISPO + REPO-R recipe to Qwen3.8-27B and 600 steps, with a one-change-per-run holdout ladder and an advantage-floor failure we found in a harsher environment.

Part 1 analyzed how PPO, GRPO, DAPO, and GDPO balance competing reward objectives in theory. In this follow-up empirical benchmark, we test the practical question engineering teams actually face: which trainer algorithm actually produces a model that solves brand-new, unseen problems?

We ran seven distinct RL algorithms on Qwen3-14B in our deterministic decentralized exchange (DEX) arbitrage gym across two reasoning regimes: No-think (direct execution) and Thinking-2048 (with chain-of-thought tokens enabled). The results revealed a severe cautionary tale for post-training teams.

1. What the agent does: The DEX gym

DEX stands for decentralized exchange. In our gym, the agent is presented with a frozen snapshot of automated market maker (AMM) liquidity pools: token reserves, exchange fees, gas prices, and routing constraints. The agent’s task is to find an optimal two-to-four pool arbitrage cycle and compute the exact input amount (amount_in_wei) that returns more of the starting asset after all fees and gas deductions.

WETH (Wrapped Ether) is the asset at both ends of the route. Consider an intuitive triangular arbitrage cycle: the model starts with WETH, swaps into USDC, swaps USDC into UNI, and swaps UNI back into WETH.

• If the trade size is too small: The fixed Ethereum gas fees eat 100% of the price discrepancy, yielding a negative net return.
• If the trade size is too large: The constant-product liquidity formula ($x \cdot y = k$) causes severe price slippage, collapsing the margin.
• If the pool sequence is invalid: The simulated EVM contract reverts and the transaction fails immediately.

The agent interacts using structured tool calls: inspecting pool reserves, proposing a pool route and trade amount, evaluating feedback, and finalizing. The verifier replays the transaction bytecode using a native Foundry / revm EVM execution engine. It scores the verified mathematical execution on the simulated blockchain, rather than trusting any self-reported text output by the model.

The composite training reward consists of three distinct channels:

• Native Task Score (R_native ∈ [0, 1]): The verified net profit margin returned to the starting wallet after all pool fees and gas costs.
• Execution Efficiency (R_efficiency ∈ [0, 1]): Rewards solving the task compactly in 2–3 tool turns instead of exhausting all 6 turns: max(0, 1 − tool_steps / 6).
• Reliability Gate (R_reliability ∈ {-1, +1}): +1.0 if the transaction executed cleanly without syntax errors or contract reverts; −1.0 if the call failed.

2. What we held fixed: The training contract

To isolate the impact of the RL trainer algorithms, every other variable was strictly controlled:

• Model and Quantization: Qwen3-14B (revision 40c06982…), quantized NF4 weights with BF16 compute, LoRA rank 32, alpha 64.
• Compute Hardware: One dedicated NVIDIA RTX PRO 6000 Blackwell 96 GB per training run, Hugging Face generation, fused SDPA attention, CUDA graphs disabled.
• Optimizer Schedule: Fixed seed 42, learning rate 1e-5 with linear decay, 50 optimizer steps, group size 4, eight completions per fresh rollout batch, two policy updates per rollout. Curriculum learning disabled.
• Reasoning Modes:
  • No-think: Chain-of-thought thinking disabled; max 1,024 generated tokens per tool turn (max 8,192 trajectory cap).
  • Thinking-2048: Model allocated up to 2,048 thinking tokens inside a 3,072-token turn budget (max 22,528 trajectory cap).
• Frozen Holdout Evaluation: Evaluated on the final step-50 checkpoint. 5 task families × 2 held-out seeds × 4 samples = 40 episodes across 10 distinct unseen scenarios. Untrained starting checkpoints evaluated identically in both reasoning modes.

3. Trainer features: Seven algorithm bundles

In modern post-training, an RL algorithm is not a monolithic block. It is a bundle of three distinct modular decisions: which rollouts enter the training batch, how multiple rewards are combined into policy advantages, and how that advantage scales policy gradient updates.

GDPO decouples reward-channel normalization before combining advantages. CISPO replaces ratio clipping with direct importance weight clipping (capped at 1.20). ADAPO and REPO-R introduce adaptive entropy control: ADAPO adjusts clipping bounds based on policy entropy, while REPO-R shapes token-level advantages. DAPO-refill activates replacement sampling when a rollout group exhibits zero reward variance, preventing zero-gradient wasted batches.

4. Full results: Train vs. holdout test

No-think mode: Every trained model beats the starting baseline

Without reasoning tokens, post-training delivers an unambiguous leap forward. The untrained base model scored only 0.3856 (passing 16/40 episodes). DAPO achieved 0.6467 (39/40 successes), closely followed by GRPO at 0.6424. Every single trained configuration substantially outperformed the starting model.

Thinking-2048 mode: The starting model sets a high bar

When chain-of-thought thinking tokens are enabled, the starting model achieves 0.6830 straight out of the box (passing 37/40 episodes). Only DAPO-refill surpassed the untrained model on native holdout reward (0.7105, an incremental gain of +0.0275). GRPO (0.6798) and standard DAPO (0.6703) finished slightly below the base model’s score despite high success rates (38/40), demonstrating that success counts and solution quality answer different questions.

5. Explore the curves: Rollout telemetry

Each 50-step run generated 25 fresh rollout reward checkpoints (even optimizer steps update on rollouts generated on odd steps). The chart below shows three of them in thinking mode:

Use the interactive explorer below to compare any two trainer configurations across the 50-step trajectory:

Over its last ten reward points CISPO averaged 0.895 in thinking mode and DAPO 0.792, yet on unseen tasks DAPO scored 0.670 and CISPO 0.607. The training reward is measured on training tasks and includes bonuses, so it is no substitute for a frozen holdout.

6. Time and memory: The cost of thinking

Test-time reasoning comes with a heavy computational invoice. Across all seven algorithms, an average thinking step took 4.5–5.9× longer than a no-think step. Fifty training steps required approximately 12.3–14.1 hours of GPU execution in thinking mode, compared to 2.2–3.1 hours in no-think mode.

In thinking mode, peak memory stayed between 66.5 and 71.8 GiB allocated (74.0–79.6 GiB reserved), within the 96 GB budget of the Blackwell GPU. The primary operational bottleneck is wall-clock rollout latency, not GPU memory.

7. Combining features: The CISPO + REPO-R hybrid

Can we rescue CISPO’s optimization speed while fixing its generalization collapse? We tested a hybrid architecture: using CISPO’s capped importance weights (cap 1.20) combined with REPO-R’s token-level advantage shaping.

To stabilize entropy dynamics, we introduced a nonnegative window-5 controller: the first 5 optimizer steps record entropy drift without intervention (ζ = 0). Once calibrated, the entropy control strength is constrained strictly to 0 ≤ ζ ≤ 0.05.

The completed hybrid achieved 0.6243 native holdout reward and 34/40 successes, outperforming standalone CISPO (0.6068, 33/40). While this +0.0175 lift confirms that advantage shaping mitigates policy collapse, the hybrid still trailed the raw untrained thinking baseline (0.6830).

Enabled is not the same as exercised

In no-think mode, DAPO-refill triggered 19 replacement rollout rounds. In thinking mode, DAPO-refill triggered zero refills—the model never produced an all-identical reward batch. Therefore, we cannot attribute the thinking DAPO-refill lead (0.7105) to the refill mechanism itself; it functioned identically to standard DAPO under that seed.

8. What to test next: Next iteration & replication

Based on these findings, our production post-training recommendations for multi-step reasoning workflows are:

1. In No-Think Mode: Standard DAPO and GRPO are clear, cost-effective winners. They raised the holdout score from 0.386 to about 0.64–0.65 (16/40 to 38–39/40 successes) in about 2.5 GPU-hours.
2. In Reasoning Mode: Always evaluate your untrained base model with reasoning enabled before declaring training success. DAPO-refill is the primary candidate for replication.
3. Never Pick a Checkpoint by Training Curves: CISPO proved that soaring training curves can mask test-set degradation. All promotion decisions must be gated by frozen holdout benchmarks.
4. Multi-Seed Replication: Expand seed sweeps (seeds 43, 44) across novel scenario families to confirm that DAPO-refill’s +0.0275 edge holds across market distributions.

Evidence and method references

All raw empirical telemetry is publicly available. The public JSON export contains complete run manifests, SHA256 hashes, all 791 recorded optimizer steps, 396 fresh reward logs, and 680 evaluated holdout episodes. The Markdown companion provides searchable per-step records.

Foundational algorithm publications: DeepSeekMath / GRPO; DAPO; GDPO; MiniMax-M1 / CISPO; ADAPO and REPO-R.