Latent-GRPO on 8× B300, Part 2: Two Silent Adapter Bugs, Parity Gates That Learned, and Rollouts on All Eight GPUs

In part 1 we made one latent-GRPO training step on Qwen 3.8 27B 14× faster by moving from one H100 to an 8× NVIDIA B300 node. That article ended with five green steps. This one is about what happened when we ran for hundreds of steps.

Long runs found problems that five steps could not. Two bugs in how vLLM applied our adapter were silently changing the model the rollouts came from. Our parity gates, strict as they were, stopped runs for reasons that turned out to be noise, and we had to redesign them without making them weaker. A safety check stopped a healthy run at step 423. And once all of that was fixed, a profile showed that seven of our eight GPUs were waiting most of the time. The last section shows how we put them to work.

1. What Changed Since Part 1

Three statements in part 1 are no longer true, and we would rather say so up front:

• “We never loosened a threshold.” We changed how the rollout parity gate is applied, twice. The limits are the same (mean gap ≤ 0.1, max ≤ 1.0 nats), but the maximum now applies to tokens that either engine gives at least 1% probability, and a rare token above the limit is removed from the loss instead of stopping the run. We added a periodic check that does not depend on sampling. Section 4 explains why and what the evidence was.
• MTP speculation. Part 1 reported answer decoding at 5–9 ms per token with MTP. MTP is now off: in real training it gave no measurable step-time gain once the compile bug below was fixed.
• PIECEWISE CUDA graphs under speculation. That setting is gone for our trainer engines. They now run without torch.compile and with FULL CUDA graphs for decode only (section 2). Answers decode at about 11.6 ms per token instead of 5–9 ms. We traded that speed for a correct adapter.

2. The Compiled Adapter: Right Weights, Wrong Output

A 50-step run stopped at step 23 on the rollout parity gate: a maximum gap of 1.07 nats over 852 tokens. We stopped training and bisected with the adapter from step 22, one token at a time, on every path we could isolate:

The LoRA weights inside vLLM were identical to the trainer’s. Both of our LoRA kernels, the multi-adapter punica path and our own dense single-adapter path, gave the same wrong answer, so the kernels were not at fault either. The difference was torch.compile: with an active adapter, vLLM 0.23’s compiled graph for the Qwen 3.5-family hybrid computed a different adapter contribution. Splitting the LoRA operations out of the compiled graph did not help.

Why did our graph qualification miss it? It compared CUDA graphs against eager execution by disabling the graphs, which left the compiled kernels in place on both sides of the comparison. Both sides were wrong in the same way. The fix is one line of engine configuration for every trainer engine: compilation_config = {mode: 0, cudagraph_mode: FULL_DECODE_ONLY}. There is no compilation, prefill runs eagerly, and decode is captured in CUDA graphs, which we verified to be bitwise equal to eager for plain decode, MTP verification and batches of eight.

3. DoRA Rounded Twice: 20% of the Adapter Gone

With compilation off, runs still failed, now on a periodic check at step 10 and on single tokens deep in a run, often the token that chose the next action. A layer-by-layer comparison showed that the base weights and the LoRA output in vLLM matched the trainer bit for bit. The DoRA magnitude scaling did not.

DoRA multiplies each output row by a learned magnitude. PEFT computes the layer in one expression and rounds once. Our vLLM integration added the LoRA term to the base output after the base output had already been rounded to BF16, then multiplied by the magnitude and rounded again:

That looks harmless, but the adapter’s contribution is about 10⁻⁴ of the layer output, smaller than one BF16 step of the output. The intermediate rounding erased about 20% of the adapter’s effect in every layer, and 64 layers compound it. After fixing it, vLLM computes the base output and the LoRA term separately, adds them with the magnitude in FP32 and rounds once, for single and packed layers alike. Every layer now matches bit for bit, and the step-22 token reads −0.28 in vLLM against −0.36 in the trainer, instead of −1.33.

Both bugs had the same signature: the rollouts came from a slightly different model than the one being trained. The importance-sampling correction hides most of that on average, which is why only the maximum gap, on rare tokens, exposed it.

4. Parity Gates That Learned From Their False Alarms

With both bugs fixed, the gate still occasionally stopped runs. Each failure was investigated before we changed anything. They fell into three kinds:

Relaxing a gate is dangerous because it can hide the next real bug. So we added a check that sampling cannot fool: every 10 steps both engines score 8 fixed prompts teacher-forced with the current adapter, and a gap above the limit stops the run. The compiled-adapter bug from section 2 would have failed this probe; the tail noise does not. Every step also writes its full parity report to a history file, so we can see how close passing steps come to the limits.

In a 423-step run afterwards, all 848 rollout checks passed, and 2 tokens were masked. All 42 periodic probes passed with maximum gaps of 0.12–0.84 and a steady mean of about 0.011.

5. Why We Turned MTP Off

MTP speculation looked like the obvious suspect for the one-in-two-thousand tokens, so we reran the failing step without it. The same kind of outlier appeared later without MTP (1.17 nats at step 31), so MTP was not the cause. What MTP did not show either was a gain: with compilation off, fresh steps took 146–210 s without MTP against a median of about 172 s with it, on different rollouts. The time was going elsewhere (section 8). A speculative decoder that saves nothing and adds a code path to qualify is not worth keeping, so the trainer runs without it.

6. Bugs That Need Hundreds of Steps

7. A Third of the Context Was Hashes

Episodes on our longest tasks often ended without an answer. The model was still reading sources when the episode’s token budget ran out. The budget of 8,192 tokens counts the model’s text and every tool result, including images.

Every source a tool returned carried a provenance receipt: record hashes, payload digests and revisions, about six 64-character hex strings per read. Hex tokenizes badly. Across 200 sampled rollouts, 31% of all tokens were hashes. The model never needs them; the audit trail on the server does. We removed them from what the model sees and kept them in the sealed server-side trace, where every read is still verified.

The untrained model, on the same held-out tasks, went from a mean reward of 0.281 to 0.440, and it now finishes 95% of episodes. No training was involved: the model simply had room to finish its work.

8. Rollouts on All Eight GPUs

With the correctness problems solved, we profiled a real run. Over 90 seconds, GPU 0 was 68% busy and GPUs 1–7 about 35%. A median fresh step of 197 s spent 118 s on vLLM answers and 47 s on the HF latent loop, all on GPU 0, and only 28 s on replay across all eight GPUs. The turn-parallel replay from part 1 used the node well; generation did not use it at all.

Now every helper also hosts a vLLM engine and runs the latent loop. A step generates 8 GRPO groups (32 rollouts) instead of one. Rank 0 keeps everything that has state: the gym, tool execution, rewards and the loss. On every model turn it sends each GPU the active rows of its groups and records the returned answers, log-probabilities and latent thoughts exactly as if it had generated them. A group stays on one GPU for the whole episode, because the four rollouts share a long prompt and that GPU’s prefill cache keeps serving it across turns. The loss, replay and parity checks did not change: each row’s importance ratio pairs the engine that sampled it with the trainer’s replay, so rows from different engines mix correctly.

Getting eight engines to run side by side took five fixes:

• Identical random streams. TRL seeds every single-process vLLM engine with 0, so all eight would have drawn the same random numbers for different groups. Each rank now gets its own seed for vLLM and for the latent noise.
• Port collisions. TRL picks the engine’s rendezvous port from a short fixed list, and seven helpers starting together took the same ports. Each helper now takes a free port from the OS.
• A stale adapter. TRL refreshes an engine’s adapter only when the step counter changes, which never happens in a helper. Each helper now reloads the adapter from its synced weights at the start of every rollout round.
• A preflight that scaled with the batch. A backward-memory check before training ran the whole scoring batch on one GPU and took tens of minutes. It now checks one GPU’s share.
• Helpers stopped too early. Training end stopped the helpers, but the final held-out evaluation still needs them to generate. They now stop after evaluation.

Parity held on every engine: rollout gaps averaged 0.003–0.007 with maxima of 0.35–0.81 in the first steps. We added a signal handler that dumps every thread’s stack on SIGUSR1, because our container does not allow py-spy to attach. It turned one “hang” into a preflight that was simply slow.

9. Replay: One Row per Round Was the Real Bottleneck

With 32 rollouts per step, replay was about to become the bottleneck. The recipe scored one row per micro-batch, a setting inherited from a path where TRL builds full-vocabulary logits for every position. Our latent replay returns only token log-probabilities, so that limit did not apply. But it meant 32 sequential rounds, each spreading one rollout’s ~10 turns over 8 GPUs and waiting for the longest one.

Micro-batches can now hold several whole groups. Scoring all 32 rows in one round took 94 s, against an estimated 220 s for 32 one-row rounds (8× the 28 s of one group). It also ran out of memory on step 3: each GPU retained activations for about 210k tokens next to its own vLLM engine. We now score 16 rows per round, keep the first 30k tokens per GPU without checkpointing, checkpoint up to 100k, and offload the rest to CPU. Replay takes 130–260 s for 32 rollouts depending on their length.

10. Results and Next Steps

The gain is 1.6×, not the 4× we first estimated. Generation still waits at every turn: all eight GPUs finish their turn, rank 0 runs the tools for all 32 rollouts, and only then does the next turn start. The longest episode on any GPU sets the pace. The other gain is statistical. Each step now averages over 8 tasks instead of 1, so reward curves are readable per 10 steps and a step with no learning signal needs all 8 groups to be uninformative.

Asynchronous GRPO (generating step N+1 while training on step N) and tensor parallelism for vLLM are the obvious next ideas. Neither is free here: both would share the same eight GPUs that generation and replay already use, so we will measure where the time goes first and only then decide.

For the method itself, see our latent GRPO deep dive, and for the adapters, our overview of LoRA and DoRA.