Latent-GRPO: Reinforcement Learning in Continuous Thought Space

When you sit down to solve a complex puzzle or plan three moves ahead in chess, do you narrate every synaptic firing to yourself in full, grammatically correct English sentences? Of course not. Human cognition operates across continuous, multi-dimensional mental representations—spatial, relational, intuitive. We only convert our thoughts into human language when we need to speak to someone else.

Yet the dominant paradigm in AI reasoning today (from DeepSeek-R1 to OpenAI o1) forces neural networks to do the exact opposite. We train models to emit thousands of discrete text tokens enclosed in <think>...</think> blocks: “Wait, let me think... Actually, let me reconsider... Hmm, what if I try...”

During reinforcement learning (RL) post-training, this imposes an astronomical token tax. Models burn 80% to 90% of their generation time emitting sprawling English prose that no human will ever read. In multi-turn engineering workflows (such as multi-step SQL querying, hardware synthesis, or complex network routing), verbose text reasoning exhausts context limits, balloons the KV-cache, and stretches training runs from hours into days. Worse: in our experiments, up to a third of all model rollouts got chopped off mid-sentence by token limits, scoring zero reward and poisoning gradient updates.

Latent-GRPO asks the obvious question: Why should an artificial neural network think in human words? By replacing discrete vocabulary tokens with continuous recurrent thought vectors in embedding space, models can reason internally across continuous manifold dimensions. Below, we break down the mathematics of Latent-GRPO, explore why policy gradients over continuous thoughts are surprisingly tricky, and share empirical data from our research platform (gft-studio) training 27B parameter agents on NVIDIA H100 and H200 clusters.

1. The Token Tax: How Textual Chain-of-Thought Kills Rollouts

In discrete Chain-of-Thought (CoT), reasoning is forced through an artificial bottleneck: the model’s vocabulary projection layer. For a modern foundation model like Qwen3.6-27B:

• The Autoregressive Penalty: Every single intermediate thought word requires a full forward pass through 64 transformer layers, a massive projection across 152,064 vocabulary logits, and an addition to the active KV-cache state.
• Context Window Choke: In agent tasks, environment observations and schema definitions already consume 1,500 to 3,000 tokens. Generating another 3,000 tokens of internal soliloquy per turn blows right past sequence limits.
• The Truncation Disaster: In our gft-studio multi-hop routing runs with discrete CoT,19% to 34% of all generated trajectories hit the sequence ceiling mid-sentence. In reinforcement learning, a truncated completion fails its verifier and receives a reward of exactly 0. Your trainer ends up penalizing a model not because its reasoning was wrong, but because it didn’t finish talking before the bell rang.

Continuous latent thinking eliminates this bottleneck: instead of projecting hidden states onto discrete words, can the model feed its internal recurrent representations directly back into itself in continuous space (d = 5,120)?

2. The Core Dilemma: Why Continuous Policy Gradients Are Hard

If continuous reasoning is so obviously superior, why isn’t everyone doing it? The obstacle is not deep learning architecture—it is the foundational mathematics of policy gradients.

In Group Relative Policy Optimization (GRPO), the trainer samples a group of G candidate trajectories {o₁, o₂, ..., o_G} from an old policy π_{θ_old} for a given prompt q. The environment computes scalar task rewards r_i, which are normalized across the group into advantages:

The policy network is then optimized using the clipped surrogate objective:

This formulation relies on two absolute requirements:

1. Stochastic Exploration: The policy must sample distinct candidate rolloutso_i ≠ o_j so that the advantage A_i can tell winners from losers.
2. Tractable Action Densities: The trainer must be able to compute the exact log-probability log π_θ(a_t | s_t) of the sampled action under both current and old weights to form the importance sampling ratio π_θ / π_old.

3. The Research Landscape: How to Add Noise to Thought

Every major research paper in continuous reasoning over the last two years is essentially an answer to one question:Where do we inject randomness into the thoughts, and how do we calculate the probability density of that noise?

3.1 SofT-GRPO: Treating Gumbel Perturbations as the Action

In SofT-GRPO (Zheng & Lee, arXiv:2511.06411), the model forms “soft tokens” by perturbing top-K logits with Gumbel noise: g′_i = log p_i + ε_i. The input embedding is the softmax-weighted superposition of those top embeddings.

The breakthrough insight: the action is the noise vector ε_t itself, not the continuous embedding. This allows computing exact closed-form importance sampling updates through continuous thought steps.

3.2 Latent-GRPO: Stabilizing the Mixture

Building on soft tokens, Latent-GRPO (Deng et al., 2026) adds three practical guardrails:

• Dead-Sample Masking: Trajectories that fail to finish within the turn budget are immediately excised from group variance calculations.
• One-Sided Perturbations (ξ > 0): Ensures that noise boosts promising paths without accidentally degrading probabilities on valuable tokens.
• Winner-Take-All Reinforcement: When multiple rollouts in a group solve the problem, only the top-scoring trajectory’s initial latent vector receives positive gradient reinforcement, avoiding muddy representation drift.

3.3 CoLaR: Direct Gaussian Latent Heads

Instead of constraining thoughts to the vocabulary hull, CoLaR (Tan et al., NeurIPS 2025) equips the transformer with an MLP head that predicts mean and variance (μ_t, σ_t) directly in representation space. The thought is sampled as z_t = μ_t + σ_t ⊙ ε, yielding a clean analytical Gaussian log-density:

4. Putting It into Code: The gft-studio Implementation Ladder

In our research platform (gft-studio), we implemented this progression step-by-step:

• L0 (Deterministic Loop, Answer-Only GRPO): A simple residual ThoughtProjectorfeeds the last hidden state back: e_t+1 = e_t + Projector(h_t). The model takes 8 deep thought steps silently. GRPO gradients only optimize the subsequent discrete action tokens.
• L1 (Dynamic Thought Budgets): The model learns boundary tokens allowing it to decide dynamically whether a problem needs 2 latent cycles or 12.
• L2b (Gaussian Latent Policy): The projector predicts Gaussian parameters, allowing full policy gradient backpropagation through the internal thought cycles.

The Engineering Wall: Two-Pass Gradient Replay

Here is a practical systems trap that cost us days of debugging: gradient checkpointing deletes KV caches inside checkpointed transformer layers. If you try to extend an existing cached sequence during backward passes in PyTorch, you get violent tensor shape mismatch errors between cached keys and attention masks.

We solved this with a Two-Pass Replay Architecture:

1. Pass 1 (Thought Generation in Eval Mode): Recompute the exact continuous thought vectors without gradients using an active KV cache (mirroring rollout inference).
2. Pass 2 (Unified Embedding Forward): Stitch the tensors into a single unified sequence[prompt_embeds, thought_embeds, answer_embeds] and run a standard forward pass with gradient checkpointing, computing exact loss and gradients across all answer tokens in one shot.

5. Benchmarks: Discrete CoT vs. Latent Thinking on Qwen3.6-27B

We pitted discrete Chain-of-Thought against continuous latent thinking in a controlled 4-arm study on a complex multi-hop-routing-gym using Qwen3.6-27B with QLoRA on NVIDIA H100 and H200 clusters:

What the Data Actually Tells Us

• The 14x Speedup Dissected: In our initial runs, the Latent L2b arm finished rollouts in 132.7 seconds compared to 1,878.4 seconds for Hugging Face Discrete CoT—a staggering 14.2x speedup. To be completely honest, that jump comes from two combined factors: moving from H100 to H200 memory bandwidth (4.8 TB/s) and compressing sequence length from ~4,131 chatty tokens down to ~688 action tokens.
• Continuous Thoughts Consume Zero Token Budget: This is the true superpower. While the discrete CoT arm repeatedly truncated its second reasoning cycle and forfeited its reward, the Latent arm executed 8 deep recurrent thought cycles without consuming a single token from the 1,024-token sequence ceiling.
• Total Training Wallclock: Over the full 50-step run, Latent L2b finished all training in 2.38 hours of total cluster time, compared to a projected 26+ hours for discrete CoT.

6. The Road Ahead: Pure Latent Cognition

Teaching language models to reason by making them write endless stream-of-consciousness essays was a brilliant stepping stone. But as agents tackle multi-step tools, deep code refactors, and complex simulations, forced textualization becomes an unacceptable bottleneck.

The future of agent post-training isn’t models that talk faster. It is models that think silently, deeply, and continuously in latent space—and only speak when they have something to say.