Pushing the Limits: Extreme Inference Speedup of Qwen 3.8 27B on NVIDIA B300 (100 to 10k+ tok/s)

Deploying a 27-billion parameter reasoning model like Qwen 3.8 27B on modern hardware presents a stark paradox. If you boot a default configuration on an NVIDIA B300 SXM6 GPU and send a solitary stream, you will measure roughly 104 tokens per second. The GPU sits largely cold, constrained by memory bus round-trips for every single token it emits.

Yet under the right architecture, that exact same model on the exact same B300 silicon can pump out over 10,000 tokens per second in our custom stack, and surge past 16,900 tokens per second under maximum cluster saturation. That is an astonishing 162× throughput expansion.

Getting there is not a matter of flipping a single “turbo” flag. It requires systematically dismantling five consecutive systems bottlenecks: memory bandwidth latency, token serialization, redundant prompt prefill, repetitive structure generation, and multi-GPU interconnect overhead.

Over the past several days on our research cluster, we ran over 1,000 controlled benchmark trials across 2×, 4×, and 8× NVIDIA B300 GPUs, exploring tensor parallelism, data parallelism, multi-token speculation (MTP4 vs. MTP8), prompt prefix caching, and suffix reuse. Below is the verified engineering telemetry and the practical playbook for scaling Qwen 3.8 27B in production.

1. The 104 tok/s Baseline: Why Raw Models Crawl

To understand how to make an LLM run at 10,000 tokens per second, you must first understand why it runs at 100 tokens per second out of the box.

During the autoregressive decode phase, an LLM generates one token at a time. To emit a single token for a 27B parameter model in FP8 precision (~27 gigabytes of weights), the GPU must stream those entire 27 GB from high-bandwidth memory (HBM) into its compute registers. Even with the formidable memory bandwidth of NVIDIA B300 silicon, doing this sequentially for a single stream caps your decode rate at roughly 100 to 135 tokens per second.

At concurrency 1, over 95% of the GPU’s Tensor Core compute capability sits completely idle. The execution units are starved, waiting on memory loads. Therefore, scaling inference throughput requires solving two orthogonal engineering goals:

1. For Low Concurrency (Interactive Chat & Agent Tooling): How do we minimize latency per token so a human or autonomous agent experiences instant responses?
2. For High Concurrency (Batch APIs & Synthetic Data Generation): How do we pack requests together so that each memory-load pass over the weights computes tokens for dozens of requests simultaneously?

2. The Speedup Journey: From 100 to 16,900+ tok/s

We took a disciplined, step-by-step approach. Each experiment introduced a single architectural intervention: slicing layers across NVLink, deploying speculative draft heads, caching repeated prompt prefixes, mining repetitive output structures, and scaling independent data-parallel replicas.

The interactive chart below traces the complete empirical trajectory of our tests on Qwen 3.8 27B across 2×, 4×, and 8× NVIDIA B300 GPUs. Click on any milestone node to see the hardware configuration, the technique applied, and the bottleneck eliminated:

Notice the distinct operational leaps along this curve:

• The Latency Leap (104 → 305 tok/s): Tensor Parallelism (TP2 and TP4) combined with MTP4 speculative decoding cut inter-token latency by nearly 3×, pushing solitary stream speed to 305 tok/s.
• The Concurrency Multiplier (305 → 4,464 tok/s): Bringing concurrency to 64 streams allowed continuous batching on 2 B300 GPUs to reach 4,464 tok/s with MTP4.
• The Caching & Suffix Breakthrough (4,464 → 7,012 tok/s): On structured catalogs, prefix caching eliminated 85% of input prefill, while suffix history predicted output sequences with 99.7% acceptance, breaching 7k tok/s on just two cards.
• The Multi-GPU Expansion (7,012 → 9,825 tok/s): Scaling to 4 and 8 B300 GPUs with a balanced hybrid topology (DP4×TP2) pushed our custom stack to 9,825 tok/s.
• The Peak Cluster Saturation (9,825 → 16,931 tok/s): At concurrency 256 with deep provider worker scaling on Fireworks, the cluster delivered an astonishing 16,930 tok/s.

3. Topology Dynamics: Tensor Parallelism vs. Data Parallelism

A frequent misconception in inference engineering is the dogma that “Data Parallelism is always best for throughput, and Tensor Parallelism is only for fitting big models in VRAM.”

With Qwen 3.8 27B FP8, the model easily fits onto a single 80GB B300. Yet when we benchmarked different allocations across 4 and 8 GPUs, the reality proved far more nuanced:

Key architectural principles from the topology sweeps:

• Low Concurrency (c = 1 to 4): Tensor Parallelism is mandatory if user-facing latency matters. DP1×TP4 delivered 304.8 tok/s at c1 on 4× B300, compared to 243.1 tok/s for DP4×TP1. Every token decode step is shared over high-speed NVLink.
• High Concurrency (c = 64 to 256): Data Parallelism dominates. At c=64 on 4 GPUs, DP4×TP1 delivered 6,048 tok/s, beating DP1×TP4 (4,454 tok/s) by +35.8%. Under heavy concurrency, TP’s repeated AllReduce synchronization across GPUs creates a barrier that serializes batch scheduling.
• The 8-GPU Limit (TP8): Never run TP8 across a full 8-GPU node for throughput. DP1×TP8 at c=64 delivered only 4,252 tok/s—slower than a 4-GPU system. The communication latency of 8-way AllReduce completely wiped out the compute gains.
• The Long-Context Sweet Spot (DP4×TP2): On long prompts (2.8k tokens), running 8 independent replicas (DP8×TP1) suffered from KV-cache memory pressure and scheduling contention. The balanced hybrid topology—four replicas of TP2 pairs (DP4×TP2)—won the overall crown at 9,825 tok/s.

4. Speculative Decoding: MTP4, MTP8, and the Hybrid Myth

Qwen 3.8 natively incorporates Multi-Token Prediction (MTP). Unlike traditional speculative decoding, which requires loading a separate smaller draft model (and paying memory bandwidth to read two sets of weights), MTP uses lightweight auxiliary prediction heads baked directly into the primary model.

In a single forward pass, the base model generates the target token, while the MTP heads propose k future candidate tokens. The subsequent forward pass validates all k candidates simultaneously in parallel:

We tested MTP with 4 draft heads (MTP4), 8 draft heads (MTP8), and a dynamic “Hybrid” engine that first attempts suffix matching before falling back to MTP:

The Lessons of Speculative Tuning

1. Why MTP4 beats MTP8 on reasoning: Reasoning workloads have high branch entropy. The model often changes course, backtracking or exploring logic. Under these dynamics, predicting 4 tokens ahead hit a healthy 59.1% acceptance rate. Pushing to 8 tokens (MTP8) caused acceptance to collapse to 38.4%. The compute spent generating and verifying rejected draft tokens 5 through 8 degraded throughput by over 200 tok/s at c=64.
2. When MTP8 wins: When sequences are highly structured or predictable (or when deep queue batching at c=256 keeps all workers saturated), MTP8 pulled ahead on 4 GPUs (8,239 tok/s vs 7,230 tok/s for MTP4).
3. The Hybrid Fallacy: In architecture discussions, engineers often propose: “Let us try suffix history first, and if that misses, fall back to MTP.” We built and measured this exact runtime. It underperformed across the board. Why? Maintaining MTP hidden states in lockstep while simultaneously executing unpadded suffix drafts introduced CPU-GPU synchronization bubbles. A clean, dedicated speculative pipeline consistently beat the complex hybrid.

5. Caching Strategies: Prefix Caching vs. Suffix Decoding vs. N-gram

In real-world applications, models rarely process isolated prompts. Enterprise workloads fall into two broad categories:

• Ad-hoc High-Variance Prompts: Unique user queries, diverse reasoning tasks, or creative writing.
• Structured Enterprise Prompts: System prompts, few-shot examples, JSON schemas, catalog extraction, and multi-turn agent tool loops.

We tested our dual-B300 stack with identical models under three caching regimes:

Key Architectural Findings

• Prefix Caching on Short Prompts is a Trap: If your prompts do not share a common prefix, enabling prefix caching provides 0.0% hit rate while adding hash computation and Radix-tree lookup latency. Throughput dropped from 2,925 to 2,871 tok/s at c=64. Do not turn prefix caching on blindly for randomized workloads.
• The 1.78× Prefill Unlock: On long documents sharing a common system prompt or catalog base (2,820 shared tokens), prefix caching was transformative. Throughput at c=64 jumped from 1,599 to 2,851 tok/s (+78.3%), while Time-To-First-Token dropped by over 60%.
• The Suffix 99.7% Acceptance Wonder: In repetitive workflows like catalog formatting or code generation, the model repeatedly outputs structured syntax. Suffix decoding stores and speculates matching n-grams from prompt and generation history. It achieved an astonishing 99.69% draft acceptance, catapulting long-context throughput to 5,705 tok/s at c=64 and 7,012 tok/s at c=256.

6. Scaling 4 vs. 8 B300 GPUs: Where Efficiency Bends

One of the most consequential findings for platform teams is the scaling relationship between 4 and 8 B300 GPUs. When you double your hardware budget from 4× to 8× B300 SXM6, do you get a 2× throughput increase?

The hard telemetry answer is no. Under identical workloads, software stacks, and fixed concurrency, the scaling ratio was surprisingly modest:

Why did doubling GPUs not double throughput? Three systems bottlenecks emerged:

1. Gateway and Reverse Proxy Saturation: At 8,000+ tokens per second, the HTTP gateways (LiteLLM / Envoy) and Python async event loops spend measurable CPU time serializing JSON chunks and parsing streaming SSE events.
2. Client Serialization: Even with an 8-CPU client node, receiving and verifying 256 concurrent SSE streams under AIPerf incurred client-side scheduling latency.
3. Queue Starvation at Fixed Concurrency: When 8 workers divide 256 concurrent requests, each worker averages only 32 active requests—not enough to keep all B300 compute units fully saturated.

7. Head-to-Head: Fireworks AI vs. g factor Stack

Because we benchmarked both our custom serving stack and Fireworks AI on confirmed, identical budgets of 2×, 4×, and 8× NVIDIA B300 SXM6 GPUs with the exact same Qwen 3.8 27B FP8 weights, we can draw a direct, empirical comparison between an optimized enterprise managed endpoint and a tailored open-engine deployment.

Neither system is uniformly faster across all workloads. Instead, the architectural trade-offs divide cleanly between interactive low-concurrency responsiveness and deep-queue batch saturation:

8. The Systems Architecture Playbook: What to Deploy When

Based on our verified findings, here are four concrete architectural rules for deploying Qwen 3.8 27B in production:

gft deploy cloud --model Qwen/Qwen3.8-27B-Instruct --dp 4 --tp 1 --speculation mtp4 --prefix-cache

All custom stack benchmarks conducted on dedicated NVIDIA B300 SXM6 AC nodes in Nebius uk-south1, September 20–21, 2026. Standardized AIPerf 0.12.0 test suites streaming over HTTPS and internal HTTP, evaluated across 3 independent repeats. Total measured requests exceeded 90,000 calls with zero failed completion validations.