Benchmarking Qwen-27B Across Inference Providers: Together, Fireworks, Nebius, and Our Sovereign Stack

If you browse AI tech Twitter or vendor landing pages, every inference provider claims to be “the fastest engine on Earth.” You see sleek bar charts boasting 2,000 tokens per second, sub-10ms Time-To-First-Token, and 10x cost savings.

Then you deploy a 27-billion parameter reasoning model like Qwen3.8-27B into production, point real multi-turn traffic at those endpoints, and reality hits you like a cold splash of water. Some providers choke on long system prompts. Others quietly blow past sequence budgets because their API proxies fail to cap internal reasoning tokens. And when you try to reproduce a vendor’s headline benchmark on your own hardware, you discover their numbers relied on single-instance tensor parallelism over custom NVLink switches that don’t exist across standard cloud instances.

Over the past month, we ran exhaustive, rigorous benchmarking campaigns across our research platform (gft-studio) to evaluate Qwen3.8-27B across dedicated infrastructure and leading inference providers: Together AI, Fireworks AI, Nebius, Doubleword, and our own sovereign g factor inference stack.

Below, we share the unvarnished engineering telemetry: what happened when we compared like-for-like 2× H100 hardware, the staggering throughput delta between Tensor Parallelism (TP) and Data Parallelism (DP), the bizarre “15,000-token reasoning trap” that derailed our Doubleword tests, and the exact architectural tricks needed to maximize throughput on your own GPU nodes.

1. The Experimental Setup: Isolating Real Performance

To make an inference benchmark meaningful, you must eliminate confounding variables. Comparing a 7B model on FP8 with a 70B model on BF16 tells you nothing. Comparing an API called from a laptop in London with a server hosted in Oregon tells you nothing about engine speed.

Here is how we standardized our test harness:

• The Model: Qwen/Qwen3.8-27B (and its official FP8 quantized variant), pinned to tokenizer revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0.
• The Workload: Standardized AIPerf 0.12.0 test suites streaming over public HTTPS. Prompts averaged ~564 input tokens, requesting exactly 128 generated output tokensat temperature 0 and random seed 42.
• Concurrencies Tested: Concurrency levels 1, 4, and 8. Each cell ran 16 warmup requests followed by 60 strictly measured, validated requests across three separate repetitions (180 measured requests per cell).
• The Hardware Baseline: A fixed hardware budget of 2× NVIDIA H100 80GB SXM GPUs per target (except where exploring next-generation B200 accelerators).

2. The 2× H100 Provider Shootout

Let us start with the headline numbers. Below is the verified empirical telemetry from our September 2026 runs, comparing dedicated endpoints across Fireworks AI, Together AI, and our own sovereign g factor stack running on Nebius infrastructure:

At first glance, you might look at Together’s 978.7 tok/s and assume their software engine is twice as fast as everyone else’s. But when you look under the hood at the systems architecture, you realize the truth is much more nuanced.

3. The Topology Trap: Tensor Parallelism vs. Independent Replicas

The most important lesson from our benchmark is that two GPUs do not make a system; how those two GPUs are connected makes the system.

Why Together Won Single-Stream Latency: High-Speed NVLink

Look at the concurrency=1 row: Together achieved 191.2 tok/s and an Inter-Token Latency (ITL) of just 3.9 milliseconds! Everyone else was hovering around 70–97 tok/s and 7–14 ms.

Why? Because Together used Tensor Parallelism (TP2) inside a single server. In TP2, every linear layer in Qwen-27B is sliced in half: GPU 0 computes the top half of the matrix multiply, GPU 1 computes the bottom half, and they exchange intermediate activations via an all-reduce collective. At concurrency 1, both H100s are working on that single user’s sentence simultaneously!

By contrast, in Data Parallelism (DP2), GPU 0 handles User A while GPU 1 handles User B. At concurrency 1, User A only has access to one GPU. Their decode speed is physically bounded by that single chip’s memory bandwidth (3.35 TB/s on H100).

The Disaster of Cross-Node Tensor Parallelism

Seeing Together’s TP2 result, our team ran an experiment: what if we configure our custom vLLM stack as a TP2 worker across two separate 1× H100 cloud nodes on Nebius?

The result was an unmitigated disaster:

At concurrency 8, our cross-node TP2 throughput collapsed from 483 tok/s down to 75.75 tok/s! Why did this happen?

During single-token decoding, a 27B model’s GEMV math finishes in just 5 to 10 microseconds. Inside a single server connected by NVLink, exchanging activations takes ~2 microseconds over 900 GB/s links. Across separate physical servers over standard VPC networking, that same transfer takes 800 to 2,000 microseconds! The GPUs spent 99% of their time stalled at network barriers waiting for TCP packets.

4. The Hardware Leap: What Happens on NVIDIA B200?

While H100 remains the workhorse of enterprise inference today, next-generation NVIDIA Blackwell (B200) chips are entering production. We benchmarked Fireworks AI running Qwen3.8-27B on a dedicated 2× B200 deployment:

Moving from H100 to B200 delivers an immediate 1.6x to 1.7x throughput increase with zero code changes. This speedup is directly explained by physical hardware specifications:

• NVIDIA H100 SXM features 3.35 TB/s of HBM3 memory bandwidth.
• NVIDIA B200 features 8.00 TB/s of ultra-dense HBM3e bandwidth (a 2.38x hardware leap).

Because memory-bound autoregressive decoding scales almost linearly with memory bandwidth, B200 allows a 2-GPU cluster to cross the 1,000 tok/s barrier even in unquantized 16-bit precision.

5. War Story: The 15,000-Token Reasoning Trap

During our evaluation of Doubleword’s public API for Qwen3.8-27B-FP8, we hit one of the most bizarre and instructive bugs in modern LLM benchmarking.

We configured our AIPerf test with standard parameters:
{"max_completion_tokens": 128, "temperature": 0}

We expected every request to complete in about 2 seconds, generating 128 tokens. Instead, the client hung. Requests took over 4 minutes (247 seconds) to finish! When the payloads finally returned, we inspected the token counters:

Why Did This Happen?

Tracing the HTTP response headers revealed an openrouter.ai cookie domain and gateway hop. In many recent reasoning models (such as DeepSeek-R1 and Qwen reasoning variants), the upstream API treats max_tokens as applying only to the final answer string, leaving internal reasoning tokens (<think>...</think>) completely uncapped!

Because the gateway didn’t pass an explicit reasoning budget, the model spent four minutes generating 15,700 tokens of internal soliloquy before writing a 2-sentence response.

The Fix: Disabling reasoning explicitly for standardized speed testing:

With reasoning disabled, Doubleword’s public API completed all 540 requests cleanly without a single overrun:

• Concurrency 1: 44.18 tok/s (TTFT p50: 708 ms, ITL: 12.4 ms)
• Concurrency 4: 116.36 tok/s (TTFT p50: 725 ms, ITL: 12.5 ms)
• Concurrency 8: 240.98 tok/s (TTFT p50: 708 ms, ITL: 12.7 ms)

6. Optimizing Our Sovereign Stack: Beating Vanilla vLLM by 6.9%

When deploying inside an enterprise private cloud (VPC) on platforms like Nebius, you have two choices: run a raw open-source vLLM container, or engineer a managed sovereign inference stack.

In our head-to-head comparison on identical 2× H100 FP8 hardware, our custom stack (LiteLLM Gateway → NVIDIA Dynamo Routing Frontend → vLLM Workers) consistently beat vanilla vLLM endpoints:

• Concurrency 1: 73.71 tok/s vs. 68.94 tok/s (+6.9% higher throughput)
• Concurrency 4: 274.64 tok/s vs. 259.42 tok/s (+5.9% higher throughput)
• Concurrency 8: 483.19 tok/s vs. 463.44 tok/s (+4.3% higher throughput)

What Drove the Win?

1. In-Cluster Client Ingress: By running the routing gateway inside the Kubernetes cluster, p50 TTFT dropped from 230 ms down to 100 ms, eliminating public internet TLS handshake latency.
2. Dynamo Queue-Depth Routing: Standard round-robin blindly sends requests to workers regardless of whether a worker is busy processing an 8,000-token prompt. Dynamo routes to the worker with the lowest active KV-cache load.
3. MTP (Multi-Token Prediction) Speculative Decoding: In our optimized profile, enabling native MTP speculative decoding pushed throughput at concurrency 8 from 483.2 tok/s up to 613.5 tok/s—matching Fireworks BF16 on raw speed while maintaining full data sovereignty.

7. The Engineer’s Checklist for Interpreting AI Benchmarks

The next time a vendor shows you an inference throughput chart, ask these five questions:

1. What was the GPU topology? Is that 1,000 tok/s running on a single server with NVLink (TP2), or across independent instances (DP2)?
2. What precision was actually served? Was it BF16, online FP8, or aggressive 4-bit quantization? Did anyone evaluate accuracy degradation on domain tasks?
3. Where was the client running? Were requests sent over the public internet through Cloudflare edges, or over internal loopback interfaces?
4. How were reasoning tokens counted? Did the API cap thinking tokens, or did an unbounded reasoning loop inflate the numerator?
5. Was the model cold or warmed up? Did the benchmark include CUDA Graph capture, Inductor compilation, and DeepGEMM warmup, or was it measured only during an idealized steady state?