The burgeoning demand for large language model (LLM) inference is pushing the boundaries of computing infrastructure, creating a bottleneck where supply struggles to keep pace. As frontier models like Claude Fable, GLM5.2, and Minimax M3 become ubiquitous, the cost of running inference on high-end GPUs, particularly NVIDIA's dominant Blackwell series, continues to climb. In this context, alternative hardware solutions offering compelling price-performance ratios are becoming increasingly critical for developers and enterprises.
What Happened
AI infrastructure provider Wafer recently announced a significant breakthrough in LLM inference performance on AMD's Instinct MI355X GPUs. The company achieved an aggregate throughput of 2626 tokens/second/node when running the GLM5.2 model on a specific workload (20k input tokens, 1k output tokens, 60% cache hit rate, 2.4 requests per second saturation). This performance, while about 80% of what was measured on an NVIDIA B200, comes with a substantial cost advantage: the MI355X is approximately 2.75 times cheaper per GPU on average than the NVIDIA B300, translating to an overall cost reduction of over 2x for comparable performance per dollar.
Wafer's results underscore that while NVIDIA typically offers superior day-0 software support, dedicated engineering effort can bridge this gap on the AMD ROCm stack. Achieving this required several key optimizations:
- Quantization: The base bf16 GLM-5.2 model was quantized to MXFP4 using AMD Quark. This MXFP4 quantization was found to be lossless compared to an official FP8 baseline, as demonstrated by evaluation metrics:
| Eval | FP8 baseline | MXFP4 | Δ (MXFP4 − FP8) |
|---|---|---|---|
| GSM8K (200q, 5-shot, greedy) | 0.965 ± 0.013 | 0.955 ± 0.014 | −0.010 |
| GPQA-Diamond (198q × 2 seeds, temp 1.0) | 0.9217 ± 0.027 | 0.9026 ± 0.029 | −0.019 |
| tau2 macro | 0.819 | 0.834 | +0.015 |
-
Inference Framework: Among options like vLLM, ATOM, and sglang, sglang was chosen. vLLM lacked a working MXFP4 + GlmMoeDsa path, preventing it from utilizing the quantized weights, while ATOM exhibited output degradation with long contexts. Sglang proved to be the most adaptable engine for native support and effective quantization leverage.
-
Speculative Decode Optimization: A critical step for improving throughput was enabling speculative decode in sglang. However, the default sglang ROCm image did not support this out-of-the-box. Wafer performed specific fixes, addressing how the MTP (Multi-Teacher-Prediction) head handles shared experts, which were stored in bf16 rather than MXFP4.
This optimized setup delivered robust performance metrics, with 100% success rates even at saturation:
| Sustained RPS | Aggregate tok/s/node | TTFT p50 / p95 | Success |
|---|---|---|---|
| 0.5 | 449 | 0.59s / 0.60s | 100% |
| 1.0 | 974 | 0.60s / 0.81s | 100% |
| 1.5 | 1913 | 0.62s / 1.03s | 100% |
| 2.0 | 1944 | 0.62s / 1.05s | 100% |
| 2.25 | 2089 | 0.63s / 1.23s | 100% |
| 2.4 (saturation) | 2626 | 0.81s / 2.22s | 100% |
Separately, Wafer also reported 213 tokens/second on a single stream GLM5.2 workload (10k input / 1.5k output) on the MI355X, adhering to Artificial Analysis standards. While not topping the raw performance leaderboard, this figure also demonstrated superior performance per dollar.
Why It Matters
This benchmark by Wafer is highly significant for the AI and IT landscape for several reasons:
- Cost Efficiency for Enterprises: For organizations deploying LLMs at scale, the operational expenditure (OpEx) for inference can be staggering. A 2x cost reduction directly impacts the bottom line, making advanced AI capabilities more accessible and sustainable. This could enable smaller companies or those with tighter budgets to leverage powerful models that were previously cost-prohibitive.
- Increased Hardware Competition: NVIDIA has long held a dominant position in the AI hardware market, largely due to its CUDA software ecosystem. AMD's ability to demonstrate competitive performance-per-dollar, even if requiring more upfront engineering, provides a credible alternative. This increased competition can drive innovation across the board and potentially lead to more stable or lower GPU prices in the long term, benefiting the entire industry.
- Software Ecosystem Maturity: While AMD's ROCm stack has historically lagged behind CUDA in terms of "day-0" support for cutting-edge AI models, Wafer's work highlights the rapid advancements being made. It proves that with targeted optimization and a deeper understanding of the hardware and software layers, high-performance inference is achievable on AMD GPUs. This could encourage more developers and third-party vendors to invest in and contribute to the ROCm ecosystem, accelerating its maturation.
- Developer Empowerment: For developers working on LLM deployments, this emphasizes the critical role of software optimization, beyond just raw hardware specifications. Techniques like advanced quantization (MXFP4), selecting the right inference framework (sglang), and custom kernel/library fixes are paramount for extracting maximum performance and cost efficiency from available hardware, regardless of vendor.
What To Watch
The trend of achieving superior performance per dollar on AMD hardware through diligent software optimization is a key development to monitor:
- ROCm Ecosystem Growth: Will we see more widespread out-of-the-box support for frontier models within the ROCm stack, reducing the need for extensive custom engineering? The pace at which these gaps close will determine how quickly AMD can capture more of the AI inference market.
- Diversification of AI Infrastructure: Will major cloud providers and enterprises begin to diversify their AI inference infrastructure to include more AMD-based solutions, driven by these cost efficiencies? This could profoundly shift supply chain dynamics.
- Further Benchmarks and Optimizations: Expect more benchmarks from third parties pushing the limits of AMD's MI300 series. The ongoing "token craze" will only intensify the search for cost-effective inference solutions, leading to continuous innovation in quantization, speculative decoding, and inference engine technologies.
- Competitive Responses: How will NVIDIA respond to these challenges? We may see continued focus on their own software optimizations, new hardware iterations, or even shifts in pricing strategies to maintain their market lead.
The race for efficient and affordable AI inference is far from over. Wafer's results with AMD MI355X are a strong signal that the competitive landscape is evolving rapidly, presenting exciting opportunities for developers and significant cost savings for businesses.