vLLM introduces hardware-agnostic layers for portability

To keep pace with frontier AI models like DeepSeek V4 and Kimi K3, the vLLM serving framework is shifting toward hardware-specific flat model definitions. While this shift optimizes performance for cutting-edge hardware like NVIDIA Blackwell GPUs and GB300 NVL72 rack systems, it breaks compatibility with fullgraph torch.compile. To prevent performance regressions on older GPUs, consumer hardware, and out-of-tree accelerators, vLLM developers are introducing a dedicated suite of hardware-agnostic layers.

These new layers reside in a separate directory, isolating them from hardware-specific code paths. They are designed to be fully compilable, extensible, isolated, and portable, relying on native PyTorch or portable domain-specific languages like Triton and Helion. In initial testing on NVIDIA H100 GPUs, the hardware-agnostic layers achieved a total token throughput within 3.4 percent of the native implementation, calculated as a geometric mean across three recent models.

For practitioners, this architecture preserves the ability to run diverse models on alternative hardware like IBM Spyre, AMD GPUs, Intel XPUs, Google TPUs, and Huawei Ascend. Developers can already test this pathway with the transformers backend by setting the environment variable USE_HW_AGNOSTIC=1. The team has validated this setup using the Spyre out-of-tree plugin for models such as Gemma 4, Qwen3, and Granite 4.2. Additionally, a hardware-agnostic pull request for DeepSeek V4 is currently under review.

Ultimately, this dual-track system allows vLLM to push the boundaries of hardware-specific kernel engineering on next-generation chips while maintaining a stable, high-performance fallback for the broader open-source ecosystem. By keeping the hardware-agnostic layers decoupled, developers can optimize for specific platforms without breaking compatibility for others.