Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request
| Authors | Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia et al. |
|---|---|
| Primary category | |
| Published | 20 October 2026 |
| Last updated at source | — |
Silytics shows public metadata from arXiv and links to the original. Information may be incomplete or out of date. Last refreshed 21 October 2026. Licenses and terms of the original apply to the work itself. Not financial, legal or professional advice.