As artificial intelligence continues to evolve, large language models (LLMs) are becoming more powerful but also more demanding on computing resources. A newly published research paper introduces LeapQuant, a novel technique designed to speed up these models while maintaining their accuracy. This breakthrough could help AI systems process long texts faster and more efficiently, making advanced language technologies more accessible and practical.
Key Takeaways
- LeapQuant reduces the computational cost of processing long text sequences in AI models by efficiently compressing and quantizing the model’s memory state.
- It uses a new “per-window quantization” approach that updates the model’s internal state less frequently, minimizing error accumulation.
- By preserving key “outlier” data points as high-precision tokens, LeapQuant maintains accuracy close to full-precision models while using 8-bit quantization.
- The method achieves speed improvements of up to 3.7 times at the core computation level and about 1.5 times faster end-to-end on modern GPUs.
Modern language models rely on a mechanism called attention to understand and generate text. Traditional attention methods can be slow and require a lot of memory, especially when dealing with long documents. To address this, some models use “linear attention” techniques that compress the context into a fixed-size “recurrent state,” allowing them to handle longer inputs more efficiently. However, updating and reading this recurrent state repeatedly during inference (when the model generates text) can still be a bottleneck.
The research team behind LeapQuant tackled this problem by applying quantization—a process that reduces the precision of numbers to speed up computation and shrink memory use. While quantization is common in AI, it often leads to a drop in model quality because rounding errors can build up, and certain unusual data points (“outliers”) can skew results. LeapQuant introduces two key innovations to overcome these challenges.
First, instead of quantizing the recurrent state after every token, LeapQuant uses “per-window quantization.” This means the model processes a group or “window” of tokens with a fixed low-precision state and only updates and quantizes the state once at the end of the window. This approach significantly reduces the accumulation of rounding errors that typically degrade performance.
Second, LeapQuant identifies the largest outliers in the recurrent state and keeps them as special “Compensator Tokens” in high precision. These tokens share the same update path as real tokens, helping preserve important information that might otherwise be lost during quantization. Additionally, the method smooths the residual (the remaining data after removing outliers) before quantizing it, further reducing error.
The researchers tested LeapQuant extensively on several leading model families, including Qwen, Kimi, and GLM. Their results showed that LeapQuant maintains accuracy very close to full 32-bit precision models while cutting down memory usage and computation time. On popular GPUs like NVIDIA’s B200, RTX PRO 6000, and RTX 5090, LeapQuant achieved kernel-level speedups ranging from about 2 to nearly 4 times, and overall inference speedups of around 1.5 times.
These improvements could make AI language models more practical for real-world applications where speed and resource efficiency are critical, such as real-time translation, chatbots, and large-scale text analysis. By enabling faster processing without sacrificing quality, LeapQuant may help broaden access to advanced AI capabilities and reduce the environmental impact of running large models. Future work could explore integrating LeapQuant with other model architectures and further optimizing hardware compatibility to maximize its benefits.
Based on research published on arXiv by Yi Pan, Haocheng Xi, Kan Zhu et al..