Researchers have developed a new method to shrink the memory and speed up large language models (LLMs) by using ultra-low 4-bit precision across all parts of the model, including its tricky recurrent components. This breakthrough helps make massive AI models more practical to run on everyday hardware without sacrificing their performance, potentially bringing advanced language AI closer to wider use.
Key Takeaways
- The team applied 4-bit quantization to all 496 linear layers of a 27-billion-parameter hybrid LLM, including the recurrent Gated DeltaNet (GDN) layers previously kept at higher precision.
- The resulting model, named Minima, matches the accuracy of 16-bit (BF16) versions across multiple benchmarks and long context lengths, while using less memory and running faster.
- A detailed analysis reveals why the GDN’s recurrent layers tolerate aggressive quantization without error buildup, thanks to their internal gating mechanisms and the way state updates overwrite previous information.
- The researchers also fixed technical issues related to combining quantized modules, enabling seamless deployment with existing optimized kernels and efficient memory caching.
Large language models like the one studied here—called Qwen3.8-27B—combine different types of attention mechanisms to process and generate text. One part, the Gated DeltaNet (GDN), uses a recurrent approach, meaning it updates a summarized state as it reads through a long context. This recurrent nature made experts cautious about applying aggressive quantization (reducing the precision of numbers to save memory and computation) to GDN layers, fearing that small errors would accumulate and degrade the model’s output.
Quantization reduces the number of bits used to represent each number in a model’s parameters and activations. Lower bits mean smaller model size and faster computations, but potentially more rounding errors. Previous attempts often left the GDN layers at 8- or 16-bit precision to avoid error accumulation. The new research challenges this assumption by fully quantizing the entire model—including GDN layers—to 4-bit precision using a novel format called NVFP4.
The NVFP4 format groups 16 numbers together and applies scaling to each group, which helps contain extreme values that could cause large errors. The researchers found that the gating functions inside GDN layers—mathematical operations controlling how new information is written or forgotten—are surprisingly robust to quantization noise. In fact, the nonlinear functions involved compress the errors, preventing them from growing through the recurrent updates.
Additionally, the way GDN updates its internal state means that each new write effectively overwrites older information along specific “directions,” limiting how errors propagate over thousands of tokens. This contrasts with many recurrent systems where errors can compound unchecked. The team also observed that while quantization noise appears per token, its impact averages out over long contexts rather than snowballing.
To test their approach, the researchers evaluated Minima on a wide range of language understanding and reasoning benchmarks, as well as perplexity (a measure of how well the model predicts text) over very long sequences—up to 32,000 tokens. Minima’s performance closely matched that of the higher-precision baseline, with only minor differences within the range of random variation between runs.
They also addressed a practical challenge: when quantized modules are fused into larger computation blocks (GEMMs) for efficiency, scale mismatches can occur. By carefully calibrating and shipping key-value cache scales, they ensured that the quantized model runs efficiently without performance penalties.
This work offers a practical recipe for aggressively compressing large hybrid LLMs, making them more accessible for deployment in resource-constrained environments. By showing that the recurrent half of such models is easier to quantize than previously thought, it opens the door to further innovations in efficient AI model design.
Looking ahead, these findings could help reduce the hardware requirements for next-generation language models, enabling faster, cheaper, and more energy-efficient AI applications. Future research may explore applying similar quantization strategies to other model architectures and further optimizing the trade-offs between precision, speed, and accuracy.
Based on research published on arXiv by Sergii Kozyrev, Davyd Maiboroda.