Language models—computer programs that understand and generate human language—often rely on a type of memory called n-gram embeddings to improve their performance. These embeddings store information about short sequences of words or tokens, helping the model predict what comes next more accurately. However, this memory can be enormous, sometimes requiring billions of parameters and vast amounts of storage. A newly published research paper explores a clever way to shrink this memory footprint dramatically without sacrificing the model’s effectiveness.
Key Takeaways
- Using just 4-bit precision to store n-gram embeddings can reduce memory use by roughly 75%, while maintaining over 99% of the performance gains these embeddings provide.
- Lowering precision further (below 4 bits) causes noticeable drops in accuracy, but a brief fine-tuning process can help recover much of the lost performance.
- Memory saved by using low-bit embeddings can be reinvested to expand the size of embedding tables, resulting in even better model performance than the original full-precision tables.
- These benefits apply not just to newly trained models but also to large, existing language models, cutting their embedding memory by nearly half with minimal impact on downstream tasks.
The research focuses on the problem of memory consumption in language models that use n-gram embeddings. An n-gram is a short sequence of tokens—like pairs (bigrams) or triples (trigrams)—that models use to capture local language patterns. These embeddings are stored in large tables, which can require billions of parameters and substantial memory. For example, DeepSeek-V4.1-Flash, a state-of-the-art model, dedicates 196 billion parameters just to its n-gram memory.
To tackle this, the researchers propose representing the n-gram embeddings using low-bit precision, specifically quantizing the data to fewer bits per value. Quantization is a technique that reduces the amount of memory needed by storing numbers with less detail—think of it as rounding values to fewer decimal places but in binary form. Instead of the usual 32-bit floating-point numbers, the study explores using 4-bit and fewer bits to represent these embeddings.
They tested their approach on several language models from scratch, including nanochat-, Llama-, and GPT-2-family models. The results showed that 4-bit tables slashed memory use by about 75%, yet preserved more than 99% of the improvement in validation loss (a key measure of model accuracy) that n-gram embeddings provide. When they pushed the precision below 4 bits, performance dropped more noticeably, but a short phase of fine-tuning—retraining the model slightly with the low-bit embeddings—helped regain much of the lost accuracy.
Interestingly, the memory saved by using low-bit embeddings isn’t just about using less storage. The researchers found that this saved capacity could be reinvested to create larger embedding tables. Bigger tables can store more patterns and context, which in turn can boost the model’s performance beyond what the original full-precision tables achieved, all while still using less memory overall.
Importantly, the study’s findings also apply to large, already trained language models. When they applied low-bit n-gram embeddings to models like DeepSeek-V4.1-Flash and Qwen3.8-Flash-Next, they achieved 47-55% reductions in embedding memory with only minor increases in perplexity—a measure of how well a model predicts text—and largely maintained performance on downstream tasks such as translation or question answering.
This research suggests a practical path forward for making language models more efficient, especially as models continue to grow in size and complexity. By adopting low-bit n-gram embeddings, developers can reduce memory requirements significantly, which can lower costs and energy consumption. Additionally, the ability to reinvest saved memory into larger embedding tables opens up new opportunities to improve model accuracy without additional hardware demands.
Looking ahead, these findings could influence how future language models are designed and deployed, particularly in environments where memory is limited, such as mobile devices or edge computing. Further research might explore optimizing quantization techniques or combining low-bit embeddings with other model compression methods to push efficiency even further.
Based on research published on arXiv by Stephen Chung, Wenyu Du.