Researchers have developed a novel technique that makes transformer models—the backbone of many AI systems—more adaptable to different hardware and workload demands without needing to retrain them. This approach, described in a newly published paper, allows a single pre-trained transformer to adjust its size dynamically, balancing speed and accuracy depending on the available computational resources. This breakthrough could help AI applications run more efficiently on a variety of devices, from powerful servers to edge gadgets, while maintaining strong performance.
Key Takeaways
- The new method “elastifies” pre-trained transformers, enabling them to scale smoothly across a wide range of computational budgets without retraining.
- It ranks individual components inside the model—like neurons and attention heads—using a differentiable relaxation technique, allowing fine-grained pruning.
- Tests on 27 transformer models (22 million to 8 billion parameters) show graceful performance drops even when removing up to 60% of the model’s parameters.
- For example, pruning half the parameters in a vision transformer reduced accuracy by less than 1% but significantly improved processing speed and lowered latency on GPUs.
Transformers are a class of AI models widely used for tasks involving language, images, and more. They are typically offered in fixed sizes—small, base, large—forcing users to pick one that fits their hardware capabilities and speed needs. However, real-world demands fluctuate: sometimes fast responses are critical, while other times maximum accuracy is preferred. The new method tackles this by making a single transformer model “elastic,” meaning it can adapt its size continuously depending on the current computational budget.
At the core of this innovation is a process called “differentiable relaxation” applied to the problem of ranking the importance of different parts of the model. Instead of treating entire blocks or layers as units, the method evaluates every individual neuron in the multilayer perceptrons (MLPs) and every attention head on a shared scale. This fine-grained ranking is achieved by optimizing the model’s loss function under many levels of “soft pruning” simultaneously. Soft pruning means gradually reducing the influence of less important components rather than abruptly removing them, which helps maintain performance.
What makes this approach stand out is that it requires no retraining or weight correction after pruning. The elastic transformer is produced quickly—in minutes—and scales efficiently even for very large models with billions of parameters. For example, when applied to the DINOv3 ViT-H+/16 vision transformer, removing half the parameters resulted in only a 0.8% drop in accuracy on the ImageNet benchmark, while massively reducing the model’s size. Similarly, a pruned AugReg ViT-B/16 model achieved up to 1.8 times faster throughput and 1.26 times lower latency for individual queries on an Nvidia A100 GPU.
This research opens up exciting possibilities for deploying powerful transformer models in environments with variable hardware or fluctuating workloads, such as cloud servers handling many requests or mobile devices with limited power. By allowing models to flexibly trade off accuracy for speed without retraining, developers can optimize AI applications more efficiently. Future work may explore extending this elastic approach to other model architectures and real-time adaptation during live operation.
Based on research published on arXiv by Walter Simoncini, Dheeraj Varghese, Gertjan J. Burghouts et al..