As large language models (LLMs) become more widely used for everything from chatbots to video analysis, efficiently handling the massive and unpredictable flow of user requests is a growing challenge. A newly published research paper introduces “Cap-and-Track,” a novel system designed to optimize how AI inference tasks are dispatched across computing resources that can change in capacity over time. This approach aims to reduce wasted computing power and improve responsiveness without needing complex predictions about how long each request will take—an important step for scaling AI services reliably and cost-effectively.
Key Takeaways
- Cap-and-Track dynamically adjusts how AI inference requests are sent to servers, reacting in real time to shifts in available computing capacity and varying request sizes.
- The system uses a “cap” to quickly refuse new work when resources are saturated, preventing long wait times and backlog buildup.
- It “tracks” the capacity ceiling by monitoring refusals, allowing it to probe for available resources and back off when necessary without needing to predict output lengths.
- In production tests at Meta, Cap-and-Track reduced the number of GPUs needed by 43% compared to a fixed-rate dispatch system, while maintaining similar throughput and handling over a billion requests in a day.
Traditional methods for managing AI workloads often rely on fixed dispatch rates—sending requests at a constant rate regardless of how busy the servers are or how complex each task might be. This can lead to inefficiencies: if too many requests arrive when capacity is low, queues grow long and latency increases; if capacity is high but dispatch rates are low, resources go underutilized. Moreover, the length of generative outputs (like how many words a language model will produce) is unknown when a request arrives, making it hard to predict processing time.
Cap-and-Track addresses these challenges by combining two key ideas. First, the “cap” acts as a fast admission control that immediately refuses new requests when a critical resource is fully saturated. This prevents the system from accepting more work than it can handle, avoiding long delays. Second, the “track” component uses these refusal signals to estimate the current capacity limit. It then adjusts the dispatch rate dynamically—probing for more capacity when refusals are low and backing off when refusals increase.
Importantly, this approach does not require predicting how long each request will take or setting a fixed rate for the entire fleet of servers. Instead, it relies solely on real-time feedback from refusals to guide dispatching decisions. Requests that are refused are placed in durable priority queues outside of execution, allowing lower-priority work to wait longer when needed without blocking higher-priority tasks. This stateful tracking helps the system adapt smoothly to changing conditions such as servers going offline or request sizes fluctuating.
The research team tested Cap-and-Track in production at Meta, implementing it in a traffic shaper that manages inference for tasks including post-processing and live video understanding. Compared to a traditional fixed-rate dispatch system, Cap-and-Track served the same classifier using 43% fewer high-end GPUs (NVIDIA H100s), demonstrating significant resource savings. During peak backfill periods, the system handled an astonishing 1.1 billion requests in 24 hours, processing over 300 million online requests daily with hundreds of terabytes of media.
These results suggest that Cap-and-Track could help AI service providers better match computing resources to demand in real time, improving efficiency and reducing costs. By keeping queues just long enough to efficiently fill processing batches and steering dispatch purely by capacity refusals, the system maintains high throughput without overloading servers. Going forward, this approach may be particularly valuable as AI workloads continue to grow and fluctuate unpredictably, enabling more sustainable scaling of large language models and other generative AI applications.
Based on research published on arXiv by Benjamin Novotny, Doug Szeto, Chang Wang et al..
