Cap-and-Track: A Smarter Way to Manage AI Workloads as Demand Fluctuates

Photo of author

By Sophia Chen

As large language models (LLMs) become more widely used for everything from chatbots to video analysis, efficiently handling the massive and unpredictable flow of user requests is a growing challenge. A newly published research paper introduces “Cap-and-Track,” a novel system designed to optimize how AI inference tasks are dispatched across computing resources that can change in capacity over time. This approach aims to reduce wasted computing power and improve responsiveness without needing complex predictions about how long each request will take—an important step for scaling AI services reliably and cost-effectively.

Key Takeaways

  • Cap-and-Track dynamically adjusts how AI inference requests are sent to servers, reacting in real time to shifts in available computing capacity and varying request sizes.
  • The system uses a “cap” to quickly refuse new work when resources are saturated, preventing long wait times and backlog buildup.
  • It “tracks” the capacity ceiling by monitoring refusals, allowing it to probe for available resources and back off when necessary without needing to predict output lengths.
  • In production tests at Meta, Cap-and-Track reduced the number of GPUs needed by 43% compared to a fixed-rate dispatch system, while maintaining similar throughput and handling over a billion requests in a day.

Traditional methods for managing AI workloads often rely on fixed dispatch rates—sending requests at a constant rate regardless of how busy the servers are or how complex each task might be. This can lead to inefficiencies: if too many requests arrive when capacity is low, queues grow long and latency increases; if capacity is high but dispatch rates are low, resources go underutilized. Moreover, the length of generative outputs (like how many words a language model will produce) is unknown when a request arrives, making it hard to predict processing time.

Cap-and-Track addresses these challenges by combining two key ideas. First, the “cap” acts as a fast admission control that immediately refuses new requests when a critical resource is fully saturated. This prevents the system from accepting more work than it can handle, avoiding long delays. Second, the “track” component uses these refusal signals to estimate the current capacity limit. It then adjusts the dispatch rate dynamically—probing for more capacity when refusals are low and backing off when refusals increase.

Importantly, this approach does not require predicting how long each request will take or setting a fixed rate for the entire fleet of servers. Instead, it relies solely on real-time feedback from refusals to guide dispatching decisions. Requests that are refused are placed in durable priority queues outside of execution, allowing lower-priority work to wait longer when needed without blocking higher-priority tasks. This stateful tracking helps the system adapt smoothly to changing conditions such as servers going offline or request sizes fluctuating.

The research team tested Cap-and-Track in production at Meta, implementing it in a traffic shaper that manages inference for tasks including post-processing and live video understanding. Compared to a traditional fixed-rate dispatch system, Cap-and-Track served the same classifier using 43% fewer high-end GPUs (NVIDIA H100s), demonstrating significant resource savings. During peak backfill periods, the system handled an astonishing 1.1 billion requests in 24 hours, processing over 300 million online requests daily with hundreds of terabytes of media.

These results suggest that Cap-and-Track could help AI service providers better match computing resources to demand in real time, improving efficiency and reducing costs. By keeping queues just long enough to efficiently fill processing batches and steering dispatch purely by capacity refusals, the system maintains high throughput without overloading servers. Going forward, this approach may be particularly valuable as AI workloads continue to grow and fluctuate unpredictably, enabling more sustainable scaling of large language models and other generative AI applications.

Based on research published on arXiv by Benjamin Novotny, Doug Szeto, Chang Wang et al..

Editor's note

This report is framed around the immediate news and the wider implications for regulators, companies and users following the story.

Article briefing

As large language models (LLMs) become more widely used for everything from chatbots to video analysis, efficiently handling the massive and unpredictable flow of user...

Story details

  • Author: Sophia Chen
  • Published: November 19, 2026
  • Category: AI

Key developments

  • As large language models (LLMs) become more widely used for everything from chatbots to video analysis, efficiently handling the massive and unpredictable flow of user requests is a growing challenge.
  • A newly published research paper introduces "Cap-and-Track," a novel system designed to optimize how AI inference tasks are dispatched across computing resources that can change in capacity over time.
  • Traditional methods for managing AI workloads often rely on fixed dispatch rates—sending requests at a constant rate regardless of how busy the servers are or how complex each task might be.

Why this matters

This prevents the system from accepting more work than it can handle, avoiding long delays.

Impact and next steps

Moreover, the length of generative outputs (like how many words a language model will produce) is unknown when a request arrives, making it hard to predict processing time.

Source

This article is based on source material from arxiv.org.

About the author

Sophia Chen

Sophia Chen covers artificial intelligence and emerging technology. With a background in computer science and a decade of tech journalism, she specialises in AI policy, machine learning applications and the societal impact of automation.

editorial@peacknews.com

Categories AI