When IT support tickets are automatically routed to the wrong team, it can lead to wasted time and increased costs—sometimes even more than if a human had handled the triage. A newly published research paper explores how large language models (LLMs) and other AI classifiers can be taught not only to make predictions but also to “abstain” or defer decisions when they’re unsure, reducing costly errors in IT ticket classification.
Key Takeaways
- Introducing cost-aware abstention—where AI systems can choose to defer uncertain cases—cuts wrong routing errors from 15.3% to 3.5% in a large IT ticket dataset.
- A simple, cost-conditional thresholding approach using traditional TF-IDF features with logistic regression reduced average costs by 38.1% compared to always making a routing decision.
- Large language models with zero- or few-shot prompting performed worse than fine-tuned models like DistilBERT on this task, highlighting challenges in applying LLMs directly for ticket routing.
- Calibration methods that adjust AI confidence scores improved reliability but did not always enhance the model’s ability to distinguish correct from incorrect predictions.
The research focuses on “cost-aware abstention,” a strategy that allows AI systems to assess their own confidence and decide whether to route an IT ticket automatically or defer it for human review. This approach explicitly considers the different costs associated with correct routing, manual triage, and wrong routing. For example, misrouting a ticket was modeled as being five times more costly than routing it correctly or deferring it to a human.
To test this, the researchers used a public dataset of nearly 48,000 IT support tickets classified into eight categories. They compared four traditional classifiers—including one using TF-IDF (a way to represent text based on word frequency) combined with logistic regression—and four large language models like GPT-4o-mini, using both zero-shot (no prior examples) and few-shot (a few examples) prompting methods. The key was to select a routing policy by calibrating the models on part of the data while explicitly incorporating the cost model, then testing the fixed policy on held-out data to measure real-world performance.
One notable finding was that a simple TF-IDF plus logistic regression model, when combined with a threshold that allowed abstention, significantly reduced the overall cost and the number of wrong routes. This means the model learned when it was better to say “I’m not confident” and let a human handle the ticket instead. Interestingly, large language models did not outperform the fine-tuned DistilBERT model for this task, suggesting that off-the-shelf LLMs may require more specialized tuning or additional techniques for effective IT ticket classification.
The study also examined calibration—techniques that adjust the model’s confidence scores to better reflect true accuracy. While methods like isotonic regression improved calibration error, they did not consistently improve the model’s ability to discriminate between correct and incorrect predictions. The researchers introduced metrics such as correctness AUROC (a measure of how well the model ranks correct vs. incorrect predictions) and risk-coverage area to provide a fuller picture of performance beyond simple accuracy.
Though the research is focused on a specific dataset, prompt designs, and cost assumptions, it offers a valuable evaluation framework for organizations looking to implement AI systems that can strategically defer uncertain cases. This approach could lead to smarter IT support automation that balances efficiency with accuracy, potentially saving time and reducing costs.
Future work could explore how to improve the calibration and discrimination capabilities of large language models in this context, or how to adapt abstention strategies to other real-world tasks where errors carry different costs. For now, this study highlights the promise and challenges of cost-aware AI in practical IT operations.
Based on research published on arXiv by Sachin Gupta.