Managing a busy email inbox can be overwhelming, especially when messages come in multiple languages and vary widely in importance. To help users stay on top of their emails, personal email assistants aim to automatically sort messages, flag urgent ones, draft replies, or create tasks. But how well do these AI helpers actually perform, and how can we measure their accuracy before training them on biased data? A newly published research paper introduces a fresh approach to evaluating personal inbox agents and compares different AI techniques on real multilingual emails.
Key Takeaways
- The common production keyword-based rules used in many email assistants performed poorly, scoring only 28% accuracy on identifying message types—worse than simply guessing the most frequent category.
- A straightforward machine learning method combining TF-IDF (a way to weigh words) with logistic regression achieved the highest accuracy across multiple tasks, such as classifying message type (79%) and detecting if a reply is needed (83%).
- Large language models (LLMs) like Gemma 4 performed competitively but did not clearly outperform simpler models, especially given the small amount of labeled data available.
- Identifying the sender’s relationship to the user (client, supplier, or service) cannot be reliably done from a single email message alone and requires looking at the broader contact history.
The study addresses a common problem in AI research for personal assistants: often, the data used to train and test these systems is biased because the labels (such as “urgent” or “reply needed”) come from the system itself rather than independent human judgment. To overcome this, the researchers developed a rigorous measurement protocol that includes human labeling of emails before any training happens. This “ground truth” data set ensures a fair comparison of different models.
They tested nine different systems on a real-world dataset of 739 emails in Italian and English, collected from the inbox owner. The dataset included both unbiased samples and more challenging “adversarial” emails designed to stress the models. Among the tested approaches were traditional keyword rules, a baseline machine learning model using TF-IDF and logistic regression, locally run decision models with typed data inputs, and hosted generative large language models (LLMs) like Qwen2.5 and Gemma 4.
TF-IDF, short for Term Frequency-Inverse Document Frequency, is a method that scores words based on how important they are in a document relative to a collection of documents. Logistic regression is a simple yet effective statistical model often used for classification tasks. Together, these tools provided a surprisingly strong baseline that outperformed more complex models in many respects, especially when only about 200 labeled emails were available for training.
Interestingly, the study found that the most accurate model in terms of raw classification did not always yield the best decisions when integrated into user actions. For example, it often failed to flag emails that actually required the user’s attention, ignoring 18% of such messages compared to just 4% for the best performing LLM. Adjusting thresholds for identifying “noise” emails helped reduce this problem and also lowered system costs.
Another important insight is that understanding the sender’s relationship to the user—a key factor for prioritizing emails—is not something that can be inferred from a single message. Instead, it requires analyzing the user’s contact graph, or the network of past interactions with that sender. This suggests that personal email assistants need to incorporate broader context beyond individual messages to make smarter decisions.
Overall, this study highlights the importance of carefully measuring AI models before training and deploying them in personal email management. By establishing a reliable evaluation process and comparing a range of approaches on real, human-labeled data, the researchers provide valuable guidance for building more effective and trustworthy email assistants. Looking ahead, integrating contact history and refining decision thresholds may help these systems better support users in managing their multilingual inboxes without missing critical messages.
Based on research published on arXiv by Federico Lucca.