Artificial intelligence agents are increasingly expected to perform complex tasks that rely on recalling past information accurately and efficiently. However, most existing tests for AI memory focus narrowly on question-answering formats and prioritize accuracy alone, often ignoring how quickly or cheaply an AI system can retrieve and use information. A newly published research paper introduces DolphinBench, a benchmark designed to evaluate AI memory in a more realistic and comprehensive way by measuring not just accuracy but also the cost and speed of memory use during task completion.
Key Takeaways
- DolphinBench tests AI agents on completing real-world tasks that depend on recalling long-term memory, rather than just answering isolated questions.
- The benchmark uses three distinct “personas” or user profiles, each with about 500,000 tokens of message history, to simulate complex knowledge work scenarios.
- Each persona has 200 verified tasks where success depends on using relevant memory, confirmed by agents failing without access to that history.
- Evaluations require reporting not only accuracy but also the total cost and latency (response time), enabling a balanced assessment of memory systems.
Traditional AI memory benchmarks typically ask agents to retrieve specific facts in a question-answer format, where the question itself hints at what needs to be remembered. This setup can oversimplify real-world applications where agents must recall context over long periods and apply it flexibly to perform tasks. Moreover, prior benchmarks often focus solely on accuracy, allowing AI systems to trade off excessive computational cost or slower responses for better performance, which is impractical for real-world use.
DolphinBench addresses these limitations by evaluating AI agents in scenarios that mimic knowledge workers managing ongoing conversations or projects. The researchers created three personas—representing different user types—with extensive message histories totaling around half a million tokens each. The benchmark includes 200 tasks per persona that require the agent to use information from these histories to complete the task successfully. To verify the importance of memory, the researchers ran tests both with and without the relevant history, observing that agents only succeeded when they had access to it.
Another key innovation in DolphinBench is its holistic evaluation approach. Instead of just measuring whether the agent’s answer was correct, the benchmark also requires reporting the total cost involved (such as computational resources or API usage fees) and the latency, which is the time taken to produce a response. This approach reflects the real-world tradeoffs that developers face when designing AI systems that must balance speed, cost, and accuracy.
By providing an open-source dataset and evaluation code, the DolphinBench team offers the AI research community a valuable tool for developing and comparing memory systems in agents. This benchmark could help guide improvements in AI that better handle long-term memory and context in practical applications, such as virtual assistants, customer support bots, or workflow automation tools.
Looking ahead, DolphinBench’s more nuanced evaluation framework may encourage the development of AI memory systems that are not only accurate but also efficient and responsive. As AI becomes more integrated into daily tasks, benchmarks like this will be essential for ensuring these systems operate effectively without prohibitive costs or delays. Future research may build on DolphinBench to explore even richer memory scenarios and further optimize the balance between performance and resource use.
Based on research published on arXiv by Soumil Rathi, Deshraj Yadav, Taranjeet Singh.