Researchers have tested how well some of today’s leading artificial intelligence language models can work together to identify a secret target from a large set of Wikipedia article summaries using a very limited number of yes/no questions. This experiment sheds light on how efficiently AI models can communicate under strict information limits, which is important for developing smarter, more resourceful AI systems that must make decisions based on limited data.
Key Takeaways
- Six top language models were evaluated on a “log(N)-Questions” game, where a questioner must identify a secret target from N Wikipedia excerpts using only log₂(N) yes/no questions.
- One model, Claude Opus 5, performed noticeably worse than the others, while five models—GLM-5.3, GPT-5.6 Sol, Grok 4.6, Gemini 3.8 Flash, and Kimi K3—had similar success rates that declined predictably as the number of documents increased.
- The chance of winning the game fits a formula where success per question is about 92.8%, indicating that each yes/no answer carries slightly less than a full bit of reliable information.
- Errors were evenly split between incorrect answers and failures to narrow down to the correct document, with models rarely guessing documents they lacked evidence for.
In this study, the researchers set up a two-player game involving AI language models to explore how well they can communicate complex information efficiently. The questioner is given a set of Wikipedia lead paragraphs—ranging from as few as 4 to as many as 1024—and must identify a secret target article by asking only a limited number of yes/no questions. Specifically, the number of questions allowed grows logarithmically with the number of documents, following the formula log₂(N), meaning the questioner must be very strategic to pinpoint the target quickly.
The answerer, who sees only the secret target and the question, responds with a single word—either “Yes” or “No.” Both roles are powered by the same AI model, which means the study effectively measures how well a model can communicate with itself across an information gap. This setup mimics real-world scenarios where AI systems must make decisions or find answers with incomplete information and limited communication.
To carry out the experiments, the team ran 408 games across different document set sizes, incurring an API cost of $363. They tested six advanced language models, including well-known names like GPT-5.6 Sol and others such as GLM-5.3 and Claude Opus 5. The results showed one model, Claude Opus 5, lagging behind the pack, while the other five performed comparably. Interestingly, as the number of documents increased, the overall win rate dropped in a mathematically predictable way, confirming that the difficulty scales with the size of the search space.
The researchers analyzed the types of errors made by the models. About half were due to incorrect answers to questions, and half were failures to discriminate between candidate documents effectively. Notably, the models rarely guessed documents that contradicted the evidence they had, indicating cautious reasoning. The team also found that only two models consistently extracted a full bit of information per question by cleverly using document titles to partition the search space—a strategy that became more common as the number of documents grew beyond 32.
Another interesting finding was that the computational effort, measured by the number of reasoning tokens used, varied significantly between models but did not strongly correlate with success. This suggests that simply “thinking harder” or using more computational resources does not guarantee better communication or accuracy in this task.
This research provides valuable insights into the communication efficiency of AI language models when facing information asymmetry—situations where one agent has more information than the other. Understanding these dynamics can help improve AI systems designed for collaborative problem-solving, search, and decision-making with limited data exchange. Future work may explore refining questioning strategies or answer generation to boost reliability further, or apply these findings to real-world applications such as interactive search engines or multi-agent AI teams.
Based on research published on arXiv by Peter Potash.