While artificial intelligence has made impressive strides in writing correct code for individual problems, it still struggles with the more complex task of developing full machine learning projects on its own. This involves not just coding but iteratively revising data, models, and validation strategies over hours or days. A new research paper introduces TraceML, a detailed analysis comparing how humans and AI agents tackle these development challenges in real-world machine learning competitions. Understanding these differences is crucial for improving AI’s ability to work autonomously on complex projects.
Key Takeaways
- The study collected and analyzed 4,465 human development trajectories from 134 Kaggle competitions, alongside agent attempts on seven of these competitions.
- Humans alternate between data processing, validation, model adjustments, and ensembling, often revisiting earlier strategies they had previously set aside.
- AI agents tend to follow narrow, repetitive loops—such as tuning model weights or mutating models in place—without the flexibility to pivot or revisit abandoned approaches like humans do.
- Incorporating a short planning prompt based on human practices helps AI agents adopt more human-like behaviors and improves their scores, but does not fully close the gap in development style.
The research team developed TraceML as a new dataset and framework to capture the entire development process of machine learning projects, rather than just evaluating the final results. They gathered thousands of “trajectories,” which are sequences of code versions created during competition work. Each version is annotated with detailed labels: what action was taken (e.g., data cleaning, model change), the intent behind it, how big the edit was, and how the change affected the model’s performance score. This granular approach allows researchers to see not just what changes were made, but how and why, providing insight into the decision-making process.
To compare human and AI behavior, the authors analyzed two AI agents—Codex and MLEvolve—working on the same competitions as humans. Codex primarily focused on adjusting ensemble weights and fine-tuning submissions, while MLEvolve made incremental mutations to models without shifting strategies. In contrast, human experts showed a dynamic workflow, switching among data work, validation, modeling, and ensembling, and often revisiting approaches they had previously abandoned. This flexibility appears to be a key factor in human success.
The researchers also experimented with giving AI agents a concise planning prompt distilled from human expert practices. This prompt encouraged the agents to plan their steps in a more human-like fashion. While this helped agents improve their scores and adopt some human-like behaviors, their overall workflow remained constrained by their underlying programming, failing to fully replicate the diverse and adaptive strategies humans use.
By releasing the TraceML dataset, schema, labeling tools, and extraction pipeline publicly, the authors provide a valuable resource for the AI research community to better understand and improve autonomous machine learning development. This work highlights that closing the performance gap between humans and AI in machine learning projects requires more than just better code generation—it demands agents that can plan, pivot, and revisit strategies over extended development cycles. Future research may build on these insights to design AI systems capable of more flexible and effective machine learning development, potentially accelerating progress in data science and AI applications.
Based on research published on arXiv by Jiarui Yan, Weiwei Sun, Sijie Li et al..