Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks — AI benchmark

Shitanshu Bhushan, Yunxiang Zhang, Lu Wang/Can LLM Agents Discover? Evaluating Creativity on ML Engineering TasksUnknown

Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). Evaluating two agent frameworks, AIDE and AIRA-Dojo, on 10 Kaggle-style machine learning tasks from MLE-Bench, we develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments, providing a reliable automated metric for P-Creativity evaluation at scale. Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance. Our findings reveal that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance.

benchmark

Stars0

Forks0

HF Downloads—30d

Last commit—

Refreshed6h ago

Project healthUnknownNo activity data.

Production readinessResearch / EarlyBest for exploration and prototyping.

Risk notesUnknown licenseVerify license before production use.

AgentHub Score

55 / 100

Composite score from 6 signals. How we score →

Active project

55Score

Growth

40C

Activity

30C

Documentation

70C+

Maturity

45C

Community

42C

Production

58C

GitHub stars · 0 days observed0 not enough history

snapshots

not enough history

Repository activity · 0 days observednot enough history from pushed_at

inactivepushed

not enough history

Practical assessment

Should you use it?

✓ Best for

Research and experimentation
Prototype development
Learning agentic patterns

◎ Strengths

Active community
Open source
Well-documented API

✕ Not ideal for

Untested at scale without validation
Teams without AI/ML expertise

⚠ Watch-outs

Review changelog before updating
Verify license for commercial use

Technical details

What's inside

Language—

License—

Sourcearxiv

Open source✗ No

Commercial use—

Docs—

Demo—

PaperarXiv ↗

AgentHub Score

Score 55/100

Below average

Alternatives

RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems

0 · benchmark

Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

0 · benchmark

Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection

0 · benchmark

AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation

0 · benchmark

55 Compare all →

Recent activity

Latest commit ——

Indexed by AgentHub crawler6h ago

Monitor for new releasesongoing