Explore/benchmark/How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks
H

Yanlin Fei, Nazhou Liu, Xinmiao Yu, Shaolong Chen, Lei Li, Rahul Thapa, Madalina Ciobanu, Qingqing Mao, Ritankar Das/How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research TasksUnknown

This paper introduces AutoResearchEval, an evaluation suite with 100 real-world scientific research tasks that assess end-to-end autonomous research agents, and presents the AutoResearch Failure Taxonomy (ARFT) detailing 45 failure patterns. It is aimed at researchers studying agentic systems in scientific discovery.

benchmark
GitHubCompare
Refreshed 12d ago
OverviewActivity52wAlternativesDocs
Stars0
Forks0
HF Downloads30d
Last commit
Refreshed12d ago
Project healthUnknownNo activity data.
Production readinessResearch / EarlyBest for exploration and prototyping.
Risk notesUnknown licenseVerify license before production use.
AgentHub Score
55 / 100
Composite score from 6 signals. How we score →
Active project
55Score
Growth
40C
Activity
30C
Documentation
70C+
Maturity
45C
Community
42C
Production
58C
GitHub stars · 11 days observed0 not enough history
snapshots
Repository activity · 11 days observedReal snapshots from pushed_at
inactivepushed
2026-08-192026-08-31
Practical assessment
Should you use it?

✓ Best for

  • Research and experimentation
  • Prototype development
  • Learning agentic patterns

◎ Strengths

  • Active community
  • Open source
  • Well-documented API

✕ Not ideal for

  • Untested at scale without validation
  • Teams without AI/ML expertise

⚠ Watch-outs

  • Review changelog before updating
  • Verify license for commercial use
Technical details
What's inside
Language
License
Sourcearxiv
Open source✗ No
Commercial use
Docs
Demo

AgentHub Score

55
Score 55/100
Below average

Alternatives

R
RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems
0 · benchmark
55
B
Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling
0 · benchmark
55
P
PCFBench: A Diagnostic Benchmark for Product Carbon Footprint Estimation
0 · benchmark
55
S
Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection
0 · benchmark
55
Compare all →

Recent activity

Latest commit —
Indexed by AgentHub crawler12d ago
Monitor for new releasesongoing