Explore/framework/Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific Research
B

Lezhi Yu, Xiaogang Xu, Yuhua Zhou, Shuibing He, Aimin Pan/Beyond Execution: Auditing Experimental Fidelity in LLM-Driven Scientific ResearchUnknown

LLM agents used for scientific experimentation must do more than generate executable code: they must implement the reference method faithfully, design experiments that test the paper's claims, and provide evidence supporting those claims. We show that agents often produce methodological hallucinations: silently reducing datasets or training budgets, replacing failed learning or generative components with lookup or oracle functions, or drawing conclusions from resource-limited settings where a method's claimed advantage disappears. To detect these failures, we introduce ABE-Ralph, a reference-anchored auditing framework that represents claims, protocols, required components, baselines, and metrics as structured experimental constraints, guides implementation through an 8-step workflow, and performs quantitative, qualitative, and code-level verification. Across 30 long-horizon reproduction runs covering 12 machine learning domains, ABE-Ralph achieves a 93% robust execution rate and identifies five scientific failure modes. In 23 NatureBench discovery tasks, ABE-Ralph matches or exceeds state-of-the-art performance on 5 tasks. These results show that reliable evaluation of AI scientists must assess whether the experimental design faithfully tests the intended claim and whether the resulting evidence supports it, rather than treating code execution or plausible metrics as evidence of scientific success.

framework
GitHubCompare
Refreshed 26d ago
OverviewActivity52wAlternativesDocs
Stars0
Forks0
HF Downloads—30d
Last commit—
Refreshed26d ago
Project healthUnknownNo activity data.
Production readinessResearch / EarlyBest for exploration and prototyping.
Risk notesUnknown licenseVerify license before production use.
AgentHub Score
55 / 100
Composite score from 6 signals. How we score →
Active project
55Score
Growth
40C
Activity
30C
Documentation
70C+
Maturity
45C
Community
42C
Production
58C
GitHub stars · 39 days observed0 not enough history
snapshots
Repository activity · 39 days observedReal snapshots from pushed_at
inactivepushed
2026-08-292026-10-08
Practical assessment
Should you use it?

✓ Best for

  • Multi-agent orchestration
  • Production agentic workflows
  • Stateful long-running tasks

◎ Strengths

  • Stable API
  • Active release cadence
  • Strong GitHub community

✕ Not ideal for

  • Simple single-step automation
  • Teams without Python/ML expertise

⚠ Watch-outs

  • Breaking changes between minor versions
  • Ecosystem lock-in if tightly coupled
Technical details
What's inside
Language—
License—
Sourcearxiv
Open source✗ No
Commercial use—
Docs—
Demo—

AgentHub Score

55
Score 55/100
Below average

Alternatives

O
OccuReward: LLM-Guided Occupant-Centric Reward Shaping for Demographic Equity in Grid-Interactive Buildings
0 · framework
55
C
CHAL: Council of Hierarchical Agentic Language
0 · framework
55
T
Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents
0 · framework
55
S
Structuring agentic AI for HPC code modernization
0 · framework
55Compare all →

Recent activity

Latest commit ——
Indexed by AgentHub crawler26d ago
Monitor for new releasesongoing