Explore/benchmark/Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment
C

Derek Yohn, Luke Flancher, Mirajul Islam, Khaled Slhoub/Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical AssessmentUnknown

This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions.

benchmark
GitHubCompare
Refreshed 1mo ago
OverviewActivity52wAlternativesDocs
Stars0
Forks0
HF Downloads—30d
Last commit—
Refreshed1mo ago
Project healthUnknownNo activity data.
Production readinessResearch / EarlyBest for exploration and prototyping.
Risk notesUnknown licenseVerify license before production use.
AgentHub Score
55 / 100
Composite score from 6 signals. How we score →
Active project
55Score
Growth
40C
Activity
30C
Documentation
70C+
Maturity
45C
Community
42C
Production
58C
GitHub stars · 0 days observed0 not enough history
snapshots
Repository activity · 0 days observedReal snapshots from pushed_at
inactivepushed
2026-07-262026-07-27
Practical assessment
Should you use it?

✓ Best for

  • Research and experimentation
  • Prototype development
  • Learning agentic patterns

◎ Strengths

  • Active community
  • Open source
  • Well-documented API

✕ Not ideal for

  • Untested at scale without validation
  • Teams without AI/ML expertise

⚠ Watch-outs

  • Review changelog before updating
  • Verify license for commercial use
Technical details
What's inside
Language—
License—
Sourcearxiv
Open source✗ No
Commercial use—
Docs—
Demo—

AgentHub Score

55
Score 55/100
Below average

Alternatives

A
AgentBench
3.6k · benchmark
61
W
WebArena
1.6k · benchmark
61
V
VisualAgentBench
274 · benchmark
60
A
ALFWorld
810 · benchmark
55Compare all →

Recent activity

Latest commit ——
Indexed by AgentHub crawler1mo ago
Monitor for new releasesongoing