BOHM: Zero-Cost Hierarchical Attribution for Compound AI Systems — AI observability

Joss Armstrong/BOHM: Zero-Cost Hierarchical Attribution for Compound AI SystemsUnknown

Compound AI systems route tasks through hierarchies of specialised components. Attribution is dominated by Shapley-based methods (SHAP), which decompose a coalition value function into per-component marginal contributions and require evaluation of the system on arbitrary component subsets. That requirement fails for third-party APIs, opaque endpoints, and agentic orchestrators that concentrate routing on a few tools, leaving most coalitions un-evaluable from the deployed orchestrator. We introduce BOHM, which extracts a hierarchical attribution tree directly from the routing weights such systems already maintain: leaf attribution is the path product of root-to-leaf routing weights; level-k attribution is the induced distribution over depth-k nodes. The method has zero marginal cost, requires no access to component internals, and provides multi-resolution attribution at every level simultaneously, which flat methods cannot offer at any evaluation budget. BOHM and SHAP answer different questions and converge when the deployed router routes near-optimally. On 18 LLMs in a 3-level hierarchy over 880 LiveCodeBench problems, BOHM yields Kendall tau=0.928; SHAP reaches tau=0.980 at 9,000x more coalition evaluations per seed. On a 5-driver, 7-benchmark agentic study (35 cells, complete coverage), drivers concentrate routing on a single tool (top-share median 0.65), and cell-level tau(BOHM,SHAP) is predicted by whether the driver's top pick is the empirically best tool (mean +0.22 vs ~+0.01). On a US Census hierarchy (475 leaves, 4 levels), BOHM recovers ground-truth rankings at every level (tau up to 0.722). BOHM satisfies efficiency, monotonicity, symmetry, and weak suppression but not Shapley's additivity. It is best understood as a complementary primitive: a multi-resolution decomposition computable wherever routing state exists, whose disagreement with Shapley is itself diagnostic.

observability

Stars0

Forks0

HF Downloads—30d

Last commit—

Refreshed4mo ago

Project healthUnknownNo activity data.

Production readinessResearch / EarlyBest for exploration and prototyping.

Risk notesUnknown licenseVerify license before production use.

AgentHub Score

55 / 100

Composite score from 6 signals. How we score →

Active project

55Score

Growth

40C

Activity

30C

Documentation

70C+

Maturity

45C

Community

42C

Production

58C

GitHub stars · 78 days observed0 not enough history

snapshots

Repository activity · 78 days observedReal snapshots from pushed_at

inactivepushed

2026-07-262026-10-13

Practical assessment

Should you use it?

✓ Best for

Agent tracing in production
LLM cost tracking
Debugging agent failures

◎ Strengths

Integrates with major frameworks
OpenTelemetry compatible

✕ Not ideal for

Small prototype projects
Teams without existing observability culture

⚠ Watch-outs

Data retention costs at scale
Privacy implications of storing LLM traces

Technical details

What's inside

Language—

License—

Sourcearxiv

Open source✗ No

Commercial use—

Docs—

Demo—

PaperarXiv ↗

AgentHub Score

Score 55/100

Below average

Alternatives

Efficient Auditing of Adversarial AI Agent Behavior from Agent Traces

0 · observability

AgentSpy: Making AI Agent Behavior Observable

0 · observability

Traccia: An OpenTelemetry-Based Governance Platform for AI Systems

0 · observability

Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems

0 · observability

55 Compare all →

Recent activity

Latest commit ——

Indexed by AgentHub crawler4mo ago

Monitor for new releasesongoing