Explore/tool protocol/LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI Agents
L

Yogeswar Reddy Thota/LUMOS: A Semantic Operating-System Layer for Accessibility-Grounded AI AgentsUnknown

Current operating systems expose interfaces optimized for human users but not for AI agents. Humans benefit from pixels, icons, windows, visual grouping, mouse movement, and keyboard shortcuts; AI agents instead need compact semantic state, grounded actions, and reliable feedback. As a result, many computer-use agents are forced to interpret screenshots, OCR output, and visual crops, introducing high token costs, visual ambiguity, latency, and coordinate uncertainty. This paper introduces LUMOS (Language Model Unified Machine-Readable Operating-System Semantics), a semantic interaction layer between AI agents and operating systems. LUMOS converts native accessibility metadata and browser UI structures into machine readable semantic blueprints with stable identifiers, roles, names, values, bounds, and action affordances. It also supports live semantic pointer grounding by querying the UI element under or near the cursor through operating-system automation APIs. An LLM then acts through an accessibility grounded observe act loop using constrained visible-UI primitives rather than application-specific scripts. LUMOS does not claim to replace visual agents; instead, it reduces dependence on screenshots when operating systems already provide semantic structure. These results suggest a path toward AI-native operating systems and machine-readable interaction layers.

tool protocol
GitHubCompare
Refreshed 3mo ago
OverviewActivity52wAlternativesDocs
Stars0
Forks0
HF Downloads—30d
Last commit—
Refreshed3mo ago
Project healthUnknownNo activity data.
Production readinessResearch / EarlyBest for exploration and prototyping.
Risk notesUnknown licenseVerify license before production use.
AgentHub Score
55 / 100
Composite score from 6 signals. How we score →
Active project
55Score
Growth
40C
Activity
30C
Documentation
70C+
Maturity
45C
Community
42C
Production
58C
GitHub stars · 80 days observed0 not enough history
snapshots
Repository activity · 80 days observedReal snapshots from pushed_at
inactivepushed
2026-07-262026-10-15
Practical assessment
Should you use it?

✓ Best for

  • Research and experimentation
  • Prototype development
  • Learning agentic patterns

◎ Strengths

  • Active community
  • Open source
  • Well-documented API

✕ Not ideal for

  • Untested at scale without validation
  • Teams without AI/ML expertise

⚠ Watch-outs

  • Review changelog before updating
  • Verify license for commercial use
Technical details
What's inside
Language—
License—
Sourcearxiv
Open source✗ No
Commercial use—
Docs—
Demo—

AgentHub Score

55
Score 55/100
Below average

Alternatives

A
Agent-Native Telemetry: Verifiable State-Delta Evidence for Autonomous Operations
0 · tool protocol
55
S
Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents
0 · tool protocol
55
T
Toward Auditable AI Scientists: A Hypothesis Evolution Protocol for LLM Agents
0 · tool protocol
55
H
Heartbeat-Bound Hierarchical Credentials: Cryptographic Revocation for AI Agent Swarms
0 · tool protocol
55Compare all →

Recent activity

Latest commit ——
Indexed by AgentHub crawler3mo ago
Monitor for new releasesongoing