Increasing Resilience of Smart Home Agents — AI benchmark

Christopher Terrazas, Eduardo Cotilla-Sanchez/Increasing Resilience of Smart Home AgentsUnknown

Smart homes and smart devices are becoming more prevalent across millions of homes around the world. With the rise of AI, the smart home industry is quickly increasing its integration to manage common smart home tasks. However, existing work in large language models (LLMs) as agents within smart homes have shown minimal resilience due to limited environment scenarios or poor performance in complex tasks. We explore several strategies for LLMs as agents within the popular open-source software (OSS) smart home automation framework HomeAssistant to increase overall smart home resilience. Our approach combines traditional supervised learning techniques and optimized prompting as the core learning process. We use a diverse set of LLMs covering different levels of reasoning and costs in our optimization pipeline and evaluate their performance on a subset of the hardest tasks in a smart home benchmark. We include ReAct and Reflexion agentic paradigms and reveal how both provide marginal return on investment compared to fine-tuned LLMs for multi-device control within HomeAssistant but show promise in resilience tasks such as failure response.

benchmark

Stars0

Forks0

HF Downloads—30d

Last commit—

Refreshed11h ago

Project healthUnknownNo activity data.

Production readinessResearch / EarlyBest for exploration and prototyping.

Risk notesUnknown licenseVerify license before production use.

AgentHub Score

55 / 100

Composite score from 6 signals. How we score →

Active project

55Score

Growth

40C

Activity

30C

Documentation

70C+

Maturity

45C

Community

42C

Production

58C

GitHub stars · 0 days observed0 not enough history

snapshots

not enough history

Repository activity · 0 days observednot enough history from pushed_at

inactivepushed

not enough history

Practical assessment

Should you use it?

✓ Best for

Research and experimentation
Prototype development
Learning agentic patterns

◎ Strengths

Active community
Open source
Well-documented API

✕ Not ideal for

Untested at scale without validation
Teams without AI/ML expertise

⚠ Watch-outs

Review changelog before updating
Verify license for commercial use

Technical details

What's inside

Language—

License—

Sourcearxiv

Open source✗ No

Commercial use—

Docs—

Demo—

PaperarXiv ↗

AgentHub Score

Score 55/100

Below average

Alternatives

RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems

0 · benchmark

Commitment To Cooperation With Self-Negotiated Contracts

0 · benchmark

Exploring Structures in Physics Problems: Can AI Agents Discover Statistical Mechanical Mappings?

0 · benchmark

Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation

0 · benchmark

55 Compare all →

Recent activity

Latest commit ——

Indexed by AgentHub crawler11h ago

Monitor for new releasesongoing