An agent can help diagnose and automate real infrastructure, but mistakes in a physical environment have consequences. Broad access is easy; useful access with a controlled blast radius is the actual engineering problem.
AIbuilding
Can I safely let AI operate my house?
A staged experiment in giving an AI-assisted operations environment useful visibility—and eventually narrow control—without pretending autonomy is magic.
Start with a dedicated operations VM and read-only observation. Add monitoring, backups and explicit sudo boundaries before narrowly scoped changes. Increase permissions only when the evidence and recovery path justify it.
The experiment currently has a controlled operational foundation and growing observability. Broader autonomous control is intentionally not the finish line; the work is about discovering which permissions are safe and genuinely useful.
Abstract architecture / sensitive detail removed
01 Observe
02 Explain
03 Propose
04 Approve
05 Act narrowly
06 Verify
The question behind the project
How much autonomy can an AI agent safely have in a real physical environment?
The uninteresting answer is “give it administrator access and see what happens.” The useful experiment is to build a permission model where the agent can gather evidence, explain a problem and perform a small approved action without gaining a general route through the environment.
Phase one: observation
The first useful capability is read-only visibility into health and state: enough to spot unavailable services, stale devices, failed jobs or unusual behaviour. Observation creates value while keeping the control boundary simple.
Phase two: controlled action
Any write path should be explicit, narrow and recoverable. That means task-specific commands, strong input handling, backups where state changes, and verification after action. “The command returned zero” is not the same as “the house is in the intended state.”
Series roadmap
Future notes will cover the operations VM, read-only visibility, Proxmox monitoring, backup evidence, Home Assistant observability, controlled automation changes and the permission boundaries that connect them.
Notes to future me
What this project is teaching me.
- Capability should grow more slowly than visibility.
- A proposed change and a verified outcome are separate events.
- Physical systems need smaller blast radii than disposable test environments.
This page explains design intent and lessons without publishing credentials, private addressing, access paths, device identifiers or a complete security topology.