Running my homelab with an AI agent

In May I wrote about the server in my room and the services I use most. Since then an AI agent has taken over most of the routine work, and this is how the server runs now.

Hermes does the routine work

I set up the first services by hand in 2022, before ChatGPT existed, and that's how I learned Docker and Docker Compose. These days most changes go through Hermes, an AI agent that runs on the server and that I talk to over Telegram. I ask it to deploy a service, write a script or find out why a container keeps restarting. It does most of the work. Anything even remotely sensitive or destructive needs my approval, or I do it myself over SSH.

I still keep an eye on the server myself. I check the monitoring and resource usage, and I look at containers by hand when something seems off.

Cron jobs

Hermes also runs most of the cron jobs, on schedules from every minute to once a month. A few examples:

The demo reset keeps the public Conatus demo clean, whatever visitors do to it.

Backups

At 02:00 a script dumps the PostgreSQL databases of the running containers, archives their named volumes and sends those, my configs and my media to a Hetzner Storage Box with restic. It keeps 30 daily, 12 weekly and 12 monthly snapshots.

Two recent runs, on 28 September and 1 October, saved incomplete snapshots. Some of the database dumps failed and the script couldn't read parts of the Home Assistant and Nextcloud data. The script recognised the problem and has its own exit code for a snapshot that was saved but is incomplete. But it only reported that in a local log. The runs since 4 October have been complete again.

A backup that fails loudly gets fixed straight away. One that half-works quietly can stay that way for weeks.

Backups only count if you can actually restore them successfully. In May I wrote about a PostgreSQL dump that didn't restore during a rebuild. The latest restore check, on 3 October, read 383 of the repository's 31,668 data packs, about 1%. It restored a database dump and a Home Assistant file to a scratch directory, compared their hashes with the originals and checked that the dump could be read. Both passed.

Alerts

Uptime Kuma watches the services on the server itself, with notifications through ntfy.

TwenteNotes runs on Railway, not on this server, so its uptime check watches it from the outside. Every minute it requests the API's readiness endpoint and the frontend. After two failures in a row it messages me on Telegram, and it sends another message when the site recovers. Waiting for a second failure stops one slow request from setting it off.

I chose Railway because I thought I was being smart by letting someone else run the infrastructure. Then users started telling me the site wasn't loading. When I checked, the site failed with network errors when it tried to fetch data. Railway's status page claimed there was no outage or reduced service, and I feel lied to every time I see that. It goes down often enough that I feel like I'm wasting money on the hosting, and it might be better to move it to the server in my room.

Updates

On the first of each month, a script lists every running image. Hermes then goes through the changelog of every project it can find one for and checks for breaking changes, database migrations and config changes before it sends me a recommendation on Telegram. During the review it isn't allowed to pull images, restart containers or change config.

Nothing updates until I approve it. When a release includes a database migration, the review suggests updating that service on its own after a fresh backup instead of updating everything at once.

The server does more than it did in May, and I do less of it by hand.