sascha/homelab-butler
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: Butler 2.3 deterministic overview

Browse source 
Control-plane health semantics, concurrent backup checks and lightweight-model overview.
This commit is contained in:
sascha 2026-07-22 10:38:12 +02:00
parent da1d2aaf8f
commit 619193bddd
6 changed files with 1014 additions and 134 deletions
Download patch file Download diff file Expand all files Collapse all files

175
README.md
View file

@ -1,110 +1,127 @@
# Homelab Butler 🤵
Unified API proxy + infrastructure management for Homelab Pfannkuchen.
Unified API proxy and infrastructure management for Homelab Pfannkuchen.
**Base URL:** `http://10.4.1.116:8888`
**Auth:** `Authorization: Bearer ***`
**Version:** 2.1.0
- **Base URL:** `http://10.4.1.116:8888`
- **Authentication:** `Authorization: Bearer <BUTLER_TOKEN>`
- **Version:** 2.3.0
- **Interactive API documentation:** `/docs`
## Service Proxy
## Service proxy
Proxies requests to backend services with automatic authentication:
Requests are proxied as `/{service}/{backend-path}`. Butler adds each backend's authentication automatically and forwards query parameters.
| Service | Backend | Auth |
|---------|---------|------|
| `dockhand` | 10.4.1.116:3000 | Session |
| `sonarr` | 10.2.1.100:8989 | API Key |
| `radarr` | 10.2.1.100:7878 | API Key |
| `seerr` | 10.2.1.100:5055 | API Key |
| `outline` | 10.1.1.100:3000 | Bearer |
| `n8n` | 10.4.1.113:5678 | X-N8N-API-KEY |
| `proxmox` | 10.5.85.11:8006 | PVE Token |
| `homeassistant` | 10.10.1.20:8123 | Bearer |
| `grafana` | 10.1.1.111:3000 | Bearer |
| `uptime` | 159.69.245.190:3001 | Bearer |
| `waha` | 10.4.1.110:3500 | API Key |
| `forgejo` | 10.4.1.116:3001 | Bearer |
| `semaphore` | 10.4.1.116:8090 | Bearer |
| Service | Backend | Authentication |
|---|---|---|
| `dockhand` | `10.4.1.116:3000` | Session |
| `sonarr` / `sonarr1080p` | `10.2.1.100:8989/8990` | API key |
| `radarr` / `radarr1080p` | `10.2.1.100:7878/7879` | API key |
| `seerr` | `10.2.1.100:5055` | API key |
| `outline` | `10.1.1.100:3000` | Bearer |
| `n8n` | `10.4.1.113:5678` | n8n API key |
| `proxmox` | `10.5.85.11:8006` | PVE API token |
| `homeassistant` | `10.10.1.1:8123` | Bearer |
| `grafana` | `10.1.1.111:3000` | Bearer |
| `uptime` | `10.5.85.5:3001` | Web UI only; no supported REST API |
| `waha` | `10.4.1.110:3500` | API key |
| `forgejo` | `10.4.1.116:3001` | Bearer |
| `semaphore` | `10.4.1.116:8090` | Bearer |
| `fileflows` | `10.2.1.104:8268` | Local API, no additional auth |
Usage: `GET/POST/PUT/DELETE /{service}/{path}`
Known secret response fields such as Dockhand's `hawserToken` and `webhookSecret` are redacted before data leaves Butler.
## VM Lifecycle
## Operations
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/vm/list` | GET | All VMs across all 7 Proxmox nodes |
| `/vm/create` | POST | Full VM deployment (~10 min): ISO build, VM create, SSH wait, inventory, Ansible |
| `/vm/status/{vmid}` | GET | VM status (CPU, RAM, uptime) |
| `/vm/{vmid}` | DELETE | Destroy VM |
|---|---:|---|
| `/info` | GET | Secret-free machine-readable context |
| `/status` | GET | Concurrent authenticated functional probes with deterministic states |
| `/overview` | GET | Compact overall verdict and ordered findings for lightweight models; `details=true` adds raw data |
| `/audit` | GET | Recent proxied/management calls |
| `/health/all` | GET | SSH reachability and Docker status for inventory hosts |
| `/backup/status` | GET | Concurrent Borgmatic checks with age, state and summary |
| `/disk/usage` | GET | Root filesystem usage per host |
| `/logs/{host}/{container}` | GET | Docker logs, `tail` query supported |
| `/docker/inspect/{host}/{container}` | GET | Sanitized image, runtime, resources, mounts and state |
| `/docker/restart/{host}/{container}` | POST | Restart container; supports `dry_run=true` |
| `/config/reload` | POST | Reload YAML configuration and credential cache |
### POST /vm/create
```json
{"node": 5, "ip": "10.5.1.115", "hostname": "lychee", "cores": 2, "memory": 4096, "disk": 32}
```
Steps: iso-builder → Proxmox VM → wait SSH → add to pfannkuchen.ini → Ansible base setup (Docker, Borgmatic, Hawser)
## Ansible / Inventory
## VM lifecycle and inventory
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/inventory/host` | POST | Add host to pfannkuchen.ini (idempotent, with group) |
| `/ansible/run` | POST | Run Ansible playbook on host |
|---|---:|---|
| `/vm/list` | GET | All VMs across the seven Proxmox nodes |
| `/vm/create` | POST | VM deployment; supports `dry_run=true` |
| `/vm/status/{vmid}` | GET | CPU, RAM, uptime and state |
| `/vm/destroy/{vmid}` | DELETE | Full lifecycle cleanup; supports `dry_run=true` |
| `/inventory/host` | POST | Create or update inventory host and host vars |
| `/ansible/run` | POST | Run the standard setup for a host |
Example inventory upsert:
### POST /ansible/run
```json
{"hostname": "lychee"}
{"name":"example","ip":"10.5.1.115","group":"auto","user":"sascha"}
```
**Post-Run Automation** (after successful Ansible):
1. **Hawser Token Sync** – Reads `/etc/hawser/config` from VM, syncs token to Dockhand environment
2. **SOPS + .env Setup** – If `compose.yaml` exists in `/app-config/github/{hostname}/`:
- Generates secure secrets (admin password, DB password, secret key)
- Creates `.env` file with service-specific variables
- Encrypts to `.env.enc` using SOPS (Age key from automation1)
- Copies both files to VM's git repo directory
- Stores secrets in Butler vault cache for future reference
Node-7 VMs default to SSH user `chris`; Proxmox nodes default to `root`.
### POST /inventory/host
```json
{"name": "lychee", "ip": "10.5.1.115", "group": "auto"}
```
## TTS / Speech
## TTS
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/tts/speak` | POST | Text-to-speech via Chatterbox |
| `/tts/voices` | GET | Available voices |
| `/tts/health` | GET | Speaker + Chatterbox status |
|---|---:|---|
| `/tts/speak` | POST | Chatterbox/speaker TTS |
| `/tts/voices` | GET | Available Chatterbox voices |
| `/tts/health` | GET | Speaker and Chatterbox status |
### POST /tts/speak
```json
{"text": "Hallo!", "target": "speaker"}
```
- `"target": "speaker"` → plays on Pi5 speaker (10.10.1.166)
- `"target": "telegram"` → generates OGG on hermes, use `MEDIA:/tmp/trulla_voice.ogg`
The speaker endpoint is `10.5.85.2:10800`; Chatterbox runs at `10.2.1.104:8004`.
## Credentials
## Deployment
Reads from Vaultwarden cache (synced by host cron) with flat-file fallback (`/data/api/`).
Required mounts and settings are defined in `compose.yaml`:
## Stack
- `.env` containing `BUTLER_TOKEN`
- `/app-config/kiro/api/` as flat-file credential fallback
- persistent Vaultwarden cache volume
- SSH key mounted read-only at `/root/.ssh`
- `butler.yaml` mounted read-only at `/data/butler.yaml`
```
docker compose build && docker compose up -d
Git is the source of truth. Build/recreate the Compose service only after committing and pushing changes.
## Tests
```bash
python -m pytest -q tests/test_app.py
```
Requires:
- `.env` with `BUTLER_TOKEN`
- `/app-config/kiro/api/` flat-file credentials
- SSH key mount (`/home/sascha/.ssh:/root/.ssh:ro`) for VM operations
Integration Compose definition: `tests/compose.integration.yaml` (binds only to `127.0.0.1:8889`).
## Changelog
### v2.1.0 (22.04.2026)
- ✅ **Hawser Token Auto-Sync** – After `/ansible/run`, reads token from VM and updates Dockhand environment
- ✅ **SOPS + .env Automation** – Auto-generates and encrypts environment files for Git-centric deployments
- ✅ **Service Detection** – Recognizes Paperless-ngx and other services from hostname, generates appropriate env vars
### 2.3.0 — 22.07.2026
### v2.0.0
- Initial unified API proxy release
- Added `/overview`, a compact schema-versioned verdict for lightweight language models.
- Added explicit `healthy`, `degraded`, `auth_failed`, `misconfigured` and `offline` service states.
- Service probes now use the configured backend credentials and run concurrently.
- Backup checks now run with bounded concurrency and a 12-second per-host timeout.
- Backup results include age and severity (`healthy` up to 30 h, `warning` up to 48 h, then `critical`).
- Host and disk collection now run concurrently.
- Added regression tests for classification, authentication, backup age/concurrency and overview output.
### 2.2.0 — 17.07.2026
- Restored and modernized `/info`, `/health/all`, `/backup/status`, `/disk/usage` and Docker log/restart endpoints.
- Fixed stale Home Assistant, Uptime Kuma and speaker addresses.
- Fixed Forgejo credential fallback by deploying the YAML-driven service config.
- Correct SSH defaults for Proxmox nodes and Node-7 VMs.
- Added recursive secret redaction for proxied JSON.
- Forward query parameters through the generic proxy.
- Added validated inventory upserts.
- Added regression tests and a loopback-only integration Compose setup.
### 2.1.1
- Full VM destruction lifecycle cleanup.
### 2.1.0
- YAML service configuration, status/audit endpoints, dry-run support, Hawser token sync and SOPS automation.