Operations — routine procedures¶
The implementation guide (Phases 0 to 6) builds the HomeLab once. This section collects what you do afterwards, every time it is needed: adding pieces, maintaining them and rotating credentials.
Each procedure says when to use it, what must be ready beforehand, the steps in order, how to verify and what to check if something fails.
| Procedure | When | Where it runs |
|---|---|---|
| Product URLs and access | Know where each tool lives, how to sign in and whether it responds | Browser, kubectl |
| SSH into the servers | Sign in to invincible, oliver or deborah with your GitHub account |
Your workstation (opkssh, ssh) |
| Monitoring and alerts | Check node temperatures and usage, understand a Discord notice, silence an alert | Grafana, kubectl |
| Add a repo | Every new project in the maceosoft org |
1Password + PR in infra |
| Add a node | Add a machine to the Incus and K3s cluster | Ansible (homelab) |
| Remove or replace a node | Retire a machine or swap it for another | kubectl, Incus and Ansible |
| Add a disk to a node | More storage for Longhorn or Incus | Ansible (homelab) |
| Incus backups and snapshots | Schedule backups or recover an instance | Incus |
| Upgrade K3s | Review the automatic upgrades or pin a version | GitOps + Ansible |
| Patch the nodes | Install Debian security patches when monitoring warns you | Ansible + apt |
| Upgrade Argo CD | It upgrades itself to the latest version; first upgrade, pinning a version, what to do if it fails | GitOps + Ansible |
| Change the Gateway controller | Switch between Kong, Traefik and NGINX; TLS for *.homelab.local |
GitOps + kubectl |
| Upgrade Debian on deborah (12 → 13) | Move the control plane to Debian 13, with a backup and a boot gate | Ansible (homelab) |
| Calico (CNI) with Argo CD | How Argo CD manages and upgrades the pod network | GitOps + kubectl |
| Runner cache (ARC) | What is cached between jobs, how to check it, prune it or empty it | kubectl + GitOps |
| Disk and DNS health | What to do when a SMART, eMMC wear or BIND alert fires | Ansible (homelab) + nodes |
| Incus metrics certificate | What the read-only client certificate used by monitoring is and how to register it in Incus | kubectl + Ansible (homelab) |
| New version notices (Renovate) | Which products have a new version, Discord notices and how to handle each PR | GitHub + Discord + GitOps |
| Rotate credentials | Periodically, or if a credential leaked | 1Password + playbooks |
| gitops pipeline | What each gitops PR validates; enable the Argo CD diff (once) |
GitHub Actions + 1Password |
Where the commands run
Unless the procedure says otherwise, Ansible commands run from the
ansible/ folder of the homelab repo, and kubectl commands use
the HomeLab kubeconfig (~/.kube/homelab-k3s.yaml).