# HomeLab — K3s + Incus + CAPN

> Documentation for a hybrid x86_64 + ARM64 HomeLab (Incus, K3s, CAPN, Argo CD)

Documentación en español de un HomeLab híbrido (x86_64 + ARM64) con Incus, K3s, Argo CD (GitOps), Calico, MetalLB, Gateway API y Longhorn. Está pensada para quien empieza y para quien ya sabe: cada fase explica qué hacer, por qué y cómo verificarlo.

# Inicio y glosario

# Guía de implementación (Fases 0 a 6)

# Appendix — Uninstall K3s

Outside the normal [implementation](https://homelab.maceo.cl/en/implementacion/index.md) path. Use it for a full reset of the management cluster before reinstalling Phase 3.

## Goal

Remove the K3s (lightweight Kubernetes distribution) server and agents on the 3 nodes in an orderly way.

## Stack

- **[K3s](https://docs.k3s.io/)** — Uninstall scripts in `/usr/local/bin/k3s-uninstall.sh` and `k3s-agent-uninstall.sh`.
- **[Ansible](https://docs.ansible.com/)** — `playbook-k3s-uninstall.yml`.

## Run

```
cd ansible
ansible-playbook -i inventory.ini playbook-k3s-uninstall.yml
```

## Verify

```
ansible -i ansible/inventory.ini incus_cluster -m command -a "systemctl is-active k3s" || true
# inactive / failed en todos
```

## Next

Reinstall: **[Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)**

# Phase 0 — Preparation

## Objective

Have the repo cloned, Ansible operational, inventory validated, and (optionally) a measured hardware report before touching network or clusters.

## What you learn

How Ansible uses an **inventory** as the source of truth for hosts, and why an optional **discovery** avoids assumptions about RAM, disk, and architecture when choosing roles (control plane, Longhorn, etc.).

Learn this phase

**In short:** Ansible is like a task list you hand to several helpers: they all run it the same way, without you having to go one by one. The **inventory** is the address book with the name, address, and role of each helper (each node).

**Key concepts**

- **Inventory:** the file that says which machines exist and how to connect to them.
- **Playbook:** the recipe of tasks Ansible applies to those machines.
- **Idempotence:** repeating a playbook leaves the same result; it doesn't break what was already correct.
- **Simulation mode (`-C`):** shows what would change without touching anything.
- **SSH (Secure Shell):** the encrypted channel Ansible uses to reach each node.

**Hands-on challenge (safe, read-only):** verify Ansible reaches the three nodes and see what a playbook would change without applying it.

```
ansible -i ansible/inventory.ini incus_cluster -m ping
ansible-playbook -i ansible/inventory.ini ansible/playbook-discovery.yml -C
```

What is an inventory for?

So Ansible knows which machines to manage, how to reach them, and which group each belongs to. It is the source of truth for the nodes.

Why can an idempotent playbook be repeated safely?

Because each task checks the current state first and only changes what is missing. If everything is already correct, it does nothing.

What is the difference between running a playbook and running it with `-C`?

With `-C` Ansible simulates: it reports what would change but does not modify the machines.

Questions? Use the **Learn with AI** button next to each heading, or the [Learn](https://homelab.maceo.cl/en/aprende/index.md) page.

## Stack for this phase

```
flowchart LR
  Op[Operador] --> Repo[Repo_homelab]
  Op --> Ansible[Ansible]
  Ansible --> Inv[inventory.ini]
  Ansible --> Disc[playbook_discovery]
  Disc --> Reports[reports_md]
```

- **[Ansible](https://docs.ansible.com/)** — Bootstrap orchestrator; Phases 0–3.
- **[Debian](https://www.debian.org/)** — OS on invincible, oliver, and deborah.

Deep dive: [Hardware discovery](https://homelab.maceo.cl/en/hardware/discovery/index.md) · [Inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md)

## Before you start

- Three nodes accessible via SSH (even if still on DHCP is fine).

- Python 3.11+ on your workstation.

- SSH key configured to the nodes.

- **GitHub:** the **`maceosoft`** organization created, with a team **`devops`** (its members are who can access Argo CD) and an account that is **owner** of the organization to create the Apps. Details in [maceosoft Platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/index.md). Creating the organization is **web-only**; the team can be created either way:

  In the org: **Teams** → **New team** → name `devops`, visibility *Visible* → **Create team**, then **Add a member** for each person who should access Argo CD.

  Requires org admin permission (`gh auth refresh -s admin:org`, one time). Whoever creates the team becomes its *maintainer*; others are added with the second command:

  ```
  gh api orgs/maceosoft/teams -f name=devops -f privacy=closed
  gh api -X PUT orgs/maceosoft/teams/devops/memberships/<usuario> -f role=member
  ```

- **1Password:** desktop app installed, with a vault named **`HomeLab`** and *Settings → Developer → Integrate with other apps* enabled. Credentials used by playbooks are stored there ([which items and fields](https://homelab.maceo.cl/en/maceosoft/github/#1password-items)). App integration is enabled **only from the app**; the vault can be created either way:

  In the 1Password app: sidebar → **+** next to vaults (*New Vault*) → name `HomeLab` → **Create**.

  With the app unlocked and integration already enabled (if you have more than one account, export `OP_ACCOUNT` first):

  ```
  op vault create HomeLab
  op vault list
  ```

## Run

### Clone repo and Ansible environment

```
git clone https://github.com/maceosoft/homelab.git
cd homelab
python3 -m venv .venv
source .venv/bin/activate
# Ansible, mkdocs, ansible-lint, etc.
pip install -r requirements.txt
ansible-galaxy collection install -r ansible/requirements.yml
```

requirements.txt path

`requirements.txt` lives in the **repo root**, not in `ansible/` — it's a common mistake to confuse it with `ansible/requirements.yml` (that's the Galaxy *collections* manifest, a different YAML file, not something `pip install -r` can install).

### Day 0 — freshly installed nodes (only the first time per node)

A freshly flashed node (Debian netinst) still doesn't have `python3` or `sudo` ready, and the admin user isn't in the `sudo` group — the `ping` below (uses the `ansible.builtin.ping` module, which does need Python on the target) will fail without this step first:

```
cd ansible
ansible-playbook -i bootstrap/inventory.ini bootstrap/setup_sudo.yml --ask-pass
cd ..
```

**Which user gets `sudo`.** The playbook grants passwordless `sudo` to the admin user you created during Debian install. By default it's the **same name you use on your workstation**, because Ansible connects later with that name. If the node uses a different name, specify it and tell Ansible how to connect:

```
ansible-playbook -i bootstrap/inventory.ini bootstrap/setup_sudo.yml --ask-pass -e bootstrap_admin_user=<usuario>
```

and add `ansible_user=<usuario>` to the nodes in `inventory.ini`. The user must already exist on the node (the playbook stops if not) and only allows lowercase, digits, `_` and `-`.

Full detail (why it connects as `root` and where it fits in the rest of the bootstrap) in [`ansible/bootstrap/README.md`](https://github.com/maceosoft/homelab/blob/main/ansible/bootstrap/README.md). If your 3 nodes already have a working admin user with sudo, skip this step.

### Validate inventory

Check [`ansible/inventory.ini`](https://github.com/maceosoft/homelab/blob/main/ansible/inventory.ini): `ansible_host`, `static_ip`, `bridge_iface` per node.

```
ansible -i ansible/inventory.ini incus_cluster -m ping
```

### Optional discovery (recommended first time)

```
cd ansible
ansible-playbook -i inventory.ini playbook-discovery.yml
```

Output in `ansible/reports/` (gitignored): one card per node + `summary.md`.

Collect CPU, RAM, disk, and network with `lscpu`, `free -h`, `lsblk`, `ip -br link` and compare with [Hardware inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md).

### GitHub and 1Password — maceosoft org (manual, one time only)

Here you prepare, once, the GitHub accounts and credentials the HomeLab needs: the Apps the cluster uses to talk to GitHub (including the one Argo CD uses to read the private repo `maceosoft/gitops`) and the base repos (`gitops`, `core-pipelines`, and `infra`). Everything is created by hand, and credentials are stored in 1Password so playbooks can read them from there later. The details for each one (permissions, where to find each piece of data in GitHub, and how to load it into 1Password) are in [maceosoft Platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/index.md).

1Password items go in the **`HomeLab`** vault, almost all as **Secure Note** with custom fields. The item title and each field name must be **exact**, because the playbook looks them up by name: [how to create the item](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields).

With an **organization owner** account `maceosoft`:

1. **GitHub App `maceosoft-terraform`** → item `maceosoft-terraform`. [Permissions and where to find each datum](https://homelab.maceo.cl/en/maceosoft/github/#maceosoft-terraform). Creating the App, generating its private key (`.pem`), and installing it are done **only from the web**; saving its data in 1Password can be done either way:

   Secure Note in vault `HomeLab` with fields `app_id` (Text), `installation_id` (Text), and `private_key` (Password, the full `.pem`).

   `gh api` gives you the `app_id` and `installation_id` of the installed App (once before: `gh auth refresh -s admin:read:org`), and the `.pem` is read from the file, so no value ends up written in the command:

   ```
   APP=maceosoft-terraform
   read -r APP_ID INST_ID < <(gh api orgs/maceosoft/installations \
     --jq ".installations[] | select(.app_slug==\"$APP\") | \"\(.app_id) \(.id)\"")
   op item create --category "Secure Note" --vault HomeLab --title "$APP" \
     "app_id[text]=$APP_ID" "installation_id[text]=$INST_ID" \
     "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
   ```

1. **GitHub App `maceosoft-arc-runners`** → item `maceosoft-arc-runners`. [Permissions and steps](https://homelab.maceo.cl/en/maceosoft/github/#maceosoft-arc-runners). Same as above: the App is created and installed **only from the web**.

   The same three fields but with this App's data: `app_id` (Text), `installation_id` (Text), and `private_key` (Password), in a Secure Note in vault `HomeLab`.

   ```
   APP=maceosoft-arc-runners
   read -r APP_ID INST_ID < <(gh api orgs/maceosoft/installations \
     --jq ".installations[] | select(.app_slug==\"$APP\") | \"\(.app_id) \(.id)\"")
   op item create --category "Secure Note" --vault HomeLab --title "$APP" \
     "app_id[text]=$APP_ID" "installation_id[text]=$INST_ID" \
     "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
   ```

1. **OAuth (Open Authorization) App `Maceosoft Dex`** → item `maceosoft-dex`. [Steps](https://homelab.maceo.cl/en/maceosoft/github/#oauth-app-maceosoft-dex). Creating the OAuth App and generating its *client secret* is **only from the web** (GitHub has no API for OAuth Apps).

   Fields `GITHUB_CLIENT_ID` (Text) and `GITHUB_CLIENT_SECRET` (Password) in item `maceosoft-dex` (Secure Note, vault `HomeLab`).

   Paste each value when prompted; the secret is not shown on screen or saved in history:

   ```
   # pega el Client ID y Enter
   read -r  GH_CID
   # pega el Client secret y Enter
   read -rs GH_CSEC
   op item create --category "Secure Note" --vault HomeLab --title maceosoft-dex \
     "GITHUB_CLIENT_ID[text]=$GH_CID" "GITHUB_CLIENT_SECRET[password]=$GH_CSEC"
   unset GH_CID GH_CSEC
   ```

1. **Private repos**, with their content pushed via git:

   - `gitops`.
   - `core-pipelines`: accessible from other repos in the org. No need to create tags: while testing, other repos use it with `@main`.
   - `infra`.

   Create each repo in the org with **New repository** (private) and push its content with git. In `core-pipelines`, under *Settings → Actions → General → Access*, choose "Accessible from repositories in the 'maceosoft' organization".

   From each project's folder:

   ```
   gh repo create maceosoft/gitops --private --source . --remote origin --push
   # igual para core-pipelines e infra
   gh api -X PUT repos/maceosoft/core-pipelines/actions/permissions/access -f access_level=organization
   ```

1. **GitHub App `maceosoft-argocd`** (read-only, installed **only in `gitops`**) → item `maceosoft-argocd`. Comes after the repos because it installs into `gitops`. [Permissions and steps](https://homelab.maceo.cl/en/maceosoft/github/#step-2-repos-argocd-app-and-ftp-manual). The App is created and installed **only from the web**.

   The same three fields: `app_id` (Text), `installation_id` (Text), and `private_key` (Password), in a Secure Note in vault `HomeLab`.

   ```
   APP=maceosoft-argocd
   read -r APP_ID INST_ID < <(gh api orgs/maceosoft/installations \
     --jq ".installations[] | select(.app_slug==\"$APP\") | \"\(.app_id) \(.id)\"")
   op item create --category "Secure Note" --vault HomeLab --title "$APP" \
     "app_id[text]=$APP_ID" "installation_id[text]=$INST_ID" \
     "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
   ```

1. **Sealed Secrets key** *(optional until Phase 6)*: a key pair generated with `openssl`, in item `sealed-secrets`, so it survives a K3s reinstall. [Command and steps](https://homelab.maceo.cl/en/maceosoft/github/#step-2-repos-argocd-app-and-ftp-manual).

   Secure Note in vault `HomeLab` with `certificate` (Text, all of `tls.crt`) and `private_key` (Password, all of `tls.key`).

   Generate the pair in a temp folder, save it by reading the files, and delete them:

   ```
   cd "$(mktemp -d)"
   openssl req -x509 -days 3650 -nodes -newkey rsa:4096 \
     -keyout tls.key -out tls.crt -subj "/CN=sealed-secret/O=sealed-secret"
   op item create --category "Secure Note" --vault HomeLab --title sealed-secrets \
     "certificate[text]=$(cat tls.crt)" "private_key[password]=$(cat tls.key)"
   rm -f tls.key tls.crt
   ```

1. **FTP (File Transfer Protocol) credentials for repo `api`**: `api` is the first project in the repo map and deploys via FTP, so it needs its item `ftp-api` (Secure Note) from the first pipeline. Every repo you add afterward follows the [routine procedure — Add a repo](https://homelab.maceo.cl/en/operacion/agregar-repo/index.md). [Table with examples](https://homelab.maceo.cl/en/maceosoft/github/#step-2-repos-argocd-app-and-ftp-manual).

   The item has 8 fields: `dev_host`, `dev_user`, `dev_password`, `dev_remote_dir`, and the same four with `prod_`. The `*_password` go as Password and the rest as Text.

   Change hosts, users, and folders to yours; the two passwords are pasted without echo:

   ```
   # pega cada contraseña y Enter
   read -rs FTP_DEV_PASS;  read -rs FTP_PROD_PASS
   op item create --category "Secure Note" --vault HomeLab --title ftp-api \
     "dev_host[text]=dev.tu-cpanel.example.com" "dev_user[text]=usuario_dev" \
     "dev_password[password]=$FTP_DEV_PASS" "dev_remote_dir[text]=/home/usuario/dev.maceosoft.example.com" \
     "prod_host[text]=tu-cpanel.example.com" "prod_user[text]=usuario_prod" \
     "prod_password[password]=$FTP_PROD_PASS" "prod_remote_dir[text]=/home/usuario/public_html"
   unset FTP_DEV_PASS FTP_PROD_PASS
   ```

All commands to create these items with the `op` CLI (and the GitHub ones with `gh`) are in the [maceosoft Platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/index.md) sections: the Apps and the OAuth App are **created only from the web**, and from there on saving data to 1Password, listing installations, and creating repos is done with `op` and `gh`.

Verify that all credentials can be read from 1Password. Two commands, and neither touches a cluster nor reveals secret values:

- **`--check`**: checks each expected item and marks fields that are correct with ✓ and those that are missing, have a different name, or are inside a section with ✗. Run it first: if something is wrong, it tells you exactly what.
- **`--dry-run`**: reads all values and lists the Secrets that will be created in the cluster, with hidden values (`<redacted>`). Confirms that the keys can be interpreted.

The script reads 1Password via the desktop app, so the app must be **unlocked** and have *Settings → Developer → Integrate with other apps* enabled. You also need to tell it **which account** to use with the **`ONEPASSWORD_ACCOUNT_NAME`** variable: it's your account name as it appears in the top left of the app sidebar (for example `Mi Cuenta`). If you don't define it, it uses `Personal`.

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
.venv/bin/python ansible/scripts/apply_platform_secrets.py --check
.venv/bin/python ansible/scripts/apply_platform_secrets.py --dry-run
```

`--check` exits successfully when all items have ✓ (the notice that there are no items `ftp-<repo>` is only informational if no repo deploys).

## Options

| Option           | When                                     |
| ---------------- | ---------------------------------------- |
| Skip discovery   | Re-deployment and hardware already known |
| Update inventory | After changing IPs in Phase 1            |

Trade-off analysis

Ansible Discovery

Measured RAM/disk/I/O → `reports/`

Requires SSH and a few minutes per node

Manual Discovery

No playbook

Outdated values; planning errors

Skip Discovery

Faster on re-deployments

RAM/disk budget may not match reality

## Verify

```
ansible -i ansible/inventory.ini incus_cluster -m ping
# todos → SUCCESS

# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
.venv/bin/python ansible/scripts/apply_platform_secrets.py --check
# todos los items con ✓
.venv/bin/python ansible/scripts/apply_platform_secrets.py --dry-run
# lista los Secrets de plataforma (valores <redacted>), sin errores de 1Password
```

## If it fails

| Symptom                                       | Check                                                                                                                                                                                                                                                                                                                       |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `UNREACHABLE`                                 | `ansible_host`, firewall, SSH key                                                                                                                                                                                                                                                                                           |
| `ping` fails due to missing Python            | Freshly installed node — run `setup_sudo.yml` ("Day 0" step above)                                                                                                                                                                                                                                                          |
| Discovery without reports                     | Run from `ansible/`; write permissions                                                                                                                                                                                                                                                                                      |
| `--check` or `--dry-run` don't read 1Password | App unlocked with *Integrate with other apps* and `ONEPASSWORD_ACCOUNT_NAME` set to your account name. If it says *field cannot be found* or *Vault not found*, check the output of `--check`: it marks which item or field doesn't match ([expected names](https://homelab.maceo.cl/en/maceosoft/github/#1password-items)) |

## Next

**[→ Phase 1 — Network](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md)**

# Phase 1 — Network

## Objective

Bridge `br0` with **static IP** on the 3 nodes, ready for Incus, K3s (a lightweight Kubernetes distribution), and CAPN (Cluster API Provider for Incus).

## What You Learn

Why the Homelab requires a fixed IP: Incus records `cluster.https_address`, K3s uses the IP as the kubelet identity, and CAPN references `https://<IP>:8443` in secrets.

Learn this phase

**In simple terms:** a fixed IP is your postal address: if it changes every day, no one can find you. A **bridge** is like a virtual multi-outlet: it lets the machine and virtual machines share the same network connection.

**Key Concepts**

- **Static or dynamic IP:** A static IP stays the same; a dynamic one is assigned by the router (DHCP, Dynamic Host Configuration Protocol) and can change.
- **Mask and CIDR (Classless Inter-Domain Routing):** The notation `/22` indicates how many addresses share the network.
- **Gateway:** The device, usually the router, through which you exit to other networks.
- **DNS (Domain Name System):** The "phone book" that translates names to IP addresses.
- **Bridge `br0`:** The virtual interface that the node, Incus, and K3s share.

**Practical challenge (read-only):** On a node, find its IP, its mask, and its gateway.

```
ip -br addr show br0
ip route | grep default
```

Why does the Homelab need a fixed IP on the nodes?

Because Incus stores the cluster address, K3s uses the IP as the node identity, and other components reference it: if it changes, the cluster will stop finding itself.

What is a bridge and what is it for here?

It is a virtual interface that connects several interfaces into a single network. It allows Incus instances and pods to use the node's physical network.

In `192.168.20.5/22`, which part is the address and which is the mask?

`192.168.20.5` is the address and `/22` indicates that the first 22 bits are the network.

Questions? Use the **Learn with AI** button next to each title, or the page [Learn](https://homelab.maceo.cl/en/aprende/index.md).

## Stack for this phase

```
flowchart TB
  PHY[Interfaz_fisica] --> BR0[br0_L2]
  BR0 --> IP[IP_estatica]
  IP --> SSH[ansible_host]
  IP --> Incus[Instancias_Incus]
  IP --> K3s[kubelet_node_ip]
```

- **[Netplan](https://netplan.io/)** — Declarative network configuration on Debian; Phase 1.
- **[Ansible](https://docs.ansible.com/)** — Applies Netplan idempotently via `playbook-set-static-ip.yml`.
- Role: [`netplan_bridge`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/netplan_bridge/README.md)

Further reading: [Network — Netplan examples](https://homelab.maceo.cl/en/networking/index.md)

## Before You Start

- [Phase 0 — Preparation](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/index.md) completed (`ansible ping` OK).
- Know the physical interface per node (`eno1`, `eno2`, `enP4p65s0`).
- Target IPs defined in inventory.

| Node       | Interface   | IP on br0    |
| ---------- | ----------- | ------------ |
| invincible | `eno2`      | 192.168.20.6 |
| oliver     | `eno1`      | 192.168.20.7 |
| deborah    | `enP4p65s0` | 192.168.20.5 |

> Confirm always with `ip -br link` on the node before assuming these values.

## Execute

SSH Disconnect

If you change the IP, Ansible may lose the session. The playbook uses `serial: 1` — one node at a time. Update `ansible_host` after each node if the connection IP changed.

```
cd ansible
# Un nodo a la vez si la IP de conexión cambia
ansible-playbook -i inventory.ini playbook-set-static-ip.yml \
  --limit invincible
# Actualiza ansible_host en inventory.ini si hace falta, luego:
ansible-playbook -i inventory.ini playbook-set-static-ip.yml \
  --limit oliver
ansible-playbook -i inventory.ini playbook-set-static-ip.yml \
  --limit deborah
```

See YAML examples in [Network](https://homelab.maceo.cl/en/networking/index.md):

```
sudo apt install -y netplan.io
sudo netplan apply
```

## Options

| Decision   | Default Homelab             |
| ---------- | --------------------------- |
| Node order | One at a time (`serial: 1`) |

All three nodes change IP upon applying

No shortcut: update `ansible_host` in `inventory.ini` after each node (that's why `serial: 1`).

Trade-off analysis

Ansible (`playbook-set-static-ip.yml`)

Idempotent; Netplan template per host

May disconnect SSH if IP changes

Manual Netplan

Full control over YAML

No automatic rollback; easy to misalign the three nodes

`serial: 1`

Node-by-node recovery

Slower than three in parallel

## Verify

```
ansible -i ansible/inventory.ini incus_cluster -m command -a "ip -br addr show br0"
# Cada nodo muestra su IP estática en br0
ping -c 2 192.168.20.5
ping -c 2 192.168.20.6
ping -c 2 192.168.20.7
```

## If It Fails

| Symptom              | Check                                                              |
| -------------------- | ------------------------------------------------------------------ |
| Lost SSH after apply | Connect by new IP; update `inventory.ini`                          |
| No `br0`             | Interface in `bridge_iface`; logs `journalctl -u systemd-networkd` |

## DNS (BIND on deborah)

The three nodes use `192.168.20.5` (deborah) as the primary DNS and the gateway `192.168.20.1` as the secondary. With the fixed IP already applied on deborah, install and configure BIND (Berkeley Internet Name Domain) there (runs only on deborah):

```
ansible-playbook -i inventory.ini playbook-bind-dns.yml
```

Until it runs, the nodes resolve by the gateway, with some delay in each query.

Two things that `playbook-set-static-ip.yml` leaves and that subsequent phases depend on:

- A fixed `/etc/resolv.conf` in each node, with `search homelab.local mco.local` and first the BIND (`192.168.20.5`). If the node has NetworkManager, install also `dns=none` to prevent it from rewriting with the router's DNS. Without this, a node stops resolving `*.homelab.local` (the nodes cannot validate the SSO —single sign-on— of the [4.12](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#412-ssh-to-hosts-with-dex-optional)).
- The BIND zone `homelab.local` brings a record for each node and each published service, without wildcard (see [4.8](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#48-homelablocal-dns)).

Check on each node: `cat /etc/resolv.conf` and `getent hosts oliver.homelab.local`. Details: [`ansible/roles/bind_dns/README.md`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/bind_dns/README.md).

## Hardening (recommended before proceeding)

With the IP already fixed, it's time to harden the host before it joins the cluster. This goes **here** in the sequence (see [Post-incident hardening](https://homelab.maceo.cl/en/security/hardening-post-incidente/index.md)):

```
ansible-playbook -i inventory.ini playbook-hardening.yml --check --diff
ansible-playbook -i inventory.ini playbook-hardening.yml
```

Details: [`ansible/roles/host_hardening/README.md`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/host_hardening/README.md).

## Next

**[→ Phase 2 — Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md)**

Complete bootstrap order (includes hardening, BIND, WiFi): [`ansible/bootstrap/README.md`](https://github.com/maceosoft/homelab/blob/main/ansible/bootstrap/README.md).

# Phase 2 — Incus

## Objective

**Incus** cluster of 3 nodes with quorum, cluster groups by architecture, and `oliver` in manual scheduler.

## What you learn

Incus as a **shared hypervisor**: dqlite quorum across 3 members, cluster groups to steer x86 vs ARM64 (64-bit ARM architecture) workloads, and manual scheduler to protect RAM.

Learn this phase

**In short:** Incus is an apartment building. Each apartment is an **instance**: a container (a lightweight room sharing the building's utilities) or a virtual machine (a full apartment with its own kitchen). With three coordinated buildings, if one closes the other two keep running.

**Key concepts**

- **Container and virtual machine:** the container is lighter; the VM (Virtual Machine) isolates more.
- **Cluster and quorum:** with 3 members, the cluster decides by majority and tolerates the loss of one.
- **Cluster group:** a label to direct workloads to a node type (for example, ARM64 only).
- **x86_64 and ARM64:** two distinct processor architectures; they don't mix in the same instance.
- **Manual scheduler:** the `oliver` node does not receive instances automatically; protects its memory.

**Practical challenge (read-only):** view cluster members and their instances.

```
incus cluster list
incus list -c ns4
```

Why a three-member cluster and not two?

With two, if one falls there is no majority and the cluster blocks. With three, the majority is two and it can keep running.

When do you prefer a container and when a VM?

Container for light, fast Linux workloads; VM when you need another kernel or more isolation.

What are cluster groups for?

To decide on which nodes an instance runs, for example keeping ARM64 workloads on `deborah`.

Questions? Use the **Learn with AI** button next to each heading, or the [Learn](https://homelab.maceo.cl/en/aprende/index.md) page.

## Stack for this phase

```
flowchart TB
  subgraph cluster [Incus_cluster_3_nodos]
    INV[invincible_x86]
    NOL[oliver_x86_quorum]
    DEB[deborah_arm64]
  end
  INV --> CGx86[x86_nodes]
  DEB --> CGarm[arm64_nodes]
  NOL --> CGx86
```

- **[Incus](https://linuxcontainers.org/incus/docs/main/)** — LXC virtualization (Linux system containers)/VM; Phase 2.
- **[Zabbly Incus stable](https://github.com/zabbly/incus)** — Debian packages; Phase 2.
- **[Ansible](https://docs.ansible.com/)** — Installs package, bootstrap, join, groups, UI (user interface).
- Roles: [`incus_install`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_install/README.md), [`incus_cluster`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_cluster/README.md), [`incus_ui`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_ui/README.md)

Deep dive: [Incus Cluster](https://homelab.maceo.cl/en/incus/cluster-setup/index.md)

**This decision in other environments**

An Incus cluster with 3 members (dqlite quorum) on hardware you already had, free and with containers and VMs in the same tool. If a member falls, the other two keep deciding; `oliver` only votes and receives no workloads.

- **Benefits:** a virtualization cluster pools several servers into a single resource: move VMs between nodes for maintenance without cutting service, restart workloads on another node if one fails, and manage everything from a single API or console, with permissions and audit trail of who did what.
- **Commercial products:** the most common is [VMware vSphere](https://www.vmware.com/products/cloud-infrastructure/vsphere) (now Broadcom, with licensing changes driving many companies to seek alternatives), [Nutanix AHV](https://www.nutanix.com/products/ahv) or [Proxmox VE](https://www.proxmox.com/en/products/proxmox-virtual-environment/overview) with support subscription. For large private clouds, [OpenStack](https://www.openstack.org/).
- **Shared storage:** instead of the local `dir` driver, shared or distributed storage (Ceph, SAN) is used, so a VM can move or start on another node without copying its disk.
- **Incus support:** Incus also has commercial support ([Zabbly](https://zabbly.com/), [FuturFusion](https://futurfusion.io/)); the real cost is in the people who know how to operate it.
- **When Incus makes sense:** escaping expensive VMware licenses, edge or branch sites with few servers, and labs or CI that need to create and destroy many instances quickly.

| Cloud              | Equivalent service                                                                    |
| ------------------ | ------------------------------------------------------------------------------------- |
| GCP                | [Compute Engine](https://cloud.google.com/products/compute)                           |
| AWS                | [Amazon EC2](https://aws.amazon.com/ec2/)                                             |
| Azure              | [Azure Virtual Machines](https://azure.microsoft.com/en-us/products/virtual-machines) |
| Oracle Cloud (OCI) | [OCI Compute](https://www.oracle.com/cloud/compute/)                                  |

You get VMs instantly, without buying or maintaining hardware, with SLA and in many regions. You lose control over the hypervisor, pay for every hour the VM is running, and are locked into that provider's network and images.

## Before you start

- [Phase 1 — Network](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md) completed (static IPs in `br0`).
- `br0` operational on all 3 nodes.
- ≥ **20 GB free** on each node (see storage requirements).

## Storage requirements

Default HomeLab: driver **`dir`** (directory on the OS filesystem, no dedicated disk). Variables in [`group_vars/incus_cluster/vars.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/vars.yml).

| Aspect             | Value / requirement                                                     |
| ------------------ | ----------------------------------------------------------------------- |
| Driver             | `dir` — no dedicated partition or loop                                  |
| Pool per node      | `local` (each member has its local pool)                                |
| Pool path          | `/var/lib/incus/storage-pools/local`                                    |
| Filesystem         | ext4 or xfs; **not** NFS/CIFS as backend                                |
| Minimum free space | **20 GB** per node; **50 GB+** if you will create VMs or many instances |
| Metadata           | dqlite and daemon config share disk with the pool                       |

### Per node

| Node         | Disk                                         | Implication                                                                                        |
| ------------ | -------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `invincible` | Kingston 224 GB SSD                          | Leader + UI; largest pool of the x86 nodes                                                         |
| `oliver`     | TECLAST 120 GB SSD                           | Quorum; local pool mandatory even if it receives no instances; avoid filling disk with CAPN images |
| `deborah`    | eMMC 233 GB (NVMe 250 GB not detected today) | If eMMC tight: `incus_storage_path: /srv/incus/storage-pools/local` in host_vars                   |

### Limitations of `dir`

- No pool-level copy-on-write (instance snapshots and export tarball do work).
- VMs (virtual machines) slower than with `zfs`/`btrfs`/`lvm`.
- No thin provisioning or native compression.
- Each member maintains its own `local` pool; no distributed storage like Ceph.

Deep dive: [Incus Cluster — Storage](https://homelab.maceo.cl/en/incus/cluster-setup/#storage)

## Run

### Step 1 — Install Incus package on all 3 nodes

```
cd ansible
ansible-playbook -i inventory.ini playbook-bootstrap.yml
```

On each node:

```
curl https://pkgs.zabbly.com/get/incus-stable | sudo bash -x
```

### Step 2 — Bootstrap, join, groups, scheduler, and UI

A single playbook: bootstrap on **invincible**, join on **oliver** and **deborah**, cluster groups, manual scheduler, and UI.

```
cd ansible
ansible-playbook -i inventory.ini playbook-incus-cluster.yml
```

Optional tags: `--tags bootstrap`, `--tags join`, `--tags groups`, `--tags ui`.

On **invincible** (bootstrap):

```
sudo incus admin init
# clustering=yes, no te unes (bootstrap), storage backend dir
```

On **invincible**, generate tokens:

```
incus cluster add oliver
incus cluster add deborah
```

On **oliver** and **deborah**: `incus admin init` → join with token.

Cluster groups and scheduler:

```
incus cluster group create x86-nodes
incus cluster group create arm64-nodes
incus cluster group assign invincible x86-nodes,default
incus cluster group assign oliver x86-nodes,default
incus cluster group assign deborah arm64-nodes,default
incus cluster set oliver scheduler.instance manual
```

UI on invincible:

```
sudo apt install incus-ui-canonical
sudo incus config set core.https_address=:8443
```

## Network profile (instances on LAN)

**Mandatory manual step** — neither the Ansible playbook nor `incus admin init` do this automatically, in neither of the two paths above. Without this, Incus instances remain on the default private NAT (Network Address Translation) network, not on your LAN (local network).

On **invincible** (profiles are shared across the whole cluster, no need to repeat on oliver/deborah):

```
incus profile device add default eth0 nic nictype=bridged parent=br0
```

Verify:

```
# tiene que mostrar eth0 → parent br0
incus profile show default
```

## Admin UI

Ansible installs `incus-ui-canonical` and `core.https_address` on **invincible** (play `ui` of the playbook). Initial access via **client certificate**; SSO (single sign-on) (Dex) in [Phase 4](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#incus-ui-oidc).

1. Add `incus.homelab.local` in `/etc/hosts` (full block in [HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/#dns-on-your-workstation)).
1. Open [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443)
1. Accept the self-signed certificate
1. Generate/import a **client certificate**

OIDC with Dex

SSO login (GitHub via Dex) is configured in **[Phase 4 — Incus UI OIDC (OpenID Connect)](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#incus-ui-oidc)**. Only the `devops` team from `maceosoft` enters (Dex filter) and, with Incus default configuration, those OIDC users have full access.

Deep dive: [Incus Cluster — UI](https://homelab.maceo.cl/en/incus/cluster-setup/#admin-ui)

## Backup and snapshots (optional)

Not configured in bootstrap. The procedure is in [Incus Backups](https://homelab.maceo.cl/en/operacion/backups-incus/index.md); the alternatives, in [Backup and snapshots](https://homelab.maceo.cl/en/incus/cluster-setup/#backup-and-snapshots-optional).

## Options

| Decision        | Default HomeLab             | Alternative                                                   |
| --------------- | --------------------------- | ------------------------------------------------------------- |
| Storage backend | `dir` (Ansible)             | `btrfs`, `zfs`, `lvm` via `incus_storage_driver` in host_vars |
| oliver          | `scheduler.instance manual` | —                                                             |
| Pool on deborah | Default path on eMMC        | `incus_storage_path` on NVMe (`/srv/incus/...`)               |

Change driver in existing cluster

An already initialized cluster **cannot** be re-initialized with a preseed without manual migration. Choose the driver before the first `playbook-incus-cluster.yml`.

Trade-off analysis

Ansible (`playbook-incus-cluster.yml`)

Idempotent preseed, tokens, and UI in one step

Requires Ansible and SSH to all 3 nodes

Manual (`incus admin init`)

No playbook; useful for debugging a node

Single-use tokens; easy to drift versions

Storage `dir`

Zero friction, 20 GB free is enough

Slow VMs; no pool CoW

`zfs` / `btrfs`

Snapshots and better I/O

Dedicated disk/loop; cannot change after bootstrap

Pool on eMMC (deborah)

No NVMe mount needed

Risk of filling eMMC with images

Pool on NVMe (`/srv/incus`)

More space and I/O

Extra config in `host_vars`

## Verify

```
incus cluster list
# 3 miembros ONLINE
incus cluster group list
incus storage list
# eth0 → parent br0
incus profile show default
curl -kI https://incus.homelab.local:8443
```

## If it fails

| Symptom                 | Check                                                |
| ----------------------- | ---------------------------------------------------- |
| Join rejected           | Single-use token; stable IP in `br0`                 |
| Broken quorum           | Incus versions aligned on all 3 nodes                |
| Insufficient space      | `df -h`; minimum 20 GB (`incus_storage_min_free_gb`) |
| Playbook not idempotent | Node already clustered; check `incus cluster list`   |

## Next

**[→ Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)**

URLs and global `/etc/hosts`: [HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md) (after Phase 4).

# Phase 3 — K3s

## Objective

Set up a **K3s (lightweight Kubernetes distribution)** management cluster on 3 nodes (control plane on deborah, workers on invincible and oliver), with **Argo CD** controller ready for GitOps.

## What You Learn

K3s as a lightweight bare-metal Kubernetes: server/agent, CNI (Container Network Interface) configurable (Flannel by default in K3s; in this homelab, Calico), disable Traefik/ServiceLB to control the Gateway and LoadBalancer via GitOps later.

Learn this phase

**In simple terms:** Kubernetes is the manager of a restaurant: you tell it "I want three cooks working at all times" and it takes care of replacing any that are missing. The **control plane** is the management (decides) and the **workers** are the kitchen (work). K3s is a version of Kubernetes designed for small teams.

**Key concepts**

- **Node:** a machine in the cluster; can be control plane (decides) or worker (executes).
- **Pod:** the smallest unit of execution: one or more containers that live together.
- **Deployment:** declares how many copies of a pod should exist and keeps them running.
- **CNI:** the component that provides network to pods; here, Calico.
- **`kubectl` and kubeconfig:** the command-line tool and the file with the cluster access data.

**Practical challenge (safe):** create a test deployment, delete one of its pods, and watch Kubernetes replace it.

```
kubectl get nodes -o wide
kubectl create deployment demo --image=nginx --replicas=2
kubectl get pods -l app=demo
kubectl delete pod -l app=demo --wait=false
# aparecen pods nuevos
kubectl get pods -l app=demo
# limpieza
kubectl delete deployment demo
```

What is the difference between a pod and a deployment?

A pod is a running copy; a deployment is the rule that says how many copies should exist and recreates them if they disappear.

What happens if a worker fails?

Kubernetes detects that the pods have stopped responding and recreates them on other healthy nodes.

Why is a CNI required?

Because Kubernetes does not come with its own pod network in this setup: the CNI provides addresses and inter-node communication.

Have questions? Use the **Learn with AI** button next to each title, or the page [Learn](https://homelab.maceo.cl/en/aprende/index.md).

## Stack for this phase

```
flowchart TB
  subgraph cp [Control_plane]
    DEB[deborah_server]
    VIP[kube_vip_opcional]
  end
  subgraph workers [Workers]
    INV[invincible_agent]
    NOL[oliver_agent]
  end
  DEB --> API[API_6443]
  VIP -.-> API
  INV --> API
  NOL --> API
  DEB --> ArgoCD[ArgoCD_controller]
```

- **[K3s](https://docs.k3s.io/)** — Management cluster; Phase 3.
- **CNI** — [Flannel](https://github.com/flannel-io/flannel) (default), [Canal](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/flannel), [Calico](https://docs.tigera.io/calico/latest/about/), [Cilium](https://docs.cilium.io/); Phase 3.
- **[kube-vip](https://kube-vip.io/)** — Optional VIP (virtual IP) `:6443`; fork Phase 3.
- **[Argo CD](https://argo-cd.readthedocs.io/)** — Controller bootstrap; Phase 3 → GitOps Phase 4.
- **[Ansible](https://docs.ansible.com/)** — `playbook-k3s.yml`.

Further reading: [Playbook options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md) · [Ansible role catalog](https://homelab.maceo.cl/en/ansible/index.md) · [K3s manual (legacy)](https://homelab.maceo.cl/en/k3s/index.md)

**This decision in other environments**

A single control plane on deborah, with the default K3s datastore (SQLite), and two workers. It is simple, uses little memory, and is sufficient for learning; if deborah goes down, the pods continue running but you cannot make changes until it comes back up. If you want high availability later, K3s can switch to [embedded etcd](https://docs.k3s.io/datastore/ha-embedded).

- **Benefits:** Kubernetes provides a unique way to deploy, scale, and recover applications on any server; a small team operates many services with the same manifests, and K3s achieves this with a lightweight binary that installs and updates in minutes.
- **High availability:** three control planes are used (an odd number to maintain quorum in etcd), with embedded or [external etcd](https://docs.k3s.io/datastore), behind a virtual IP or a load balancer. SQLite does not work with multiple servers.
- **Zero-downtime updates:** with multiple control planes, one is updated at a time and the API never goes down.
- **Commercial support:** it is common to pay for a distribution with support and security patches, such as [RKE2](https://docs.rke2.io/) within [SUSE Rancher Prime](https://www.suse.com/products/rancher/) or [Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift).
- **When K3s is suitable:** at the edge (stores, branches, factories, IoT), where there is little hardware, little technical staff on-site, or intermittent connectivity. There, a small and easy-to-replace cluster is worth more than a large one.

| Cloud              | Equivalent service                                                                              |
| ------------------ | ----------------------------------------------------------------------------------------------- |
| GCP                | [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine)                    |
| AWS                | [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)                          |
| Azure              | [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service) |
| Oracle Cloud (OCI) | [OCI Kubernetes Engine (OKE)](https://www.oracle.com/cloud/cloud-native/kubernetes-engine/)     |

Gain a managed control plane provided by the cloud provider: high availability, patches, and SLA without you having to manage etcd. You lose control over that control plane, pay for the service, and become more tied to that cloud's integrations.

## Before You Start

- [Phases 0](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/index.md), [1](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md), and [2](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md) completed.
- Healthy Incus cluster (optional but recommended before CAPN —Cluster API Provider for Incus—).

## Run

Playbook options are activated in [`group_vars/incus_cluster/k3s_install.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/k3s_install.yml), not via command line. Today, Argo CD (`gitops_argocd`), node labels (`node_labels`), and local kubeconfig (`fetch_kubeconfig`) are enabled, with CNI Calico. Review the file and run:

```
cd ansible
ansible-playbook -i inventory.ini playbook-k3s.yml
```

Do not use `-e k3s_install.<opción>=…`

Ansible ignores it: `k3s_install` is a dictionary and a key with dots in `-e` does not modify it (the default CNI would be installed without warning). Passing `-e` with JSON is worse: it replaces the entire dictionary and loses `core.version`, `cluster_cidr`, etc. That's why the playbook **fails** in both cases. To change the CNI use `-e k3s_cni=canal` (or edit the file); see [playbook options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md).

Manual curl steps in [K3s — manual installation](https://homelab.maceo.cl/en/k3s/index.md). Use only if Ansible is not available; the supported path is the playbook.

## Options

| If you need…         | In `k3s_install.yml`             | Today    | Doc                                                                           |
| -------------------- | -------------------------------- | -------- | ----------------------------------------------------------------------------- |
| Stable API VIP       | `kube_vip.enabled: true`         | disabled | [playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md) |
| Longhorn/arch labels | `node_labels.enabled: true`      | enabled  | [playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md) |
| Local kubeconfig     | `fetch_kubeconfig.enabled: true` | enabled  | [playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md) |
| OS hardening         | `os_hardening.enabled: true`     | disabled | [playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md) |

| Decision | Option A (default)                                                                                           | Option B                                                                   |
| -------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------- |
| K3s API  | Name `k3s.homelab.local` (variable `k3s_api_endpoint`) to the control plane IP, today deborah `192.168.20.5` | kube-vip `192.168.20.4`                                                    |
| CNI      | Calico (configured in this repo)                                                                             | Flannel `vxlan` (default of K3s), Canal or Cilium (`k3s_install.core.cni`) |

### CNI

| CNI                          | Variable       | Notes                                                  |
| ---------------------------- | -------------- | ------------------------------------------------------ |
| **Flannel** (default of K3s) | `cni: flannel` | Embedded in K3s; `flannel_backend: vxlan` or `host-gw` |
| **Canal**                    | `cni: canal`   | Flannel overlay + Calico policies; role `k3s_cni`      |
| **Calico**                   | `cni: calico`  | NetworkPolicy, optional BGP; `--flannel-backend=none`  |
| **Cilium**                   | `cni: cilium`  | eBPF, Hubble; validate on ARM64 (deborah)              |

The CNI is chosen in `k3s_install.core.cni` (today: `calico`), **before** installing K3s: changing it in an already installed cluster requires reinstalling K3s.

```
# group_vars/incus_cluster/k3s_install.yml
k3s_install:
  core:
    cni: calico  # flannel | canal | calico | cilium
```

Details: [playbook-options — CNI](https://homelab.maceo.cl/en/k3s/playbook-options/#cni).

Trade-off analysis

Ansible (`playbook-k3s.yml`)

Flags in `group_vars`; role `k3s_cni`

Requires SSH and order server → CNI → agents

Manual curl

Escape hatch

No idempotence; easy to forget `--disable traefik`

Fixed IP `192.168.20.5`

No extra components

If the CP migrates, agents need reconfiguration

kube-vip `192.168.20.4`

Stable API VIP

Additional daemon; different from MetalLB

Flannel

Zero friction; low consumption

No native NetworkPolicy

Canal

Calico policies over overlay

Legacy project; two stacks

Calico

Mature NetworkPolicy; BGP at L2

More pods/RAM; IP pools

Cilium

eBPF; L3–L7 policies; Hubble

More CPU; kernel/BPF; test on deborah

`node_labels` / `fetch_kubeconfig`

Longhorn and local kubeconfig ready

Longer play; misapplied labels affect storage

## Verify

```
# En deborah o con kubeconfig
sudo k3s kubectl get nodes
# 3 nodos Ready
sudo k3s kubectl get pods -n argocd
# argocd-server Running (si gitops_argocd.enabled=true)
```

Wait for pods `argocd-server` in `Running` before Phase 4.

## If it fails

| Symptom              | Check                                          |
| -------------------- | ---------------------------------------------- |
| Agent does not join  | Token, `K3S_URL`, firewall `:6443`             |
| Residual Traefik     | `--tags traefik` or role `k3s_cleanup_traefik` |
| CP down after reboot | `systemctl enable --now k3s` in deborah        |

## Next

**[→ Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)**

# Phase 4 — GitOps

## Objective

Deploy the full platform via **Argo CD** from [`gitops/`](https://github.com/maceosoft/gitops): storage, MetalLB, Gateway API, Dex + GitHub OAuth (Open Authorization), and operation waves.

## What you learn

**GitOps**: the cluster converges to the state declared in Git; a single `kubectl apply` (ApplicationSet `homelab-root`) and Argo CD sync the rest. Clear separation: Ansible bootstrap vs continuous GitOps.

Learn this phase

**In short:** Git is the approved recipe and Argo CD is the chef who compares the kitchen with the recipe and corrects any difference. If someone changes something manually in the cluster, Argo CD puts it back as Git says. The team doesn't "run commands": they change the recipe.

**Key concepts**

- **GitOps:** the repository is the source of truth; the cluster converges to what it says.
- **Application and sync:** Argo CD watches a Git folder and applies it; "Synced" means they match.
- **Drift and self-heal:** drift is the difference between Git and the cluster; self-heal corrects it automatically.
- **Waves (sync-waves):** the deployment order; the foundation (network, certificates) goes before applications.
- **Secrets outside Git:** keys live in 1Password and are delivered to the cluster without committing them to the repo.
- **Gateway and TLS (Transport Layer Security):** the HTTPS entry door and the certificate that protects it.

**Hands-on challenge (read-only):** compare the cluster state with what Git declares.

```
kubectl -n argocd get applications
kubectl get gateway homelab -n gateway
```

Open `gateway/kong/gateway.yaml` in the `gitops` repo and look for the same ports and the name `homelab` you see in the cluster.

What does it mean for an Application to be Synced and Healthy?

Synced: what's in the cluster matches Git. Healthy: the resources are working.

Why aren't secrets stored in Git?

Because anyone with repo access would see them, and Git history preserves everything ever committed.

What does self-heal do if someone changes something manually?

It detects the difference from Git and reapplies what's declared.

Questions? Use the **Learn with AI** button next to each heading, or the [Learn](https://homelab.maceo.cl/en/aprende/index.md) page.

**This decision in other environments**

A single [Argo CD](https://argo-cd.readthedocs.io/) inside the cluster itself syncs everything from the `gitops` repo, starting from an ApplicationSet, and secrets live in 1Password. It's free, you manage it, and it teaches you GitOps as used in production.

- **Benefits:** every change goes through a reviewed PR and is audited in Git (who, what, when); rolling back is a `git revert`; people no longer need `kubectl` with write permissions in production, and *drift* (manual changes) is detected and corrected automatically.
- **Multiple clusters and environments:** a central instance (or one per environment) with ApplicationSets per cluster, and promotion from dev to prod also by PR.
- **Corporate identity:** SSO with the company provider (Entra ID, Okta…) and RBAC by team and project, instead of a single GitHub team. Secrets come from a corporate vault instead of a playbook.
- **Commercial support:** [Akuity](https://akuity.io/) (managed Argo CD, from the project creators), [Codefresh](https://octopus.com/codefresh) (now part of Octopus Deploy) or [Red Hat OpenShift GitOps](https://www.redhat.com/en/technologies/cloud-computing/openshift/gitops).
- **When the same approach still makes sense:** with few clusters, in disconnected sites or at the edge (Argo CD *pulls* from Git, so no need to open inbound ports to the cluster) and where you don't want to depend on a SaaS.

| Cloud              | Equivalent service                                                                                                                                                                                                                     |
| ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCP                | [Config Sync](https://docs.cloud.google.com/kubernetes-engine/config-sync/docs/overview) (GKE managed GitOps)                                                                                                                          |
| AWS                | [EKS Capability for Argo CD](https://docs.aws.amazon.com/eks/latest/userguide/argocd.html) (AWS-managed Argo CD)                                                                                                                       |
| Azure              | AKS GitOps extensions and Azure Arc: [Flux](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-flux2) and [Argo CD](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/conceptual-gitops-argocd) |
| Oracle Cloud (OCI) | No managed GitOps service: Oracle documents [installing Argo CD on OKE](https://docs.oracle.com/en/solutions/deploy-gitops-argocd-oke/index.html)                                                                                      |

Open-source Argo CD (or [Flux](https://fluxcd.io/)) runs the same on any of these clouds. With the managed version you gain the provider installing, updating, and integrating it with their identity; you lose control over versions and tuning, and you're locked into their rules (for example, EKS only deploys to EKS clusters).

## Stack for this phase

```
flowchart LR
  subgraph wave0 [Wave_0]
    GA[Gateway_API_CRDs]
    CM[cert_manager]
    SS[Sealed_Secrets]
    OB[OpenEBS]
    SC[StorageClasses]
    ML[MetalLB]
  end
  subgraph wave1 [Wave_1]
    MLC[MetalLB_config]
    CMC[CA_del_HomeLab]
    GW[Gateway_Kong]
    LH[Longhorn]
  end
  subgraph wave2 [Wave_2]
    AR[ArgoCD_HTTPRoute]
    ACFG[ArgoCD_Dex]
  end
  subgraph wave3 [Wave_3]
    UPG[K3s_upgrade]
  end
  subgraph manual [Manual]
    VC[vCluster]
    CAPN[CAPN_demo]
  end
  root[root_app] --> wave0 --> wave1 --> wave2 --> wave3
  wave2 -.-> manual
```

- **[Argo CD](https://argo-cd.readthedocs.io/)** — GitOps; the entire phase.
- **[MetalLB](https://metallb.universe.tf/)** — LAN LoadBalancer (local network); wave 0–1.
- **[Gateway API](https://gateway-api.sigs.k8s.io/)** — Standard Kubernetes API for exposing HTTP(S) services (replaces `Ingress`); its CRDs go in wave 0.
- **[Kong Ingress Controller](https://developer.konghq.com/kubernetes-ingress-controller/)** (KIC) — default Gateway API controller; wave 1. Alternatives: [Traefik](https://doc.traefik.io/traefik/routing/providers/kubernetes-gateway/) and [NGINX Gateway Fabric](https://docs.nginx.com/nginx-gateway-fabric/).
- **[cert-manager](https://cert-manager.io/docs/)** — issues TLS certificates with a HomeLab-owned CA (Certificate Authority); waves 0–1.
- **[OpenEBS](https://openebs.io/docs)** / **[Longhorn](https://longhorn.io/docs/)** — Storage; waves 0–1.
- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** — Secrets in Git; wave 0.
- **[Dex](https://dexidp.io/docs/)** + **[GitHub](https://docs.github.com/en/apps/oauth-apps)** — SSO (Single Sign-On); wave 2.
- **[1Password Python SDK](https://github.com/1Password/onepassword-sdk-python)** — OAuth outside Git; step 4.7 (local app, DesktopAuth).
- **[system-upgrade-controller](https://github.com/rancher/system-upgrade-controller)** — K3s upgrades (lightweight Kubernetes distribution); wave 3.

Why Gateway API instead of Ingress

The Ingress NGINX project was archived in March 2026 and `Ingress` is frozen in Kubernetes. The HomeLab publishes everything with `Gateway` and `HTTPRoute`. **Only one** controller is enabled at a time (Kong, Traefik, or NGINX Gateway Fabric): see [Switch the Gateway Controller](https://homelab.maceo.cl/en/operacion/cambiar-gateway/index.md).

Deep dive: [GitOps Catalog](https://homelab.maceo.cl/en/gitops/index.md) · [Storage](https://homelab.maceo.cl/en/storage/index.md) · [Secrets](https://homelab.maceo.cl/en/secrets/index.md)

## Before you start

- [Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md) completed (`argocd-server` Running).
- `kubectl` configured on your workstation.
- [Phase 0 — GitHub and 1Password](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/#github-and-1password-maceosoft-org-manual-one-time-only) completed (Apps, repos, and 1Password items).
- On the router, the range **`192.168.23.200–192.168.23.220`** outside DHCP (reserved): this is the MetalLB pool for `LoadBalancer` services, such as the Gateway ([IP scheme](https://homelab.maceo.cl/en/networking/#ip-scheme)).

______________________________________________________________________

## 4.1 — kubectl access

Copy the control plane kubeconfig (`deborah`) to your workstation, as `~/.kube/homelab-k3s.yaml`: it is the file used by the playbooks and the procedures in this guide. If you ran `playbook-k3s.yml` in full during Phase 3, you already have it (`k3s_install.fetch_kubeconfig.enabled: true`).

```
cd ansible
ansible-playbook -i inventory.ini playbook-k3s.yml --tags kubeconfig
cd ..
```

The `k3s_fetch_kubeconfig` role reads `/etc/rancher/k3s/k3s.yaml` in `deborah`, replaces the API (Application Programming Interface) address with the real control plane IP (or the VIP —virtual IP— if you use kube-vip) and saves it to `~/.kube/homelab-k3s.yaml` with `0600` permissions.

The file on the node is owned by root with `600` permissions, so it is read with `sudo`; then `127.0.0.1` is replaced with the control plane IP. `<usuario-admin>` is your node admin user ([Phase 0, Day 0](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/#day-0-freshly-installed-nodes-only-the-first-time-per-node)): SSO access ([4.12](#412-ssh-to-hosts-with-dex-optional)) does not exist yet, which requires Dex.

```
mkdir -p ~/.kube
ssh <usuario-admin>@192.168.20.5 sudo cat /etc/rancher/k3s/k3s.yaml \
  | sed 's#https://127.0.0.1:6443#https://192.168.20.5:6443#' \
  > ~/.kube/homelab-k3s.yaml
chmod 600 ~/.kube/homelab-k3s.yaml
```

Use it and verify access:

```
export KUBECONFIG=~/.kube/homelab-k3s.yaml
kubectl get nodes
```

Web alternative (Phase 5)

After deploying [vCluster Platform](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md), you can download the kubeconfig of the **management K3s** from [`https://vcluster.homelab.local`](https://vcluster.homelab.local) (connected cluster), with GitHub SSO via Dex. The local kubeconfig above remains the path for bootstrap, before Phase 5.

kubectl access

Local kubeconfig (Ansible or manual)

Works before Gateway and vCluster

File on disk; manual rotation

vCluster Platform (Phase 5)

Kubeconfig via SSO; connected cluster

Requires Phase 4+5 and assigned permissions

______________________________________________________________________

## 4.2 — Bootstrap GitOps (ApplicationSet)

A single manual apply; from here Argo CD manages the rest (auto).

**First, the platform secrets.** `maceosoft/gitops` is private: without the `maceosoft-argocd` App credentials, Argo CD cannot read the repo. The playbook reads them from 1Password and places them in the cluster. Just like in Phase 0, the 1Password app must be unlocked and `ONEPASSWORD_ACCOUNT_NAME` with your account name (the one in the app sidebar).

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
cd ansible
ansible-playbook -i inventory.ini playbook-platform-secrets.yml
cd ..
```

| Secret                       | Namespace     | Purpose                                                              |
| ---------------------------- | ------------- | -------------------------------------------------------------------- |
| `repo-gitops`                | `argocd`      | App `maceosoft-argocd` for `https://github.com/maceosoft/gitops.git` |
| `arc-runners-github-app`     | `arc-runners` | ARC runner registry                                                  |
| `terraform-app`, `tofu-vars` | `arc-runners` | OpenTofu credentials (`TF_VAR_*`) for the pipeline                   |

It also creates the `terraform` namespace, where OpenTofu stores its state. Always use `~/.kube/homelab-k3s.yaml`, not the current `kubectl` context. It is idempotent: run it again when you rotate something in 1Password. Details: [Platform maceosoft — GitHub](https://homelab.maceo.cl/en/maceosoft/github/#step-3-cluster-and-platform-secrets).

**Then, the bootstrap ApplicationSet:**

```
kubectl apply -f gitops/bootstrap/root-appset.yaml
```

[`bootstrap/root-appset.yaml`](https://github.com/maceosoft/gitops/blob/main/bootstrap/root-appset.yaml) generates a root Application, `homelab-root`, with the apps that are **uncommented** in its `apps` list. Initially only `argocd` is enabled: Argo CD managing itself, with Dex, RBAC (role-based access control), and the health check that waves need. This lets you test each component one by one:

1. Uncomment the next app in the list (they go in wave order) and push to the `gitops` repo.

1. **Re-apply the ApplicationSet**: Argo CD does not manage it (you created it with `kubectl`), so the push alone does not change the list:

   ```
   kubectl apply -f gitops/bootstrap/root-appset.yaml
   ```

1. Argo CD deploys it; wait until it becomes `Healthy` (`kubectl get applications -n argocd`).

1. Continue with the next one.

**Waves:** since all apps sync together in `homelab-root`, Argo CD respects their `argocd.argoproj.io/sync-wave` annotation and deploys by waves (from -1 to 4), waiting for each to be `Healthy` before the next. Waves 0 to 3 are described in 4.3, 4.4, 4.5, and 4.9; the diagram above shows the main ones.

To disable an app, comment it out again: `homelab-root` deletes its Application (the resources it created remain in the cluster).

**Verify:**

```
kubectl get applications -n argocd
```

Provisional UI (no Gateway yet; `argocd-server` speaks HTTP, open [`http://localhost:8080`](http://localhost:8080)):

```
kubectl port-forward svc/argocd-server -n argocd 8080:80
kubectl get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath='{.data.password}' | base64 -d; echo
```

______________________________________________________________________

## 4.3 — Wave 0: secrets + storage + MetalLB controller

Wait for `Healthy` in: `gateway-api`, `cert-manager`, `sealed-secrets`, `openebs`, `homelab-storage`, `metallb`.

| Application       | Product                                                          | Purpose                                                                                                                                                   |
| ----------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gateway-api`     | [Gateway API](https://gateway-api.sigs.k8s.io/)                  | CRDs (Custom Resource Definition) `Gateway`, `HTTPRoute`, etc., standard channel                                                                          |
| `cert-manager`    | [cert-manager](https://cert-manager.io/docs/)                    | Issues certificates; creates the one for each Gateway with the `cert-manager.io/cluster-issuer` annotation                                                |
| `sealed-secrets`  | [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) | Decrypts `SealedSecret` in the cluster; uses the 1Password key (item `sealed-secrets`) if the secrets playbook already created it                         |
| `openebs`         | [OpenEBS](https://openebs.io/docs)                               | Default LocalPV                                                                                                                                           |
| `homelab-storage` | —                                                                | StorageClasses `openebs-hostpath` and `longhorn-mixto` ([which to use and their trade-offs](https://homelab.maceo.cl/en/storage/#cluster-storageclasses)) |
| `metallb`         | [MetalLB](https://metallb.universe.tf/)                          | LoadBalancer Controller                                                                                                                                   |

**Verify:**

```
kubectl get storageclass
kubectl get pods -n metallb-system
kubectl get pods -n openebs
```

______________________________________________________________________

## 4.4 — Wave 1: MetalLB pool + Gateway + Longhorn

Wait for `Healthy` in: `metallb-config`, `cert-manager-config`, `kong`, and `longhorn`.

| Application           | Product                                                                                | Purpose                                                                                 |
| --------------------- | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `metallb-config`      | [MetalLB](https://metallb.universe.tf/)                                                | L2 Pool `192.168.23.200–.220`                                                           |
| `cert-manager-config` | [cert-manager](https://cert-manager.io/docs/)                                          | HomeLab custom CA (`ClusterIssuer homelab-ca`)                                          |
| `kong`                | [Kong Ingress Controller](https://developer.konghq.com/kubernetes-ingress-controller/) | Gateway API controller and `Gateway homelab` (HTTP 80 and HTTPS 443, `*.homelab.local`) |
| `longhorn`            | [Longhorn](https://longhorn.io/docs/)                                                  | HA replicated storage                                                                   |

```
kubectl get gateway homelab -n gateway
# ADDRESS en rango 192.168.23.200–.220 y PROGRAMMED=True
```

**Verify:**

```
kubectl get ipaddresspool -n metallb-system
kubectl get pods -n longhorn-system
```

______________________________________________________________________

## 4.5 — Wave 2: Argo CD on LAN + Dex

Wait for `Healthy` in `argocd-route` and `argocd`.

| Resource       | Product                                                                       | Purpose                                                                                                                                                                                                                      |
| -------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `argocd-route` | [Gateway API](https://gateway-api.sigs.k8s.io/)                               | `HTTPRoute` to `https://argocd.homelab.local` (the Gateway terminates TLS; `argocd-server` stays HTTP with `server.insecure`)                                                                                                |
| `argocd`       | [Argo CD](https://argo-cd.readthedocs.io/) and [Dex](https://dexidp.io/docs/) | Argo CD self-updates (tag `stable`); adds the GitHub connector, vCluster and Incus UI staticClients, RBAC, and Application health check ([Update Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-argocd/index.md)) |

______________________________________________________________________

## 4.6 — kube-vip vs MetalLB

|                       | [kube-vip](https://kube-vip.io/) | [MetalLB](https://metallb.universe.tf/) |
| --------------------- | -------------------------------- | --------------------------------------- |
| **Who installs it**   | Ansible (Phase 3)                | Argo CD (Phase 4)                       |
| **What gets the VIP** | Only API `:6443`                 | Services `LoadBalancer`                 |
| **Key config**        | `svc_enable=false`               | Pool `192.168.23.200–.220`              |

Do not use kube-vip for application Services.

### Trade-off analysis — kube-vip vs MetalLB

| Alternative            | Advantage                               | Cost / risk                                       |
| ---------------------- | --------------------------------------- | ------------------------------------------------- |
| kube-vip (Phase 3)     | Fixed VIP only for API `:6443`          | Does not expose Gateways or LB Services           |
| MetalLB (Phase 4)      | `LoadBalancer` for the Gateway and apps | L2 Pool `192.168.23.200–.220` must be free on LAN |
| Both (default HomeLab) | Separate responsibilities               | Two distinct VIP mechanisms — do not mix roles    |

## 4.7 — OAuth Secrets (GitHub + vCluster + Incus UI)

Sensitive values **do not go in Git**. Store them in 1Password and inject `argocd-secret`.

OAuth App already created in Phase 0

If you followed [Phase 0](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/#github-and-1password-maceosoft-org-manual-one-time-only), the OAuth App `Maceosoft Dex` and its fields `GITHUB_*` already exist: just run the playbook below.

**GitHub OAuth App** in the **org `maceosoft`** ([maceosoft → Settings → Developer settings → OAuth Apps → New OAuth App](https://github.com/organizations/maceosoft/settings/applications/new)). Created in the org, Dex can read its teams without approving it as a third-party app:

| GitHub Field     | Value                                                           |
| ---------------- | --------------------------------------------------------------- |
| Application name | `Maceosoft Dex` (or similar)                                    |
| Homepage URL     | `https://argocd.homelab.local`                                  |
| Redirect URIs    | `https://argocd.homelab.local/api/dex/callback` (only this one) |

**Homepage URL** is informational; the critical field is **Redirect URIs** (in older GitHub versions it was called *Authorization callback URL*): it must be exactly the URL above.

1Password items ( **personal** account, **`HomeLab`** vault by default): see [`argocd/secrets/1password.md`](https://github.com/maceosoft/gitops/blob/main/argocd/secrets/1password.md).

With the CLI (`op`)

Creating the OAuth App and its client secret is **web-only** (GitHub has no API for OAuth Apps). Saving the item can be done with a command; it's in [maceosoft Platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/#oauth-app-maceosoft-dex).

Test read (same module as apply):

```
# tu cuenta personal
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
# o HomeLab
export ONEPASSWORD_VAULT="HomeLab"
python3 ansible/scripts/check_1password.py GITHUB_CLIENT_ID
```

Inject into the cluster (recommended — Ansible):

```
cd ansible
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml
```

The `VCLUSTER_CLIENT_SECRET` field is generated in 1Password automatically if it doesn't exist. `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` come from the GitHub OAuth App created in Phase 0 (see note above); if you didn't create it, save them manually in 1Password before continuing.

Alternative Python script:

```
pip install onepassword-sdk
python3 gitops/argocd/secrets/apply-from-1password.py.example
kubectl rollout restart deployment argocd-dex-server -n argocd
```

Connector **GitHub** (only the `devops` team of the `maceosoft` org) and staticClients **vCluster** and **Incus UI** in [`dex.config`](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config).

`VCLUSTER_CLIENT_SECRET` does **not** come from GitHub. Ansible generates it in 1Password if missing and applies it per destination. Incus **does not use a secret**: it's a public OIDC (OpenID Connect) client (PKCE —Proof Key for Code Exchange—) in Dex.

| Field                    | Destinations (Ansible tags)                             |
| ------------------------ | ------------------------------------------------------- |
| `VCLUSTER_CLIENT_SECRET` | `argocd-secret` (`argocd`) + Helm Platform (`vcluster`) |

### Test login: URLs and what to expect

First verify that the three keys reached `argocd-secret` and that Dex restarted:

```
kubectl -n argocd get secret argocd-secret -o jsonpath='{.data}' | python3 -c "import sys,json;print(list(json.load(sys.stdin)))"
kubectl -n argocd get pods -l app.kubernetes.io/name=argocd-dex-server
```

They must show `dex.github.clientId`, `dex.github.clientSecret`, and `dex.vcluster.platform.clientSecret`. URLs need the [DNS for `*.homelab.local`](#48-homelablocal-dns) (or `/etc/hosts`); the certificate is signed by the HomeLab CA, so the browser warns until you import it (and `curl` needs `-k`).

| What to test          | URL                                                                                                                                              | What you should see                                                                                                              |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| **Argo CD** (UI)      | [`https://argocd.homelab.local`](https://argocd.homelab.local)                                                                                   | Login screen with the **Log in via GitHub** button. Clicking it goes through GitHub and returns to Argo CD with the Applications |
| Argo CD (HTTP)        | [`http://argocd.homelab.local`](http://argocd.homelab.local)                                                                                     | Redirects (`301`) to the HTTPS version                                                                                           |
| Argo CD (API)         | [`https://argocd.homelab.local/api/version`](https://argocd.homelab.local/api/version)                                                           | `{"Version":"v3.x.x"}`: the server responds through the Gateway                                                                  |
| **Dex** (discovery)   | [`https://argocd.homelab.local/api/dex/.well-known/openid-configuration`](https://argocd.homelab.local/api/dex/.well-known/openid-configuration) | JSON with `"issuer": "https://argocd.homelab.local/api/dex"`                                                                     |
| Dex (callback)        | `https://argocd.homelab.local/api/dex/callback`                                                                                                  | **Do not open manually.** It's the URL that must be in *Redirect URIs* of the GitHub OAuth App                                   |
| **vCluster Platform** | [`https://vcluster.homelab.local`](https://vcluster.homelab.local)                                                                               | SSO login with GitHub (Phase 5; doesn't exist before)                                                                            |
| **Incus UI**          | [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443)                                                                           | **Login with SSO** → GitHub (see below)                                                                                          |

From the terminal:

```
curl -sk https://argocd.homelab.local/api/version
curl -sk https://argocd.homelab.local/api/dex/.well-known/openid-configuration
curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}\n" http://argocd.homelab.local/
```

| If you see…                                                             | Cause                                                                                   | What to do                                                      |
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| GitHub responds **404** and the URL brings `client_id=.github.clientId` | Missing `dex.*` keys in `argocd-secret`: Dex couldn't substitute `$dex.github.clientId` | Run the playbook above (`--tags argocd`) and test again         |
| GitHub says **redirect_uri mismatch**                                   | *Redirect URIs* of the OAuth App differs                                                | Must be exactly `https://argocd.homelab.local/api/dex/callback` |
| Login succeeds but Argo CD **doesn't show Applications**                | Your account isn't in the `devops` team of `maceosoft`                                  | Add it to the team; the `groups` claim is `maceosoft:devops`    |
| `invalid_client` or `bad credentials`                                   | `GITHUB_CLIENT_SECRET` incorrect or rotated                                             | Update the field in 1Password and run the playbook again        |

### Incus UI OIDC

Requirements: UI installed in [Phase 2](https://homelab.maceo.cl/en/implementacion/fase-2-incus/#admin-ui), `argocd` Healthy, secrets injected, and Dex restarted.

Incus (7.x) only knows five OIDC keys: `oidc.issuer`, `oidc.client.id`, `oidc.audience`, `oidc.claim`, and `oidc.scopes`. It does **not** have `oidc.client.secret`, nor `oidc.groups.claim`, nor the `incus auth` command (those are from LXD —the project Incus forked from—). That's why in Dex the `incus-ui` client is **public** (`public: true`, PKCE, no secret): it's in [`dex.config`](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config).

**Ansible** (recommended) — `group_vars/dex_oauth_secrets.yml` already includes `apply_incus: true`. With the `dex.*` keys already loaded ([Test login](#probar-el-login-urls-y-que-esperar)), run:

```
export KUBECONFIG=~/.kube/homelab-k3s.yaml
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
cd ansible
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml --tags ensure,argocd,incus
```

This applies the keys in `argocd-secret`, restarts Dex, installs the HomeLab CA in **invincible** and configures `oidc.issuer` and `oidc.client.id` in Incus.

Incus must trust the HomeLab CA

Incus validates the HTTPS certificate of Dex (`https://argocd.homelab.local/api/dex`), which is signed by the HomeLab's own CA. If `invincible` doesn't know it, login fails (`curl` from the node gives `ssl_verify=20`). The playbook, with the `incus` tag, reads it from the cluster (`kubectl -n cert-manager get secret homelab-ca`), installs it in `/usr/local/share/ca-certificates/`, and **restarts the `incus` service** (instances keep running; the API is unavailable for a few seconds). Just that step: `--tags incus_ca`.

Verify on the node that it now trusts and that OIDC was configured:

```
curl -s -o /dev/null -w "%{http_code} ssl_verify=%{ssl_verify_result}\n" https://argocd.homelab.local/api/dex/.well-known/openid-configuration
incus config get oidc.issuer
incus config get oidc.client.id
```

They must return `200 ssl_verify=0`, `https://argocd.homelab.local/api/dex`, and `incus-ui`.

Manual alternative on the bootstrap node (**invincible**), after installing the CA ([manual](https://homelab.maceo.cl/en/incus/cluster-setup/#oidc-via-dex-phase-4)):

```
incus config set oidc.issuer=https://argocd.homelab.local/api/dex
incus config set oidc.client.id=incus-ui
```

#### Who can sign in and what they can do

- **Who signs in:** the Dex GitHub connector only allows members of the team `devops` in the org `maceosoft` (`orgs`/`teams` in `dex.config`). Anyone not in the team does not reach Incus.
- **What they can do:** with Incus's default configuration, any user who authenticates via the OIDC provider gets **full access** to the server. For granular permissions (single project, read-only) Incus uses [OpenFGA](https://linuxcontainers.org/incus/docs/main/authentication/) as the authorization engine: it is not configured in the HomeLab. For now, control is membership in the team `devops`.

Verify:

- [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443) → **Login with SSO** → GitHub → you land in the Incus UI
- An account **not** in the team `devops` → GitHub/Dex rejects the login before it reaches Incus

### SSO Flow

```
sequenceDiagram
  participant U as Usuario
  participant Argo as argocd_homelab_local
  participant Dex as Dex
  participant GH as GitHub
  U->>Argo: Login_SSO
  Argo->>Dex: OIDC
  Dex->>GH: OAuth
  GH->>Dex: token
  Dex->>U: sesion
```

| Decision      | Option A (HomeLab default)                           | Option B                                                            |
| ------------- | ---------------------------------------------------- | ------------------------------------------------------------------- |
| OAuth secrets | Local SDK; **Personal** account, vault **`HomeLab`** | `ONEPASSWORD_ACCOUNT_NAME` / `ONEPASSWORD_VAULT` or `kubectl patch` |

### Trade-off analysis — OAuth secrets

| Alternative             | Advantage                                                 | Cost / risk                                |
| ----------------------- | --------------------------------------------------------- | ------------------------------------------ |
| Ansible + 1Password SDK | Generates `VCLUSTER_CLIENT_SECRET` if missing; idempotent | 1Password app unlocked on the workstation  |
| Manual Python script    | Same SDK without playbook                                 | Loose steps; easy to forget restarting Dex |
| `kubectl patch` direct  | No 1Password                                              | Secrets in history; not reproducible       |

______________________________________________________________________

## 4.8 — `*.homelab.local` DNS

Names are resolved by the LAN DNS (BIND on `deborah`, [`playbook-bind-dns.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-bind-dns.yml)). The zone `homelab.local` contains:

| Record                                                                                                                                             | Points to                                                                   |
| -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `incus.homelab.local`                                                                                                                              | invincible (`192.168.20.6`)                                                 |
| `invincible.homelab.local`, `oliver.homelab.local`, `deborah.homelab.local`                                                                        | each server, with its fixed IP (`192.168.20.6`, `.7` and `.5`)              |
| `argocd.homelab.local`, `vcluster.homelab.local`, `api.homelab.local`, `grafana.homelab.local`, `open-webui.homelab.local`, `ollama.homelab.local` | the Gateway: `gateway_ip` = `192.168.23.200` (first IP of the MetalLB pool) |

No wildcard: every name is an explicit record in [`db.homelab.local.j2`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/bind_dns/templates/db.homelab.local.j2). To publish an app with a new `HTTPRoute`, add its record to that template and apply the playbook; until then its name does not resolve.

The Gateway IP is **fixed**: the `kong`, `traefik` and `nginx-gateway` Services request it via a MetalLB annotation, and the same variable `gateway_ip` (`ansible/group_vars/all.yml`) feeds DNS. If you change it, change it on both sides. Apply DNS (from `ansible/` on `homelab`):

```
ansible-playbook -i inventory.ini playbook-bind-dns.yml
# 192.168.23.200
dig @192.168.20.5 argocd.homelab.local +short
# 192.168.20.7
dig @192.168.20.5 oliver.homelab.local +short
```

Your workstation must use `192.168.20.5` as DNS. If you cannot, the fallback is `/etc/hosts`:

```
192.168.23.200  argocd.homelab.local vcluster.homelab.local api.homelab.local grafana.homelab.local open-webui.homelab.local ollama.homelab.local
192.168.20.6    incus.homelab.local invincible.homelab.local
192.168.20.7    oliver.homelab.local
192.168.20.5    deborah.homelab.local
```

Verify the Gateway has that IP:

```
kubectl get gateway homelab -n gateway -o jsonpath='{.status.addresses[0].value}'; echo
```

The certificate is signed by the HomeLab CA: for the browser to trust it, import its root certificate (`kubectl -n cert-manager get secret homelab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > homelab-ca.crt`) or use `curl -k` in the meantime.

- **macOS:** open `homelab-ca.crt` by double-clicking (adds to *Keychain Access*), open it in the list and under **Trust** choose *Always Trust*.
- **Firefox** (has its own store; if it doesn't trust the system CA): *Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import*, and mark it to trust for identifying websites.

```
# macOS (llavero del sistema)
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain homelab-ca.crt
# Debian/Ubuntu
sudo cp homelab-ca.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
```

If Firefox still doesn't trust (on Linux it doesn't read the system store), import it from its UI.

```
curl -kI https://argocd.homelab.local
curl -kI https://incus.homelab.local:8443
```

______________________________________________________________________

## 4.9 — Wave 3: K3s upgrades (optional)

Wait for `Healthy` in `k3s-upgrade` ([system-upgrade-controller](https://github.com/rancher/system-upgrade-controller)).

______________________________________________________________________

## 4.10 — Phase 4 Checklist

| Check                              | Criterion                                                                                                                   |
| ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| Nodes Ready                        | `kubectl get nodes` → 3 Ready                                                                                               |
| Apps auto Healthy                  | `kubectl get applications -n argocd`                                                                                        |
| Storage default                    | PVC `openebs-hostpath` provisions                                                                                           |
| Gateway responds                   | `curl -k https://argocd.homelab.local`                                                                                      |
| SSO Dex                            | GitHub login in Argo CD                                                                                                     |
| Incus UI OIDC                      | SSO login at [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443)                                         |
| Runner ARC                         | `arc-runners` Healthy; runner listed in *maceosoft → Settings → Actions → Runners* ([4.11](#411-arc-and-opentofu-pipeline)) |
| OpenTofu Pipeline                  | First `tofu apply` of `infra` green; `terraform/tfstate-default-github` exists ([4.11](#411-arc-and-opentofu-pipeline))     |
| SSH with Dex *(optional)*          | `opkssh login` and `ssh devops@oliver.homelab.local` sign in ([4.12](#412-ssh-to-hosts-with-dex-optional))                  |
| Monitoring *(optional)*            | Grafana opens, node panel shows temperatures and a test alert reaches Discord ([4.13](#413-monitoring-and-alerts-optional)) |
| Argo CD Notifications *(optional)* | Test command sends a message to `despliegues` ([4.14](#414-argocd-notifications-to-discord-optional))                       |

Global checklist (URLs, `/etc/hosts`, nodes): **[HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md)**.

## 4.11 — ARC and OpenTofu pipeline

With `arc-helm-repo`, `arc-controller`, `terraform-rbac` and `arc-runners` uncommented in `homelab-root`, Argo CD deploys the GitHub Actions runner:

| Application           | What it deploys                                                                                                                                                                                                                             |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `arc-controller`      | ARC controller (`arc-systems`)                                                                                                                                                                                                              |
| `terraform-rbac`      | ServiceAccount `tofu-runner`, with access only to Secrets/Leases in namespace `terraform`                                                                                                                                                   |
| `arc-runners`         | Org scale set `maceosoft` (up to 3 runners), for CI, deploys and OpenTofu. The old org has its own, `arc-runners-symintel` (see [Two organizations](https://homelab.maceo.cl/en/maceosoft/github/#two-organizations-one-scale-set-per-org)) |
| `arc-runners` (cache) | Shared volume `arc-cache` mounted at `/cache` on the runners ([Runner cache](https://homelab.maceo.cl/en/operacion/cache-arc/index.md))                                                                                                     |

```
kubectl get applications -n argocd | grep -E 'arc|terraform'
# controller + listener de arc-runners
kubectl get pods -n arc-systems
```

Verify the runner appears in GitHub:

The runner appears in *maceosoft → Settings → Actions → Runners*.

```
gh api orgs/maceosoft/actions/runners --jq '.runners[] | [.name, .status] | @tsv'
```

**First pipeline.** In `maceosoft/infra`, trigger *Actions → tofu → Run workflow* on `main`, or push to `main`. The reusable workflow [`tofu.yml`](https://github.com/maceosoft/core-pipelines/blob/main/.github/workflows/tofu.yml) runs on `arc-runners`: it does `plan` on PRs (pull requests) and `apply` on `main`. The first run creates the repo `api` with its environments and FTP secrets (File Transfer Protocol).

```
# state tras el primer apply
kubectl -n terraform get secret tfstate-default-github
```

From here, each new repo is added with the [routine procedure — Add a repo](https://homelab.maceo.cl/en/operacion/agregar-repo/index.md): FTP credentials in 1Password (if it deploys), entry in the [`infra`](https://github.com/maceosoft/infra) map per PR and the CI (continuous integration)/CD workflow.

## 4.12 — SSH to hosts with Dex (optional)

Members of the `devops` team in the `maceosoft` org on GitHub SSH (Secure Shell, encrypted terminal connection) into `invincible`, `oliver` and `deborah` using their Dex identity, without distributing keys. Incus instances are not included here: they are managed with `incus exec` and the UI SSO login. How a person logs in: [SSH into the servers](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md).

**Requirements.**

- Dex working ([4.5](#45-wave-2-argocd-en-lan-dex) and [4.7](#47-oauth-secrets-github-vcluster-incus-ui)).
- The DNS for `homelab.local` applied ([4.8](#48-homelablocal-dns)) and **each node using the `deborah` BIND** in its `/etc/resolv.conf` (set in [Phase 1](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md) with `playbook-set-static-ip.yml`). Without that a node cannot validate any login.
- `kubectl` with the HomeLab kubeconfig: the role reads the cert-manager CA so nodes trust Dex.
- Your regular key in `authorized_keys` on each node ([hardening](https://homelab.maceo.cl/en/implementacion/fase-1-red/#hardening-recommended-before-proceeding)). This access **adds** to keys, does not replace them: it's your emergency path if Dex or K3s go down.

### How it works

SSH doesn't speak OIDC. [opkssh](https://github.com/openpubkey/opkssh), based on [OpenPubkey](https://github.com/openpubkey/openpubkey), acts as a bridge:

```
sequenceDiagram
    participant U as Estación del usuario
    participant D as Dex + GitHub
    participant H as Host (sshd + opkssh)
    U->>D: opkssh login (navegador, PKCE)
    D-->>U: token firmado (solo miembros del team devops)
    U->>H: ssh devops@host (llave efímera + token)
    H->>H: opkssh verify (proveedor, firma, grupo)
    H-->>U: sesión como devops
```

1. `opkssh login` authenticates against Dex (GitHub) and leaves an SSH key that expires in 24 hours.
1. When connecting, `sshd` runs `opkssh verify` (via `AuthorizedKeysCommand`), which validates the token signature and that the issuer is Dex.
1. An opkssh *policy plugin* requires the `maceosoft:devops` group and the Linux user `devops`.

### Steps

1. **Client in Dex.** The `opkssh` client (public, PKCE, with the three callbacks in `localhost`) is in [`dex.config`](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config). When Argo CD syncs, restart Dex so it loads it:

   ```
   kubectl -n argocd rollout restart deploy/argocd-dex-server
   ```

1. **Verify nodes resolve Dex** (from `ansible/` on `homelab`):

   ```
   ansible incus_cluster -i inventory.ini -b -m command -a "getent hosts argocd.homelab.local"
   ```

   All three must return `192.168.23.200`. If any doesn't, fix its DNS first (see *If it fails* below).

1. **One test node first** (`oliver`), with `--check` before applying:

   ```
   export KUBECONFIG=~/.kube/homelab-k3s.yaml
   cd ansible && source ../.venv/bin/activate
   ansible-playbook -i inventory.ini playbook-opkssh.yml --limit oliver --check --diff
   ansible-playbook -i inventory.ini playbook-opkssh.yml --limit oliver
   ```

   The role stops before installing anything if the node doesn't resolve Dex or doesn't trust its certificate.

1. **Test the login** following [SSH into the servers](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md) and, with the session working, extend to the rest: `--limit invincible,deborah`.

What the role [`opkssh`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/opkssh/README.md) installs on each node: the binary with pinned version and its checksum, `/etc/opk/` (Dex provider and the plugin), the Linux account `devops`, the HomeLab CA and a file in `/etc/ssh/sshd_config.d/`.

### Who gets in

Access depends on **belonging to the `devops` team in GitHub**. No user lists or email lists.

- **Dex** only issues tokens to team members ([GitHub connector](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config)) and puts the team in the `groups` claim as `maceosoft:devops`.
- **Each node** requires that group with a policy plugin (`/etc/opk/devops-team.sh`). The group is changed with `opkssh_allowed_group` in the role.

Adding or removing access is changing the team in GitHub. A removal prevents getting new tokens, but the key the person already has remains valid until it expires (24 hours).

Why a plugin and not `/etc/opk/auth_id`

The standard `oidc:groups:maceosoft:devops` rule doesn't work with GitHub groups: opkssh splits the rule by `:` and only compares the last piece (`devops`) with the full group (`maceosoft:devops`), so they never match, and quoting the value doesn't help either. That's why `auth_id` is left without rules and the plugin decides (the behavior is in `policy/enforcer.go` of opkssh).

Shared account with `sudo`

Everyone logs in as `devops`, with passwordless `sudo` (`opkssh_login_user_sudo_nopasswd`): anyone in the team is administrator of the three nodes and session logs don't distinguish people by Linux user. To require a `sudo` password, set that variable in `false` and assign a password to the account.

### Trade-off analysis — how to SSH in

| Alternative                           | Advantage                                                | Cost / risk                                                                            |
| ------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| **opkssh with Dex** (this guide)      | Reuses Dex and the team; 24 h keys; no lists to maintain | Young project; needs a plugin for groups with `:`; Dex runs in K3s on these same nodes |
| SSH CA with ephemeral certs (step-ca) | Very mature                                              | Another component with its own root key to operate. Plan B                             |
| GitHub public keys filtered by team   | Simple                                                   | Doesn't use Dex; keys don't expire                                                     |
| `authorized_keys` via Ansible         | No new pieces                                            | Every add or remove is a manual change                                                 |
| Teleport / Boundary                   | Session audit and recording                              | Heavy for three nodes; Teleport Community doesn't offer generic OIDC                   |

### Undo

[`playbook-uninstall-opkssh.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-uninstall-opkssh.yml) removes what the role installs (`sshd` configuration, binary, `/etc/opk`, and the `devops` account). It's safe by default: without `-e opkssh_uninstall_confirm=true` it only shows what would be removed.

```
ansible-playbook -i inventory.ini playbook-uninstall-opkssh.yml --limit oliver
ansible-playbook -i inventory.ini playbook-uninstall-opkssh.yml --limit oliver -e opkssh_uninstall_confirm=true
```

Doesn't touch `authorized_keys` or the hardening. The HomeLab CA is preserved; to remove it as well: `-e opkssh_uninstall_remove_ca=true`.

### If it fails (on the server)

| Symptom                                                                | Probable cause                                                                                                                                     | What to check                                                                                                                                                                                                |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Role stops: "does not resolve argocd.homelab.local"                    | The node isn't using the HomeLab BIND. Happened on `deborah`: NetworkManager, without `dns=none`, rewrote `/etc/resolv.conf` with the router's DNS | `cat /etc/resolv.conf` and `getent hosts argocd.homelab.local` on the node. Fix with `playbook-set-static-ip.yml --limit <nodo>` ([Phase 1](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md)) |
| `Permission denied`; in `/var/log/opkssh.log`: `no policy to allow …`  | The token doesn't bring the `maceosoft:devops` group, or a different Linux user was requested                                                      | `opkssh inspect ~/.ssh/id_ecdsa-cert.pub` (must list `groups`); `/etc/opk/policy.d/devops-team.yml`                                                                                                          |
| `Permission denied`; in `/var/log/opkssh.log`: `lookup … no such host` | Same as first row                                                                                                                                  | Node DNS                                                                                                                                                                                                     |
| `x509: certificate signed by unknown authority` on the node            | The node doesn't trust the HomeLab CA                                                                                                              | `curl -s -o /dev/null -w "%{http_code} ssl_verify=%{ssl_verify_result}\n" https://argocd.homelab.local/api/dex/.well-known/openid-configuration` must give `200 ssl_verify=0`                                |
| No one can SSO in and Dex doesn't respond                              | Dex lives in K3s on these same nodes                                                                                                               | Enter with the emergency key from `authorized_keys`                                                                                                                                                          |

## 4.13 — Monitoring and alerts (optional)

Gives visibility of the three nodes (temperature, memory, disk, Kubernetes and platform status) and notifies via Discord only about what needs attention. Uses [VictoriaMetrics](https://docs.victoriametrics.com/) for metrics and alerts, Alertmanager for routing, and [Grafana](https://grafana.com/docs/grafana/latest/) for dashboards. What each alert means and what to do: [Monitoring and alerts](https://homelab.maceo.cl/en/operacion/monitoreo/index.md).

**Requirements.**

- Longhorn `Healthy` ([4.4](#44-wave-1-metallb-pool-gateway-longhorn)): the metrics volume uses the `longhorn` class, the same one as the ARC (Actions Runner Controller) cache.
- The `homelab.local` Gateway and DNS ([4.8](#48-homelablocal-dns)): Grafana is published at `grafana.homelab.local`.
- Cluster with internet egress: after each deploy, a Job downloads standard dashboards and rules from GitHub.
- A [healthchecks.io](https://healthchecks.io/) account (free tier is enough) and a Discord server.

### How it works

```
flowchart LR
  subgraph fuentes [Fuentes]
    NE[node-exporter\nen los 3 nodos]
    KS[kube-state-metrics]
    PL[cert-manager\nLonghorn · ArgoCD]
  end
  AG[vmagent]
  DB[(vmsingle\nvolumen Longhorn)]
  AL[vmalert]
  AM[Alertmanager]
  GF[Grafana]
  DC[Discord\ncríticas · avisos]
  HC[healthchecks.io]
  NE --> AG
  KS --> AG
  PL --> AG
  AG --> DB
  DB --> AL
  DB --> GF
  AL --> AM
  AM -->|critical / warning| DC
  AM -->|Watchdog cada 5 min| HC
  HC -.->|si el ping deja de llegar| DC
```

The ping to healthchecks.io is what alerts if **the whole** cluster goes down: `deborah` is the only control plane, so monitoring living in the cluster couldn't alert on its own failure. The `Watchdog` alert is always active and Alertmanager sends it every 5 minutes to healthchecks.io (a *dead man's switch*); if the ping stops arriving, healthchecks.io alerts on its own.

### Steps

Do steps 1 through 5 **before** enabling the app in GitOps (step 6): Argo CD needs the namespace and Secrets to exist.

1. **Discord** (manually, no bot: just webhooks). Create a role and a private category with two channels, `alertas-criticas` and `alertas-avisos`, and one webhook per channel. Step by step, with the exact menu names: [Discord and healthchecks.io — Part 1](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#part-1-discord). Your server may have people outside infrastructure, so don't use an existing channel.

1. **healthchecks.io.** Create an account and a check (`Period` of 5 minutes, `Grace Time` of 10), copy its **ping URL** and connect it to `alertas-criticas` using its Discord integration: [Part 2](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#part-2-healthchecksio).

1. **1Password** (vault `HomeLab`). Three items, as **Secure Note** and with fields **outside any section**. The item title and each field name must be exact, because the script looks them up by name:

   | Item                | Field      | Content                                    |
   | ------------------- | ---------- | ------------------------------------------ |
   | `discord-webhooks`  | `critical` | Webhook for `alertas-criticas`             |
   |                     | `warnings` | Webhook for `alertas-avisos`               |
   | `healthchecks-ping` | `url`      | Check ping URL                             |
   | `grafana-admin`     | `password` | Grafana admin password                     |
   |                     | `username` | *(optional)* username; defaults to `admin` |

   How to create and verify them (also with the `op` CLI): [Part 3](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#part-3-store-in-1password).

1. **Secrets in the cluster.** Requires the 1Password app unlocked (*Integrate with other apps*) and the homelab kubeconfig, like the rest of [platform secrets](https://homelab.maceo.cl/en/maceosoft/github/index.md). They are optional: if none of the three items exists the script does nothing; if any exists, it requires all three. Verify first, then apply (from `ansible/`):

   ```
   python3 scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

   Creates the namespace `monitoring` (with `pod-security enforce=privileged`, because `node-exporter` uses the node's disk and network) and the Secrets `discord-webhooks`, `healthchecks-ping` and `grafana-admin`. The script rejects a Discord URL that doesn't have the format `https://discord.com/api/webhooks/<id>/<token>`, to avoid sending alerts to the wrong destination.

1. **DNS.** Add the `grafana` record and apply it:

   ```
   ansible-playbook -i inventory.ini playbook-bind-dns.yml
   # 192.168.23.200
   dig @192.168.20.5 grafana.homelab.local +short
   ```

1. **Enable in GitOps.** `victoria-metrics` (wave 2) and `victoria-metrics-config` (wave 3, its rules and the Grafana route) are already in the list of [`root-appset.yaml`](https://github.com/maceosoft/gitops/blob/main/bootstrap/root-appset.yaml). Merge that change into `main` of the `gitops` repo and **then apply the ApplicationSet**: the `homelab-root` of the cluster was created manually with `kubectl apply` and Argo CD doesn't update it from git automatically, so nothing happens without this command.

   ```
   kubectl apply -f gitops/bootstrap/root-appset.yaml
   kubectl get applications -n argocd | grep victoria
   kubectl get pods -n monitoring -o wide
   kubectl top pods -n monitoring
   ```

   Expected consumption is **~1 to 1.5 GB** total (estimate; confirm with `kubectl top`). `vmsingle`, `vmagent`, `vmalert`, Alertmanager and Grafana prefer `invincible` and never go to `oliver` (the light node); `node-exporter` runs on all three.

1. **End-to-end test.** Go to [`https://grafana.homelab.local`](https://grafana.homelab.local) with **Sign in with SSO (GitHub)** (see [Grafana with SSO](#grafana-with-dex-sso)), or with `admin` and the 1Password password, and verify that the node panel shows temperatures. Then send a test alert:

   ```
   kubectl -n monitoring port-forward svc/vmalertmanager-vm-victoria-metrics-k8s-stack 9093:9093 &
   curl -s -XPOST localhost:9093/api/v2/alerts -H 'Content-Type: application/json' -d \
     '[{"labels":{"alertname":"PruebaManual","severity":"warning","nodename":"prueba"},"annotations":{"summary":"Alerta de prueba: si la ves, Discord funciona."}}]'
   ```

   It should arrive at `alertas-avisos` (change `severity` to `critical` to test `alertas-criticas`). If the Service isn't named that, find the name with `kubectl -n monitoring get svc | grep alertmanager`. To test the ping, create a 30-minute silence for `alertname=Watchdog` in the Alertmanager UI (*Silences*): the ping stops and healthchecks.io notifies at ~15 minutes; when the silence expires, it recovers.

   In the Alertmanager UI, *Silences → New Silence*: matcher `alertname=Watchdog` and 30-minute duration.

   With the same `port-forward` to port `9093`, via the Alertmanager v2 API:

   ```
   INICIO=$(date -u +%Y-%m-%dT%H:%M:%SZ)
   FIN=$(date -u -v+30M +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '+30 min' +%Y-%m-%dT%H:%M:%SZ)
   curl -s -XPOST localhost:9093/api/v2/silences -H 'Content-Type: application/json' -d \
     "{\"matchers\":[{\"name\":\"alertname\",\"value\":\"Watchdog\",\"isRegex\":false}],\"startsAt\":\"$INICIO\",\"endsAt\":\"$FIN\",\"createdBy\":\"yo\",\"comment\":\"prueba del ping\"}"
   ```

### What it monitors

| Alert                                    | When                                                                                                                                  | Severity           |
| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| `TemperaturaAlta` / `TemperaturaCritica` | `deborah` from 75 / 85 °C (the SoC throttles at those points); `invincible` and `oliver` from 80 / 90 °C (manufacturer limit: 100 °C) | warning / critical |
| `NodoNoListo`                            | a node stops being Ready for more than 2 minutes                                                                                      | critical           |
| `ServicioBaseSinReplicas`                | Argo CD, Kong or cert-manager with no replicas                                                                                        | critical           |
| `CertificadoPorVencer`                   | a certificate expires in less than 14 days / 3 days                                                                                   | warning / critical |
| `LonghornVolumenDegradado` / `Fallido`   | a volume loses a replica / ends up with no replicas                                                                                   | critical           |
| `ArgoCDAppNoSana` / `Desincronizada`     | an app has been out of `Healthy` / `Synced` for more than 15 / 30 minutes                                                             | warning            |
| Disk, memory, clock and pods             | default chart rules                                                                                                                   | warning / critical |

Temperature thresholds come from the trigger points that the hardware itself publishes in `/sys`. The rules are in [`monitoring/config/rules`](https://github.com/maceosoft/gitops/tree/main/monitoring/config/rules) and have tests in [`monitoring/tests`](https://github.com/maceosoft/gitops/tree/main/monitoring/tests) (`monitoring/tests/run.sh`, with Docker): if you change a threshold, update the test.

### Undo

Remove `victoria-metrics` and `victoria-metrics-config` from the list of `root-appset.yaml`, merge the change and reapply it (`kubectl apply -f gitops/bootstrap/root-appset.yaml`): the root deletes the Applications, but **the resources remain in the cluster** (this repo's convention). To also delete the metrics history and the Secrets, delete the namespace: `kubectl delete namespace monitoring` (destructive: the metrics volume is lost).

### Grafana with Dex SSO

Grafana uses the same Dex as Argo CD, Incus, and vCluster (see [4.7](#47-oauth-secrets-github-vcluster-incus-ui)). Everything comes from Git, with no new secrets:

- **`grafana` client in Dex** ([`dex.config`](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config)): public client with PKCE, no secret, with redirect `https://grafana.homelab.local/login/generic_oauth`. After merging the change, restart Dex so it picks it up: `kubectl -n argocd rollout restart deploy argocd-dex-server`.
- **`auth.generic_oauth` in Grafana** ([`values-vm-stack.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/values-vm-stack.yaml)): the `maceosoft:devops` group gets the Admin role and any other is rejected (`role_attribute_strict`). The local user `admin` remains active as a fallback.
- **HomeLab CA:** the Dex certificate is signed by that CA, and pods don't trust it. A cert-manager `Certificate` ([`grafana-ca.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/config/route/grafana-ca.yaml)) places the CA in Secret `grafana-ca`, which is mounted in Grafana as `tls_client_ca`.
- **Pod DNS:** by default CoreDNS forwards to `deborah` DNS, including the router, which responds NXDOMAIN (name does not exist) for `homelab.local`. An `coredns-custom` ([`coredns/`](https://github.com/maceosoft/gitops/tree/main/coredns)) forwards `homelab.local` and `mco.local` only to BIND. It is the Application `coredns-custom`: since the ApplicationSet was applied manually, after merging that change you must run `kubectl apply -f gitops/bootstrap/root-appset.yaml` again.

### Incus Metrics (optional)

Each Incus node publishes CPU, memory, disk, and network usage of **its** instances and VMs at `https://<nodo>:8443/1.0/metrics`. To read them you must present a **client certificate** (a digital ID) that Incus has in its trust list with type `metrics`: it only allows reading metrics, not administering Incus. No need to touch `core.metrics_address`: the API on port 8443 already serves that endpoint.

The certificate is issued by **cert-manager**, just like the Gateway ones, so there's nothing to create manually or store in 1Password:

1. **Merge the `gitops` change.** It brings a `Certificate` (`incus-metrics`, namespace `monitoring`, signed by ClusterIssuer `homelab-ca`, usage `client auth`, 10-year duration) and the scrape [`incus.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/config/scrapes/incus.yaml). cert-manager places the keypair in Secret `monitoring/incus-metrics`, and the private key never leaves the cluster. Verify it's ready:

   ```
   # READY = True
   kubectl -n monitoring get certificate incus-metrics
   ```

1. **Register the public part in Incus** (from `ansible/`):

   ```
   ansible-playbook -i inventory.ini playbook-incus-metrics.yml
   ```

   Read `tls.crt` from the Secret and add it to the Incus trust store with `--type=metrics` and name `vmagent`. If it's already there, it does nothing. With the 10-year duration there's no need to repeat it; if cert-manager renews it, run it again.

1. **Check** in Grafana, *Explore*: `count by (nodename) (incus_procs_total)` should return a value for each node with instances. Then open the **Incus** dashboard.

   In Grafana, *Explore*, run `count by (nodename) (incus_procs_total)`.

   The same query directly against VictoriaMetrics:

   ```
   kubectl -n monitoring port-forward svc/vmsingle-vm-victoria-metrics-k8s-stack 8428:8428 &
   curl -s http://127.0.0.1:8428/api/v1/query --data-urlencode 'query=count by (nodename) (incus_procs_total)'
   ```

Incus trusts the certificate, not the CA

Do not enable `core.trust_ca_certificates` in Incus: it would allow any certificate signed by `homelab-ca` (including the Gateway ones) to administer Incus. The playbook adds only this certificate and only with type `metrics`.

### If it fails

| Symptom                                                                                 | Probable cause                                    | What to check                                                                                                                                                         |
| --------------------------------------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `victoria-metrics` doesn't sync: *namespace "monitoring" not found*                     | Step 4 is missing                                 | `kubectl get ns monitoring`; run `playbook-platform-secrets.yml`                                                                                                      |
| `node-exporter` rejected: *violates PodSecurity*                                        | The namespace lacks the `privileged` label        | `kubectl get ns monitoring --show-labels`; run step 4 again                                                                                                           |
| `vmsingle` in `Pending`                                                                 | Volume `longhorn` is not created                  | `kubectl -n monitoring describe pvc`; Longhorn status ([4.4](#44-wave-1-metallb-pool-gateway-longhorn))                                                               |
| Alerts don't reach Discord                                                              | Alertmanager doesn't read the webhook             | `kubectl -n monitoring logs` of Alertmanager (look for `discord`) and that files exist in `/etc/alertmanager-secrets/`                                                |
| Grafana missing chart dashboards (Kubernetes, nodes, VictoriaMetrics)                   | The download Job didn't reach GitHub              | `kubectl -n monitoring get jobs` and its logs; cluster internet egress                                                                                                |
| Missing a dashboard for label `homelab` (Argo CD, Longhorn, cert-manager, temperatures) | The sidecar didn't pick up its ConfigMap          | `kubectl -n monitoring get cm -l grafana_dashboard=1` should list four `grafana-dashboard-*`; logs of container `grafana-sc-dashboard`                                |
| A `homelab` panel shows empty                                                           | Missing scrape or labels don't match              | [Monitoring and alerts — Dashboards](https://homelab.maceo.cl/en/operacion/monitoreo/#grafana-dashboards)                                                             |
| No `incus_*` metrics                                                                    | Certificate not issued or not registered in Incus | `kubectl -n monitoring get certificate incus-metrics` (READY) and, on invincible, `sudo incus config trust list` (there should be a `metrics` client named `vmagent`) |
| `grafana.homelab.local` doesn't open                                                    | Missing DNS record or route                       | step 5; `kubectl -n monitoring get httproute`                                                                                                                         |
| Grafana SSO fails with *Failed to get token from provider*                              | Grafana doesn't resolve or doesn't trust Dex      | `kubectl -n monitoring logs deploy/vm-grafana -c grafana \| grep oauth`: *no such host* is DNS (Application `coredns-custom`); *x509* is the CA (`grafana-ca`)        |
| Grafana SSO asks for a client that doesn't exist                                        | Dex didn't read the `grafana` client              | `kubectl -n argocd rollout restart deploy argocd-dex-server`                                                                                                          |
| `TargetDown` of etcd, scheduler, or controller-manager                                  | K3s monitors were enabled                         | They must be `enabled: false` in [`values-vm-stack.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/values-vm-stack.yaml)                              |

______________________________________________________________________

## 4.14 — Argo CD Notifications to Discord (optional)

Argo CD includes a notifications controller, but unconfigured it alerts on nothing. With the configuration from [`argocd/config/notifications`](https://github.com/maceosoft/gitops/tree/main/argocd/config/notifications) (repo `gitops`) you get two message types, for **all** Applications, without annotating each one:

| Event                                   | Message                                                                | Channel                    |
| --------------------------------------- | ---------------------------------------------------------------------- | -------------------------- |
| A sync succeeded and the app is healthy | 🚀 `<app>` deployed, with revision, status, and **the images it uses** | your `despliegues` channel |
| A sync failed                           | 🟡 `<app>`: sync failed, with error message                            | `alertas-avisos`           |

The images in the message reflect the actual version: when Argo CD, Longhorn, or another product updates itself (charts use floating versions like `1.13.x`, and Argo CD tracks the `stable` branch), the deployment message carries the new version. Long‑lasting states (an app `Degraded` for more than 15 minutes or `OutOfSync` for more than 30) remain covered by Alertmanager ([4.13](#413-monitoring-and-alerts-optional)), so the two don't overlap.

`despliegues` is visible to the whole team

Unlike alerts, which go to a private category, `despliegues` is a channel on your server that people outside infrastructure can see. Messages contain only the app name, revision, images, and a link to `argocd.homelab.local`.

**Prerequisites.** The 1Password items from [4.13](#413-monitoring-and-alerts-optional) (`discord-webhooks` included) and the secrets playbook already applied.

### Steps

1. **Webhook for the `despliegues` channel** in Discord, and save it in the `deployments` field of the `discord-webhooks` item: [Discord and healthchecks.io — step 7](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#7-a-webhook-for-the-deployments-channel-optional). With the 1Password CLI: `read -rs V; op item edit discord-webhooks "deployments[password]=$V"; unset V`.

1. **Create the Secret in the cluster** (from `ansible/`). The script adds the keys to the `argocd-notifications-secret` that Argo CD already ships: `discord-despliegues-url` (the `deployments` field) and `discord-avisos-url` (the `warnings` for alerts). The field is optional and validated with the same Discord webhook format:

   ```
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

1. **Merge the `gitops` change.** No need to touch the ApplicationSet: the configuration is part of the `argocd` app, which syncs automatically. The notifications controller watches its `ConfigMap` and its Secret, but if it doesn't pick up the change, restart it:

   ```
   kubectl -n argocd rollout restart deploy/argocd-notifications-controller
   ```

1. **Test.** This command sends **a real message** to the `despliegues` channel with the `argocd` app data:

   ```
   kubectl -n argocd exec deploy/argocd-server -- \
     argocd admin notifications template notify discord-deployed argocd --recipient discord-despliegues:
   ```

   To test templates **without sending anything to Discord**, the repo `gitops` includes [`argocd/config/notifications/test.sh`](https://github.com/maceosoft/gitops/blob/main/argocd/config/notifications/test.sh): it renders them against a real app in the cluster and a local receiver, and verifies the body is valid JSON and respects Discord limits. Requires `kustomize`, Docker, and a kubeconfig with read access. This is the recommended test: the command above, running inside the pod, sends a real message to the channel.

### If it fails

| Symptom                                                                    | Likely cause                                     | What to check                                                                                                                                                               |
| -------------------------------------------------------------------------- | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Nothing arrives in `despliegues`                                           | Secret missing keys, or controller didn't reload | `kubectl -n argocd get secret argocd-notifications-secret -o jsonpath='{.data}'` must list `discord-despliegues-url` and `discord-avisos-url`; restart controller           |
| `config referenced '$discord-…', but key does not exist in secret` in logs | Missing `deployments` field in 1Password         | `--check` of the script; rerun playbook                                                                                                                                     |
| An extra message arrives every time an app syncs                           | Expected: one message per successful sync        | If too noisy, tune the trigger `on-deployed` in `argocd/config/notifications`                                                                                               |
| `HTTP 404` on send                                                         | Webhook deleted in Discord                       | Create a new one and update `deployments` ([rotate](https://homelab.maceo.cl/en/operacion/rotar-credenciales/#monitoring-discord-webhooks-healthchecksio-ping-and-grafana)) |

______________________________________________________________________

## Post-Phase 4 Options

| Component                                                    | Sync    | Next step                                                                                                             |
| ------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------- |
| [vCluster Platform](https://www.vcluster.com/docs/platform/) | manual  | [Phase 5](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md)                                        |
| [CAPN](https://capn.linuxcontainers.org/)                    | manual  | [Phase 6](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md)                                            |
| External CNI (Calico in this HomeLab; Canal or Cilium)       | Ansible | [Phase 3 — CNI](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/#cni); installed by role `k3s_cni`, not Argo CD |

### Trade-off Analysis — Post-Phase 4

| Alternative       | Advantage                                     | Cost / Risk                                             |
| ----------------- | --------------------------------------------- | ------------------------------------------------------- |
| Stop at Phase 4   | Complete management GitOps; smaller surface   | No vClusters or CAPN workload clusters                  |
| Phase 5 vCluster  | Web kubeconfig; lightweight virtual clusters  | More RAM on K3s; OAuth + Platform permissions           |
| Phase 6 CAPN      | Real kubeadm clusters on Incus                | High RAM consumption; manual sync required              |
| CNI via GitOps    | Reinstall Canal/Calico/Cilium without Ansible | Only with K3s in `flannel-backend=none`; don't mix CNIs |
| Manual sync (5/6) | Doesn't delete ephemeral clusters via prune   | Extra step in Argo CD UI                                |

## If it fails

| Symptom                                                                                | Check                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| -------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Application `Degraded`                                                                 | `kubectl describe application -n argocd <nombre>`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Gateway without address                                                                | `metallb-config`, free pool                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| SSO fails                                                                              | 1Password secrets + restart `argocd-dex-server`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
| `homelab-root` `ComparisonError` / repo not accessible                                 | Missing `argocd/repo-gitops`, or App `maceosoft-argocd` not installed in `maceosoft/gitops`: re-run `playbook-platform-secrets.yml`                                                                                                                                                                                                                                                                                                                                                                                                          |
| `arc-runners` `OutOfSync` and the listener pod restarts every few minutes              | Argo CD prunes in a loop the resources created by the ARC controller (`AutoscalingListener`, `Role`, `RoleBinding`: they copy the label `app.kubernetes.io/instance`). Fixed with `application.resourceTrackingMethod: annotation` in `argocd-cm` (already in `gitops/argocd/config`). Check: `kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.application\.resourceTrackingMethod}'` → `annotation`; if not, sync `argocd` and restart the controller: `kubectl -n argocd rollout restart statefulset argocd-application-controller` |
| Apps in `Error`: `ComparisonError … terminatingReplicas: field not declared in schema` | Argo CD is prior to 3.5 and the cluster has Kubernetes 1.34 or newer: [Update Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-argocd/#the-first-time-update-manually-then-enable-the-app)                                                                                                                                                                                                                                                                                                                                          |
| GitHub login gives 404 (`client_id=.github.clientId`)                                  | Missing `dex.*` in `argocd-secret`: run `playbook-dex-oauth-secrets.yml --tags argocd` ([4.7](#47-oauth-secrets-github-vcluster-incus-ui))                                                                                                                                                                                                                                                                                                                                                                                                   |
| Runner does not appear in GitHub                                                       | `kubectl logs -n arc-systems` of the listener; Secret `arc-runners/arc-runners-github-app` (App installed in `maceosoft`)                                                                                                                                                                                                                                                                                                                                                                                                                    |
| Pipeline `tofu` without runner or without credentials                                  | Runner `arc-runners` Online; Secrets `terraform-app`/`tofu-vars` in `arc-runners`                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| Longhorn pending                                                                       | Longhorn labels ([playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md))                                                                                                                                                                                                                                                                                                                                                                                                                                              |

Extended catalog and troubleshooting: [GitOps — deep dive](https://homelab.maceo.cl/en/gitops/index.md).

## Next

Optional:

- **[→ Phase 5 — vCluster](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md)**
- **[→ Phase 6 — CAPN (Cluster API Provider for Incus)](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md)**

Or stop here if you only need the GitOps management cluster.

# Phase 5 — vCluster Platform (optional)

## Objective

Web UI for **virtual clusters** and **K3s management** (connected cluster), with SSO (single sign-on) via Dex/GitHub and kubeconfig download from the browser.

## What You Learn

A **vCluster** exposes a Kubernetes API isolated without lifting Incus VMs; [vCluster Platform](https://www.vcluster.com/docs/platform/) adds UI, SSO, and kubeconfig download from the browser.

Learn this phase

**In simple terms:** a vCluster is a department within a department: a virtual Kubernetes cluster, isolated, that lives inside another. Each team gets "its own cluster" without paying the cost of setting up a new one.

**Key concepts**

- **Virtual cluster:** an isolated Kubernetes API running as pods on the host cluster.
- **Multi-tenancy:** sharing the same infrastructure among teams without them seeing each other.
- **Kubeconfig:** the file that gives access to a cluster; the platform allows downloading it from the browser.
- **SSO and OIDC (OpenID Connect):** logging in with GitHub through Dex, without another password.

**Practical challenge (read-only):** read the login flow from Phase 4 and explain in your own words, in four steps, what happens when someone logs in with GitHub.

What advantage does a vCluster have over a full virtual machine?

It starts in seconds and uses fewer resources, because it shares the host's kernel and infrastructure.

What role does Dex play in login?

It acts as an intermediary: it receives the GitHub login and translates it to the applications (Argo CD, Platform, Incus).

Questions? Use the **Learn with AI** button next to each title, or the page [Learn](https://homelab.maceo.cl/en/aprende/index.md).

## Stack for this phase

```
flowchart TB
  User[Usuario] --> VP[vcluster_homelab_local]
  VP --> Dex[Dex_ArgoCD]
  Dex --> GH[GitHub]
  VP --> VC[vCluster_API]
  VC --> Mgmt[Pods_en_K3s_management]
```

- **[vCluster Platform](https://www.vcluster.com/docs/platform/)** — UI + virtual control plane; Phase 5.
- **[Dex](https://dexidp.io/docs/)** — OIDC IdP (identity provider); configured in Phase 4.
- **[Gateway API](https://gateway-api.sigs.k8s.io/)** — Exposes Platform on LAN (local network) with a `HTTPRoute` over the Gateway `homelab`.

Further reading: [vCluster Platform](https://homelab.maceo.cl/en/vcluster/index.md)

**This decision in other environments**

vCluster Platform runs inside the same K3s management cluster: you create virtual clusters to test things without spinning up new VMs or using up the limited RAM. It is optional and its Sync is manual.

- **Benefits:** each team or customer gets its own cluster (its API, its CRDs, its admin permissions) without the company maintaining a real cluster for each one; ephemeral environments for each PR are created in seconds and deleted when closed, and access is centralized in the corporate SSO.
- **Isolation:** vClusters share nodes and kernel with the host cluster. For customers that do not trust each other, quotas, NetworkPolicies, and dedicated nodes are added, or separate clusters are maintained.
- **Commercial support:** vCluster is open source and vCluster Platform has an [Enterprise plan](https://www.vcluster.com/pricing) with support, SLA, and security and compliance features.
- **Permissions:** instead of manually assigning cluster access, corporate IdP groups are mapped to projects and roles, and reviewed in access audits.
- **When it makes sense like in a HomeLab:** internal platforms with many small teams, test environments for PRs, and training labs, where a real cluster per team would be too costly.

| Cloud              | Equivalent service                                                                                                                                                                |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCP                | [GKE](https://cloud.google.com/kubernetes-engine) with [fleet teams](https://cloud.google.com/kubernetes-engine/fleet-management/docs/team-management)                            |
| AWS                | [Amazon EKS](https://aws.amazon.com/eks/) ([tenant isolation](https://docs.aws.amazon.com/eks/latest/best-practices/tenant-isolation.html))                                       |
| Azure              | [AKS](https://azure.microsoft.com/en-us/products/kubernetes-service) ([cluster isolation](https://learn.microsoft.com/en-us/azure/aks/operator-best-practices-cluster-isolation)) |
| Oracle Cloud (OCI) | [OCI Kubernetes Engine (OKE)](https://www.oracle.com/cloud/cloud-native/kubernetes-engine/)                                                                                       |

In the cloud, the simple option is a cluster per team: strong isolation and SLA, but you pay for each control plane and multiply what needs to be updated. vCluster runs the same on any of them and reduces the number of real clusters, at the cost of weaker isolation and another piece to operate.

## Before You Start

- [Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md) completed (Gateway, Dex, and OAuth secrets in 1Password).
- `vcluster.homelab.local` resolves (DNS from [section 4.8](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#48-homelablocal-dns) or, if not, `/etc/hosts`).

## Run

```
kubectl apply -f gitops/argocd/apps/vcluster-platform.yaml
```

Argo CD UI → Application `vcluster-platform` → **Sync**.

With the Argo CD CLI:

```
argocd login argocd.homelab.local --grpc-web
argocd app sync vcluster-platform
```

OAuth secrets (if not in 4.7): [`argocd/secrets/1password.md`](https://github.com/maceosoft/gitops/blob/main/argocd/secrets/1password.md).

After sync of `vcluster-platform`, apply `VCLUSTER_CLIENT_SECRET` in Helm:

```
cd ansible
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml \
  --tags vcluster -e dex_oauth.apply_vcluster=true
```

## Options

| Decision             | Default HomeLab                                                                                          |
| -------------------- | -------------------------------------------------------------------------------------------------------- |
| Sync Argo CD         | Manual (commented in `homelab-root`)                                                                     |
| Admin login          | `auth.password.disabled: false` until validating SSO GitHub                                              |
| SSO user permissions | **Deny by default** — without Cluster Access / project assignment, no clusters or kubeconfig are visible |

Trade-off analysis

Manual sync

Avoids accidental deletion of Platform

Extra step in Argo CD

Admin password enabled

Rescue access without GitHub

Attack surface if not disabled after SSO

GitHub SSO only

Single IdP; aligned with Dex

No login if Dex/GitHub are down

Web kubeconfig (Platform)

No `scp`; centralized permissions

Requires explicit Cluster Access assignment

CLI (`vcluster connect`)

Automatable in scripts

No SSO integrated in the flow

## Permissions: deny by default

[vCluster Platform](https://www.vcluster.com/docs/platform/administer/users-permissions/overview) does not grant access to connected clusters or vClusters just by logging in. A new SSO user **does not see resources** until an admin assigns:

- **Cluster Access** — management K3s kubeconfig (connected cluster)
- **Project + role** — create/manage vClusters within a project
- **Management Role** — platform operations (only for admins)

Typical steps for the first operator (as Platform admin):

1. **Clusters** → **Cluster Access** → Create → user/team + ClusterRole
1. For vClusters: assign the user to a **Project** with the appropriate role
1. GitHub teams arrive as groups `loft-*` (OIDC)

## Verify

```
kubectl get pods -n vcluster-platform
curl -kI https://vcluster.homelab.local
```

1. Open [`https://vcluster.homelab.local`](https://vcluster.homelab.local)
1. Login with SSO (GitHub)
1. New Virtual Cluster → Download kubeconfig

## Management K3s kubeconfig

[vCluster Platform](https://www.vcluster.com/docs/platform/) automatically registers the **host** cluster (the K3s where Platform runs) as a **connected cluster**. From the same UI, you get the management cluster's kubeconfig, not just the vClusters'.

1. SSO login in [`https://vcluster.homelab.local`](https://vcluster.homelab.local)
1. **Clusters** → select the host cluster (e.g., the K3s context name)
1. **Connect** / **Download kubeconfig**
1. `export KUBECONFIG=~/Downloads/kubeconfig.yaml && kubectl get nodes`

If you do not see the cluster or cannot download the kubeconfig, assign **Cluster Access** to the user or team in Platform. Without this assignment, SSO login works but **no clusters or kubeconfig are visible** (deny by default). GitHub OIDC groups arrive with prefix `loft-` (see [`platform.yaml`](https://github.com/maceosoft/gitops/blob/main/vcluster/values/platform.yaml)).

## HomeLab web access

| Resource       | URL                                                                    | What you get                                   |
| -------------- | ---------------------------------------------------------------------- | ---------------------------------------------- |
| Management K3s | [`https://vcluster.homelab.local`](https://vcluster.homelab.local)     | Kubeconfig of the host K3s (connected cluster) |
| vClusters      | [`https://vcluster.homelab.local`](https://vcluster.homelab.local)     | Kubeconfig of each virtual cluster             |
| Incus cluster  | [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443) | Incus admin UI (OIDC after Phase 4)            |

Alternative CLI (command-line interface) for management K3s: [`fase-4-gitops.md` 4.1](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#41-kubectl-access) (`ssh … sudo cat` or Ansible).

## If it fails

| Symptom                 | Check                                                          |
| ----------------------- | -------------------------------------------------------------- |
| SSO loops back to login | Dex secrets + `playbook-dex-oauth-secrets.yml --tags vcluster` |
| PVC pending             | OpenEBS/Longhorn Healthy                                       |

## Next

Optional: **[→ Phase 6 — CAPN (Cluster API Provider for Incus)](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md)** · **[HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md)**

# Phase 6 — CAPN (optional)

## Objective

Provision **workload clusters** kubeadm on Incus instances via [Cluster API](https://cluster-api.sigs.k8s.io/) and [CAPN (Cluster API Provider for Incus)](https://capn.linuxcontainers.org/).

## What You Learn

CAPI (Cluster API) declares clusters as Kubernetes resources; CAPN translates `LXCMachine` into Incus instances — ephemeral clusters that do not touch the management K3s (lightweight Kubernetes distribution).

Learn this phase

**In simple terms:** Cluster API is a 3D printer for clusters: instead of building each cluster by hand, you write a description and the system fabricates it. CAPN is the "accessory" that uses Incus as the factory: each node of the new cluster is an Incus instance.

**Key concepts**

- **Management cluster and workload cluster:** the former manages; the latter is the one that is created and used.
- **Provider:** the plugin that knows how to create machines on a specific platform (here, Incus).
- **Declarative resources:** a cluster is described as a Kubernetes object (for example `LXCMachine`).
- **Ephemeral clusters:** created for a test and destroyed afterward.

**Practical challenge (safe):** generate the description of a cluster and read it, **without applying it**. First export the variables that appear in the "Run" section of this phase; then:

```
clusterctl generate cluster demo -i incus \
  --kubernetes-version v1.36.2 \
  --control-plane-machine-count 1 \
  --worker-machine-count 2 > /tmp/demo.yaml
# solo lectura: no ejecutes kubectl apply
less /tmp/demo.yaml
```

Why create clusters from a manifest and not by hand?

They repeat the same way every time, they stay in Git and can be destroyed and recreated without effort.

What is the difference between a management cluster and a workload cluster?

The management cluster creates and monitors the others; workload clusters run the applications.

Questions? Use the **Learn with AI** button next to each title, or the page [Learn](https://homelab.maceo.cl/en/aprende/index.md).

## Stack for this phase

```
sequenceDiagram
  participant Op as Operador
  participant K3s as K3s_management
  participant CAPI as Cluster_API
  participant CAPN as CAPN_controller
  participant Incus as Incus_API
  participant WL as Workload_cluster
  Op->>K3s: clusterctl_init
  Op->>K3s: apply_demo_yaml
  CAPI->>CAPN: reconcile_LXCMachine
  CAPN->>Incus: crear_LXC
  Incus->>WL: kubeadm_listo
  Op->>WL: kubectl_workload
```

- **[Cluster API](https://cluster-api.sigs.k8s.io/)** — Declarative API; Phase 6.
- **[CAPN](https://capn.linuxcontainers.org/)** — Incus provider; Phase 6.
- **[clusterctl](https://cluster-api.sigs.k8s.io/user/quick-start.html)** — Bootstrap CAPI in the management cluster.
- **[Incus](https://linuxcontainers.org/incus/docs/main/)** — Instance runtime; Phase 2.
- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** — Encrypted Incus credentials in Git.

Further reading: [CAPN](https://homelab.maceo.cl/en/capn/index.md) · [Secrets](https://homelab.maceo.cl/en/secrets/index.md)

**This decision in other environments**

Cluster API with [CAPN](https://github.com/lxc/cluster-api-provider-incus) creates kubeadm test clusters as Incus instances from a manifest in Git, and deletes them when done. It serves to learn the same model that large platforms use, without paying for cloud.

- **Benefits:** clusters are declared as code: they are created the same way every time, versioned and audited in Git, updated node by node with a version change, and replaced instead of being fixed by hand. With the same API, clusters are managed in the data center and across multiple clouds.
- **Fleet scale:** with dozens or hundreds of clusters, inventory, policies, and centralized updates are needed; a fleet console is often added, such as [SUSE Rancher Prime](https://www.suse.com/products/rancher/) or [Red Hat Advanced Cluster Management](https://www.redhat.com/en/technologies/management/advanced-cluster-management).
- **The management cluster is critical:** if it is lost, the clusters keep running but no one can update or scale them; in production, it is given high availability and backup, and it is moved with `clusterctl move`.
- **Support:** CAPN is a community project of [Linux Containers](https://linuxcontainers.org/); cloud providers have more users and vendor support.
- **When the same applies as in a HomeLab:** private cloud on Incus, edge sites, and test environments that are created and destroyed often, where a managed cloud cluster is not available or does not make sense.

| Cloud              | Equivalent service                                                                                                                                                                                                                                |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCP                | Provider [CAPG](https://github.com/kubernetes-sigs/cluster-api-provider-gcp); fleet with [GKE fleet management](https://cloud.google.com/kubernetes-engine/fleet-management/docs)                                                                 |
| AWS                | Provider [CAPA](https://github.com/kubernetes-sigs/cluster-api-provider-aws); outside AWS, [EKS Anywhere](https://anywhere.eks.amazonaws.com/) or [EKS Hybrid Nodes](https://docs.aws.amazon.com/eks/latest/userguide/hybrid-nodes-overview.html) |
| Azure              | Provider [CAPZ](https://github.com/kubernetes-sigs/cluster-api-provider-azure); fleet with [Azure Arc-enabled Kubernetes](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/overview)                                                  |
| Oracle Cloud (OCI) | Provider [CAPOCI](https://github.com/oracle/cluster-api-provider-oci)                                                                                                                                                                             |

Replacing CAPN with a cloud provider gives you managed machines, networks, and load balancers, with SLAs and near-unlimited scale. You lose control of the hardware, pay for every node that is running, and the manifests use cloud-specific resources, so they do not move as-is to another cloud.

## Before You Start

- [Phase 2 — Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md) with the healthy cluster.
- [Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md) with Sealed Secrets operational.
- Sealed Incus credential (`lxc-secret`).

The [Phase 5 — vCluster](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md) is optional: it is not required for this phase.

Control plane load balancer

Each CAPN cluster creates its own load balancer for the API (`LOAD_BALANCER='lxc: {}'`, a container inside Incus). It is independent of the Incus network load balancers, which only exist in the OVN network `ovn-lan`: see [Load Balancing and OVN](https://homelab.maceo.cl/en/incus/cluster-setup/#load-balancing-and-ovn).

## Run

### Install clusterctl and init CAPN

```
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.10.10/clusterctl-linux-amd64 -o clusterctl
chmod +x clusterctl && sudo mv clusterctl /usr/local/bin/
mkdir -p ~/.cluster-api
curl -o ~/.cluster-api/clusterctl.yaml https://capn.linuxcontainers.org/static/v0.1/clusterctl.yaml
clusterctl init -i incus
```

### Generate and apply demo cluster

See full commands in [CAPN — further reading](https://homelab.maceo.cl/en/capn/index.md).

```
kubectl apply -f gitops/argocd/apps/capn-demo.yaml
```

Manual sync in ArgoCD (Application `capn-demo`).

With the Argo CD CLI:

```
argocd login argocd.homelab.local --grpc-web
argocd app sync capn-demo
```

## Options

| Decision       | Default HomeLab                                 |
| -------------- | ----------------------------------------------- |
| Sync Argo CD   | Manual (ephemeral, avoids accidental deletions) |
| Target workers | `@arm64-nodes` (deborah) / CP `@x86-nodes`      |

Trade-off analysis

Manual sync

Ephemeral clusters do not delete themselves

Apply + Sync in UI each time

Workers on `@arm64-nodes`

Leverages 31 GB of deborah

Less margin if deborah is already a CP K3s

CP on `@x86-nodes`

Separates CP workload / management

More RAM on limited x86 nodes

CAPN vs vCluster

Real VMs/Incus; full kubeadm

Much more RAM than vCluster in pods

## Verify

```
kubectl get clusters
kubectl get machines
```

## If it fails

| Symptom                        | Check                                                             |
| ------------------------------ | ----------------------------------------------------------------- |
| The `Machine` does not advance | Secret Incus, Incus IP in `lxc-secret`                            |
| Image pull                     | [images.linuxcontainers.org](https://images.linuxcontainers.org/) |

## Next

**[→ HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md)** · [Tech Stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md)

# Implementation — Step-by-Step Guide

This section is the **main journey** of the HomeLab: from bare-metal nodes to full GitOps. Each phase is executable, verifiable, and links to the [tech stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md) and pages for deeper dives.

How to use this guide

Follow the phases in order the first time. In subsequent deployments, you can skip to the phase you need if the prerequisites are already met. Where there are **multiple options**, each phase includes a brief **Trade-offs Analysis** (benefit vs cost/risk).

## Phase Timeline

```
flowchart LR
  F0[Fase_0_Preparacion] --> F1[Fase_1_Red]
  F1 --> F2[Fase_2_Incus]
  F2 --> F3[Fase_3_K3s]
  F3 --> F4[Fase_4_GitOps]
  F4 --> F5[Fase_5_vCluster]
  F4 --> F6[Fase_6_CAPN]
```

0

[Preparation](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/index.md) Completed

Repo, Ansible, inventory, optional discovery.

`playbook-discovery.yml`

1

[Network](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md) Completed

Fixed IP on `br0` on the 3 nodes.

`playbook-set-static-ip.yml`

2

[Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md) Completed

HA Incus cluster + cluster groups + UI.

`playbook-bootstrap.yml` + `playbook-incus-cluster.yml`

3

[K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md) Completed

Management cluster + Argo CD controller.

`playbook-k3s.yml`

4

[GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md) Completed

MetalLB, Gateway API, storage, Dex, waves.

Argo CD + `gitops`

5

[vCluster](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md) Optional

Virtual clusters + web kubeconfig.

Application manual

6

[CAPN](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md) Optional

Workload clusters on Incus.

`clusterctl` + Application manual

**End of guide:** **[HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md)** — global checklist, URLs, `/etc/hosts`, nodes, and verification.

## Two Repos, Two Responsibilities

| Repo                                              | What it Installs                                     | Tool                                       |
| ------------------------------------------------- | ---------------------------------------------------- | ------------------------------------------ |
| [`homelab`](https://github.com/maceosoft/homelab) | OS network, Incus, K3s, kube-vip, Argo CD controller | [Ansible](https://docs.ansible.com/)       |
| [`gitops`](https://github.com/maceosoft/gitops)   | MetalLB, Gateway API, storage, vCluster, CAPN, …     | [Argo CD](https://argo-cd.readthedocs.io/) |

## Tech Stack

Layered diagram, list of products with official links and role in the HomeLab:

**[See full tech stack →](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md)**

Complementary architectural view: [C4 Architecture](https://homelab.maceo.cl/en/architecture/overview/index.md).

## Decision Map (forks)

| Decision        | Option A (default HomeLab)                                                                               | Option B                                                                                                                                                                                         |
| --------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Stable K3s API  | Name `k3s.homelab.local` to the control plane (`deborah`, `192.168.20.5`)                                | [kube-vip](https://kube-vip.io/) (`kube_vip.enabled: true` on `k3s_install.yml`)                                                                                                                 |
| Install K3s     | `playbook-k3s.yml`                                                                                       | Manual curl ([reference](https://homelab.maceo.cl/en/k3s/index.md))                                                                                                                              |
| **CNI**         | [Calico](https://docs.tigera.io/calico/latest/about/) (`k3s_cni: calico`)                                | Embedded [Flannel](https://github.com/flannel-io/flannel) (`vxlan`), [Canal](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/flannel), [Cilium](https://docs.cilium.io/) |
| OAuth Secrets   | [1Password SDK](https://github.com/1Password/onepassword-sdk-python) — personal account, vault `HomeLab` | Corporate account or other vault via env vars                                                                                                                                                    |
| vCluster / CAPN | Commented in `homelab-root`                                                                              | Uncomment in `root-appset.yaml` when ready                                                                                                                                                       |
| HA Storage      | [Longhorn](https://longhorn.io/docs/) on invincible+deborah                                              | Only [OpenEBS](https://openebs.io/docs)                                                                                                                                                          |

Trade-off Analysis

Fixed IP vs kube-vip

Fixed IP: no extra components

kube-vip: stable VIP `192.168.20.4` if CP changes node

kube-vip

API on dedicated IP; agents do not depend on hostname

One more daemon; does not replace MetalLB

Ansible vs curl K3s

Idempotent, flags in `group_vars`

Manual only for occasional debugging

Flannel

Embedded in K3s; low consumption

No native NetworkPolicy

Calico selected · Canal / Cilium

Network policies (Canal/Calico) or eBPF (Cilium)

`--flannel-backend=none`; more RAM/CPU; choose **before** bootstrap

1Password SDK

Secrets out of Git; local DesktopAuth

Requires unlocked 1Password app on the station

env vars / `kubectl patch`

No 1Password

Secrets in shell/history; not idempotent

Manual sync vCluster/CAPN

Avoid accidental deletion of ephemeral clusters

Extra step in Argo CD UI

Longhorn

Replication x86↔arm64

Designed for 2 nodes; more pods than OpenEBS

Only OpenEBS

LocalPV on 3 nodes; simple

No HA between nodes

Details CNI (Container Network Interface): [Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/#options) · [playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/#cni).

Details K3s flags (lightweight Kubernetes distribution): [Playbook options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md).

Ansible role catalog: [Ansible — roles](https://homelab.maceo.cl/en/ansible/index.md).

## After the phases

With the platform running, add the **local AI assistant** (Ollama on several nodes behind a router, with Open WebUI and Open Terminal), the **centralized logs** with weekly reports with AI, and the **security** layers (host firewall, service tokens, stage-based NetworkPolicy). They are in [Operation](https://homelab.maceo.cl/en/operacion/index.md): [Local AI assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md), [Logs](https://homelab.maceo.cl/en/operacion/logs/index.md) and [Authentication model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md). If the control plane ever needed to change nodes, the procedure is in [Move the control plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md).

## Start here

**[→ Phase 0 — Preparation](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/index.md)**

Appendix outside the normal path: [Uninstall K3s](https://homelab.maceo.cl/en/implementacion/apendice-desinstalar/index.md).

# Homelab Summary

Completion of the [implementation guide](https://homelab.maceo.cl/en/implementacion/index.md): checklist of what was deployed, URLs, `/etc/hosts`, nodes, secrets, and global verification. Check this page upon finishing Phase 4 at a minimum; Phases 5 and 6 are optional.

## Completed Walkthrough (by phase)

| Phase                                                                                            | What is Ready                                           | Tool                                                    |
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------------------------------- |
| [0 — Preparation](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/index.md)        | Repo, Ansible, `inventory.ini`, optional discovery      | `playbook-discovery.yml`                                |
| [1 — Network](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md)                    | Fixed IP on `br0` (`192.168.20.5`, `.6`, `.7`)          | `playbook-set-static-ip.yml`                            |
| [2 — Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md)                    | HA cluster, groups, UI, storage `dir`                   | `playbook-bootstrap.yml` + `playbook-incus-cluster.yml` |
| [3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)                        | Management cluster + Argo CD controller; CNI **Calico** | `playbook-k3s.yml`                                      |
| [4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)                  | MetalLB, Gateway API, storage, Dex/GitHub SSO           | Argo CD + `playbook-dex-oauth-secrets.yml`              |
| [5 — vCluster](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md) *(optional)* | Platform, kubeconfig web (connected cluster)            | Application Argo CD                                     |
| [6 — CAPN](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md) *(optional)*         | Workload clusters on Incus                              | `clusterctl` + `capn-demo.yaml`                         |

## URLs and Web Access

| Service                       | URL                                                                              | Auth                                                         | Phase                                                                                                    |
| ----------------------------- | -------------------------------------------------------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------- |
| **Argo CD**                   | [`https://argocd.homelab.local`](https://argocd.homelab.local)                   | GitHub SSO via Dex                                           | 4                                                                                                        |
| **Dex** (issuer)              | `https://argocd.homelab.local/api/dex`                                           | —                                                            | 4                                                                                                        |
| **vCluster Platform**         | [`https://vcluster.homelab.local`](https://vcluster.homelab.local)               | SSO Dex → GitHub                                             | 5                                                                                                        |
| **Incus UI**                  | [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443)           | SSO Dex (or client cert before Phase 4)                      | 2 / 4                                                                                                    |
| **Grafana** *(optional)*      | [`https://grafana.homelab.local`](https://grafana.homelab.local)                 | GitHub SSO via Dex (backup: `admin` of item `grafana-admin`) | 4 ([4.13](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional)) |
| **Open WebUI** (AI assistant) | [`https://open-webui.homelab.local`](https://open-webui.homelab.local)           | GitHub SSO via Dex (team `devops`)                           | [local AI](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)                                     |
| **Ollama** (router)           | `https://ollama.homelab.local`                                                   | Bearer token by client (not for people)                      | [local AI](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)                                     |
| **K3s API**                   | `https://k3s.homelab.local:6443` (currently points to `deborah`, `192.168.20.5`) | kubeconfig                                                   | 3                                                                                                        |

### Kubeconfig

| Cluster        | How to Obtain It                                                                            |
| -------------- | ------------------------------------------------------------------------------------------- |
| Management K3s | vCluster Platform (connected cluster) or `playbook-k3s.yml` with `fetch_kubeconfig`         |
| vClusters      | UI [vCluster Platform](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md) |
| Workload CAPN  | `clusterctl get kubeconfig` (Phase 6)                                                       |

## DNS on Your Workstation

Use `192.168.20.5` (BIND on `deborah`) as DNS: `argocd`, `vcluster`, `api`, `grafana`, `open-webui`, and `ollama` (`.homelab.local`) point to the Gateway (`192.168.23.200`, fixed IP) and `incus.homelab.local` to invincible ([Phase 4 — 4.8](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#48-homelablocal-dns)). If you cannot, the backup is `/etc/hosts`:

```
192.168.23.200  argocd.homelab.local vcluster.homelab.local api.homelab.local grafana.homelab.local open-webui.homelab.local ollama.homelab.local
192.168.20.6    incus.homelab.local
```

Verification:

```
curl -kI https://argocd.homelab.local
curl -kI https://vcluster.homelab.local
# solo si instalaste el monitoreo (4.13)
curl -kI https://grafana.homelab.local
curl -kI https://incus.homelab.local:8443
```

## Homelab Nodes

| Hostname   | IP `br0`     | Incus                    | K3s             | Notes                                                     |
| ---------- | ------------ | ------------------------ | --------------- | --------------------------------------------------------- |
| invincible | 192.168.20.6 | Leader + UI              | agent           | 15.5 GB; Longhorn disk; Ollama node                       |
| oliver     | 192.168.20.7 | Quorum, manual scheduler | agent           | 7.7 GB: RAM adjusted; no automatic instances; Ollama node |
| deborah    | 192.168.20.5 | arm64-nodes              | **server** (CP) | 31 GB RAM; DNS (BIND); Ollama node                        |

The control plane is on `deborah`. If it ever needed to move to another node, the procedure is in [Moving the Control Plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md).

Incus storage: pool `local`, driver `dir`, profile `default` → NIC (network card) `br0`.

Hardware details: [Inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md).

## Cluster Network (CNI)

| Decision | Default Homelab                                                                                     | Alternative                                        |
| -------- | --------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| CNI      | **Calico** (`k3s_cni: calico`; later adopted by Argo CD with `calico-operator` and `calico-config`) | Flannel `vxlan` (embedded in K3s), Canal or Cilium |

Ansible variable: `k3s_install.core.cni` in [`group_vars/incus_cluster/k3s_install.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/k3s_install.yml). Details and trade-offs: [Phase 3 — CNI](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/#cni) · [playbook-options](https://homelab.maceo.cl/en/k3s/playbook-options/#cni).

Verification:

```
kubectl get pods -n calico-system -l k8s-app=calico-node
# o flannel / canal / cilium si elegiste otro CNI
```

## Secrets and SSO (1Password)

| 1Password Field                             | Usage                                              |
| ------------------------------------------- | -------------------------------------------------- |
| `GITHUB_CLIENT_ID` / `GITHUB_CLIENT_SECRET` | Dex → GitHub OAuth (manual in GitHub + 1Password)  |
| `VCLUSTER_CLIENT_SECRET`                    | Dex + Helm Platform (Ansible generates if missing) |

Personal account, vault **`HomeLab`** (configurable via env vars). Playbook: `playbook-dex-oauth-secrets.yml`.

Default Permissions

In vCluster Platform, SSO users (single sign-on) **see nothing** until access is assigned. In Incus, anyone who logs in via SSO has full access: the control is that Dex only allows the team `devops` of `maceosoft`. See [Phase 4](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md) and [Phase 5](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md).

Documentation: [OAuth Secrets](https://github.com/maceosoft/gitops/blob/main/argocd/secrets/1password.md)

## GitOps (Argo CD)

Typical apps in `Healthy` after Phase 4:

- `argocd`, `argocd-route`, `metallb`, `homelab-storage`, `kong`, …
- Optional (manual sync): `vcluster-platform`, `capn-demo`
- CNI: `calico-operator` + `calico-config` (manual sync); `canal` or `cilium` as alternatives — **one** at a time
- AI and logs: `ollama-router`, `open-webui`, `open-terminal`, `ai-tools`, `alert-triage`, `log-report`, `victoria-logs`, `vector`; `k8sgpt` is staged

Repo: [`gitops`](https://github.com/maceosoft/gitops) · [`bootstrap/root-appset.yaml`](https://github.com/maceosoft/gitops/blob/main/bootstrap/root-appset.yaml)

## After Phases: AI, logs, and security

On top of the Phase 4 platform, the following are added (each with its guide):

- **Local AI**: Ollama on the three nodes (and an optional Mac) behind a router with automatic registration; Open WebUI with Dex and Open Terminal. [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md) · [Add an Ollama Node](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md).
- **Logs and reports**: Vector → VictoriaLogs → Grafana, with a daily summary and a weekly report with AI in the Discord channel `#ia`. [Centralized Logs](https://homelab.maceo.cl/en/operacion/logs/index.md).
- **Security**: people → Dex; machines → token and network restriction; host firewall on each node. The NetworkPolicy `default-deny` and K8sGPT are **to be activated**. [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md).

## Global Verification

```
# Incus
incus cluster list
incus cluster group list

# K3s
kubectl get nodes
kubectl get applications -n argocd

# SSO — login GitHub en ArgoCD, Grafana, Open WebUI, vCluster e Incus UI

# CAPN (si Fase 6)
kubectl get clusters,machines
```

## Official stack links

- [Incus](https://linuxcontainers.org/incus/docs/main/) · [K3s](https://docs.k3s.io/) · [Argo CD](https://argo-cd.readthedocs.io/)
- [vCluster Platform](https://www.vcluster.com/docs/platform/) · [Cluster API](https://cluster-api.sigs.k8s.io/) · [CAPN](https://capn.linuxcontainers.org/)
- [1Password SDK](https://github.com/1Password/onepassword-sdk-python)

## Maintenance and deepening

- [Technology stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md)
- [Ansible Catalog](https://homelab.maceo.cl/en/ansible/index.md)
- [GitOps](https://homelab.maceo.cl/en/gitops/index.md)
- [Backup Incus (optional)](https://homelab.maceo.cl/en/incus/cluster-setup/#backup-and-snapshots-optional)
- [Uninstall K3s](https://homelab.maceo.cl/en/implementacion/apendice-desinstalar/index.md)

# Technology Stack

Inventory of HomeLab technologies: **layer diagram** + list with official links in the name of each product.

Part of the [implementation guide](https://homelab.maceo.cl/en/implementacion/index.md). Complementary C4 (C4 model) view: [Architecture](https://homelab.maceo.cl/en/architecture/overview/index.md) · [Level 3 — Components](https://homelab.maceo.cl/en/architecture/nivel-3-componentes/index.md) · [Level 4 — Deployment](https://homelab.maceo.cl/en/architecture/nivel-4-despliegue/index.md).

## Layer Diagram

```
flowchart TB
  subgraph capa0 [Capa 0 — Hardware, SO y bootstrap]
    HW[Nodos_fisicos_Debian]
    Netplan[Netplan_br0]
    Ansible[Ansible]
  end
  subgraph capa1 [Capa 1 — Virtualización]
    Incus[Incus_cluster]
    IncusUI[Incus_UI]
  end
  subgraph capa2 [Capa 2 — Kubernetes management]
    K3s[K3s]
    KubeVip[kube_vip_opcional]
  end
  subgraph capa3 [Capa 3 — GitOps, red y acceso]
    ArgoCD[Argo_CD]
    Dex[Dex]
    GitHub[GitHub_OAuth]
    MetalLB[MetalLB]
    Gateway[Gateway_API_Kong]
  end
  subgraph capa4 [Capa 4 — Storage y secretos]
    OpenEBS[OpenEBS]
    Longhorn[Longhorn]
    SealedSecrets[Sealed_Secrets]
    OnePassword[1Password_SDK]
  end
  subgraph capa5 [Capa 5 — Opcionales y operaciones]
    VCluster[vCluster_Platform]
    CAPI[Cluster_API]
    CAPN[CAPN]
    SUC[system_upgrade_controller]
  end
  subgraph capa6 [Capa 6 — IA local, logs y seguridad]
    Ollama[Ollama_en_nodos_y_Mac]
    Router[ollama_router]
    OpenWebUI[Open_WebUI_y_Open_Terminal]
    Vector[Vector]
    VLogs[VictoriaLogs]
    Firewall[nftables_por_nodo]
  end
  HW --> Netplan --> Incus
  Incus --> IncusUI
  Dex --> IncusUI
  Netplan --> K3s
  K3s --> KubeVip
  K3s --> ArgoCD
  ArgoCD --> Dex
  Dex --> GitHub
  ArgoCD --> MetalLB --> Gateway
  ArgoCD --> OpenEBS
  ArgoCD --> Longhorn
  ArgoCD --> SealedSecrets
  Dex --> VCluster
  K3s --> CAPI --> CAPN
  Incus --> CAPN
  ArgoCD --> SUC
  ArgoCD --> Router --> Ollama
  OpenWebUI --> Router
  Dex --> OpenWebUI
  Vector --> VLogs
  Ansible --> Firewall
  Ansible --> Ollama
```

## Layer 0 — Hardware, OS, and bootstrap

- **[Ansible](https://docs.ansible.com/)** — Idempotent OS, network, Incus, and K3s (lightweight Kubernetes distribution) automation; Phases 0–3.
- **[Netplan](https://netplan.io/)** — Static IP on `br0`; Phase 1; node identity for Incus/K3s/CAPN (Cluster API Provider for Incus).
- **[Debian](https://www.debian.org/)** — Base OS on the 3 nodes; global prerequisite.

## Layer 1 — Virtualization

- **[Incus](https://linuxcontainers.org/incus/docs/main/)** — LXC (Linux container) cluster/VM with quorum; Phase 2; substrate for CAPN and instances.
- **[Incus UI](https://linuxcontainers.org/incus/docs/main/howto/server_expose/#web-ui)** — Web management of the Incus cluster; Phase 2 (cert) + Phase 4 (OIDC —OpenID Connect— with Dex).
- **[Zabbly Incus stable](https://github.com/zabbly/incus)** — Package repo; Phase 2.

## Layer 2 — Kubernetes Management

- **[K3s](https://docs.k3s.io/)** — Bare-metal management cluster; Phase 3.
- **[kube-vip](https://kube-vip.io/)** — Optional API (Application Programming Interface) VIP (Virtual IP) `:6443`; Phase 3 (fork).
- **CNI (choose one; in this HomeLab, Calico)** — Phase 3:
- **[Flannel](https://github.com/flannel-io/flannel)** — Default embedded in K3s (`vxlan` in `br0`)
- **[Canal](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/flannel)** — Flannel + Calico policies
- **[Calico](https://docs.tigera.io/calico/latest/about/)** — NetworkPolicy, optional BGP (Border Gateway Protocol)
- **[Cilium](https://docs.cilium.io/)** — eBPF (extended Berkeley Packet Filter), Hubble, L3–L7 policies

CNI

Flannel

Zero extra steps in K3s

No native NetworkPolicy

Canal

Policies over known overlay

Legacy; two stacks

Calico

Maturity and BGP at L2

More pods/RAM

Cilium

eBPF observability

CPU/kernel; validate ARM64

## Layer 3 — GitOps, network, and access

- **[Argo CD](https://argo-cd.readthedocs.io/)** — Bootstrap ApplicationSet (`homelab-root`) from GitHub, by waves; Phase 4.
- **[Dex](https://dexidp.io/docs/)** — Embedded OIDC (OpenID Connect) IdP (identity provider) in Argo CD; SSO (single sign-on) for Argo CD, Grafana, Open WebUI, vCluster Platform, and Incus UI (user interface); Phase 4.
- **[GitHub](https://docs.github.com/en/apps/oauth-apps)** — OAuth (Open Authorization) connector + GitOps repos; Phase 4.
- **[MetalLB](https://metallb.universe.tf/)** — LoadBalancer in LAN (local network); Phase 4 wave 0–1.
- **[Gateway API](https://gateway-api.sigs.k8s.io/)** with **[Kong Ingress Controller](https://developer.konghq.com/kubernetes-ingress-controller/)** — HTTP(S) to Argo CD, Grafana, Open WebUI, Ollama router, and vCluster; Phase 4 wave 1 (alternatives: Traefik, NGINX Gateway Fabric).
- **[1Password Python SDK](https://github.com/1Password/onepassword-sdk-python)** — OAuth secrets from local app; Phase 4.
- **[VictoriaMetrics](https://docs.victoriametrics.com/)** (with **[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)**) — Node and platform metrics and alerts, lighter than Prometheus; Phase 4 (4.13, optional).
- **[Grafana](https://grafana.com/docs/grafana/latest/)** — Temperature, memory, disk, logs, and Ollama panels in `grafana.homelab.local`; Phase 4 (4.13, optional).
- **[Discord](https://discord.com/developers/docs/resources/webhook)** and **[healthchecks.io](https://healthchecks.io/docs/)** — Alert destination (webhooks) and external watchdog (*dead man's switch*) that receives a ping from Alertmanager, in case the entire cluster goes down; Phase 4 (4.13, optional).

## Layer 4 — Storage and secrets

- **[OpenEBS](https://openebs.io/docs)** — Default LocalPV; Phase 4 wave 0.
- **[Longhorn](https://longhorn.io/docs/)** — Cross-arch replicated storage; Phase 4 wave 1.
- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** — Encrypted secrets in Git; Phase 4 and 6.
- **[SOPS (Secrets OPerationS)](https://github.com/getsops/sops)** — Pre-bootstrap encryption; deeper dive [Secrets](https://homelab.maceo.cl/en/secrets/index.md).

## Layer 5 — Optional and operations

- **[vCluster Platform](https://www.vcluster.com/docs/platform/)** — Virtual clusters + web kubeconfig (host K3s and vClusters); Phase 5.
- **[Cluster API](https://cluster-api.sigs.k8s.io/)** — Declarative cluster API; Phase 6.
- **[CAPN](https://capn.linuxcontainers.org/)** — Incus provider for CAPI (Cluster API); Phase 6.
- **[clusterctl](https://cluster-api.sigs.k8s.io/user/quick-start.html)** — CAPI bootstrap CLI (command-line interface); Phase 6.
- **[system-upgrade-controller](https://github.com/rancher/system-upgrade-controller)** — Staged K3s upgrades; Phase 4 wave 3.

## Layer 6 — Local AI, logs, and security

- **[Ollama](https://ollama.com/)** — Language models (LLM, *Large Language Model*) on CPU (central processing unit), on-demand in `deborah`, `invincible` and `oliver`, and on nodes that join as nodes. [Guide](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md) · [add a node](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md).
- **Ollama Router** (`ollama-router`, custom) — Single entry point (`ollama.homelab.local`): automatic node registration, model selection, client tokens, and model and node metrics.
- **[Open WebUI](https://docs.openwebui.com/)** — Chat with the assistant, with Dex login; **[Open Terminal](https://docs.openwebui.com/features/open-terminal)** as a sandbox terminal for the assistant to run code.
- **K8sGPT** — Cluster analysis with Ollama; **staged** (its operator requests cluster-wide permissions).
- **[Vector](https://vector.dev/)** → **[VictoriaLogs](https://docs.victoriametrics.com/victorialogs/)** → Grafana — Centralized logs (30 days) and daily and weekly reports with AI in the `#ia` Discord channel ([guide](https://homelab.maceo.cl/en/operacion/logs/index.md)).
- **[nftables](https://wiki.nftables.org/)** — Host firewall on each node (role `host_firewall`), with automatic rollback.
- **NetworkPolicy** `default-deny` by namespace and **`oauth2-proxy`** — prepared and **to be activated** in stages ([Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md)).

## Each layer in the enterprise and in the cloud

The same stack, viewed through each lens. The details of each row (trade-offs, benefits, and official links) are in the "Decision in other environments" block of the linked page.

Everything runs on three machines with free software: the stack of the upper layers. No licenses and full control; in exchange, the operation, updates, and backups are yours.

| Layer                  | What changes in a company                                                 | Detail                                                                                  |
| ---------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Virtualization         | Platform with support (vSphere, Proxmox VE, Nutanix, OpenStack)           | [Phase 2](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md)             |
| Kubernetes             | 3 control planes with etcd and a supported distribution (RKE2, OpenShift) | [Phase 3](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)               |
| GitOps                 | Argo CD with commercial support; every change audited in Git              | [Phase 4](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)            |
| Networking and ingress | Dedicated load balancers or MetalLB in BGP; gateway with support          | [Networking](https://homelab.maceo.cl/en/networking/index.md)                           |
| Storage                | SAN/NAS or supported storage with backups outside the cluster             | [Storage](https://homelab.maceo.cl/en/storage/index.md)                                 |
| Secrets                | Central vault with rotation and auditing (Vault, 1Password Business)      | [Secrets](https://homelab.maceo.cl/en/secrets/index.md)                                 |
| Identity               | Corporate IdP with MFA (Entra ID, Okta, Google Workspace)                 | [Authentication](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md) |
| Monitoring             | Guards, scaling, and long-term retention                                  | [Monitoring](https://homelab.maceo.cl/en/operacion/monitoreo/index.md)                  |
| Logs                   | Compliance retention and SIEM                                             | [Logs](https://homelab.maceo.cl/en/operacion/logs/index.md)                             |
| AI                     | Local inference with GPU when data cannot leave                           | [Local AI](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)                    |
| Multi-tenancy          | A virtual cluster per team or customer                                    | [Phase 5](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md)          |
| Cluster fleet          | Cluster API or the provider's fleet console                               | [Phase 6](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md)              |
| Backups                | 3-2-1 rule with tested RPO/RTO                                            | [Backups](https://homelab.maceo.cl/en/operacion/backups-incus/index.md)                 |

| Layer                  | GCP                                       | AWS                                    | Azure                                | Oracle Cloud (OCI)                                  |
| ---------------------- | ----------------------------------------- | -------------------------------------- | ------------------------------------ | --------------------------------------------------- |
| Virtualization         | Compute Engine                            | EC2                                    | Azure Virtual Machines               | OCI Compute                                         |
| Kubernetes             | GKE                                       | EKS                                    | AKS                                  | OKE                                                 |
| Networking and ingress | Cloud Load Balancing and GKE Gateway      | ELB and AWS Load Balancer Controller   | Application Gateway for Containers   | OCI Load Balancer and Native Ingress (only Ingress) |
| GitOps                 | Config Sync                               | EKS Capability for Argo CD             | GitOps extensions (Flux and Argo CD) | Argo CD installed on OKE                            |
| Storage                | Persistent Disk, Hyperdisk, and Filestore | EBS and EFS                            | Azure Disk and Azure Files           | Block Volume and File Storage                       |
| Secrets                | Secret Manager                            | Secrets Manager                        | Key Vault                            | OCI Vault                                           |
| Identity               | Workforce Identity Federation             | IAM Identity Center                    | Microsoft Entra ID                   | IAM Identity Domains                                |
| Metrics                | Managed Service for Prometheus            | Managed Prometheus and Managed Grafana | Azure Monitor and Managed Grafana    | OCI Monitoring                                      |
| Logs                   | Cloud Logging                             | CloudWatch Logs                        | Azure Monitor Logs                   | OCI Logging                                         |
| AI                     | Gemini Enterprise Agent Platform          | Bedrock                                | Microsoft Foundry                    | OCI Generative AI                                   |
| Backups                | Backup and DR and Backup for GKE          | AWS Backup                             | Azure Backup                         | Block Volume Backups                                |

Generally, you gain managed operations, SLAs, and scale; you lose control, pay for usage, and become more tied to each provider.
# Operación (procedimientos de rutina)

# SSH into the Servers

How to SSH (Secure Shell, encrypted terminal connection) into `invincible`, `oliver` and `deborah` with your GitHub account, without requesting or saving keys. It works if you belong to the team `devops` of the org `maceosoft`. If you are the one who **installs** or administers it, go to [4.12 — SSH to the hosts with Dex](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#412-ssh-to-hosts-with-dex-optional).

How it works, in one line

You log in with GitHub through Dex (OIDC, OpenID Connect) and receive an SSH key that expires in 24 hours. Each server checks that you are in the team `devops` and lets you in as the user `devops`.

## What you need

- Be a member of the team `devops` of the org `maceosoft` on GitHub.
- A station that resolves the names `*.homelab.local` (use the DNS, domain name system, of the network: `192.168.20.5`) and that trusts the HomeLab CA (certificate authority).
- The client `opkssh`.

## Prepare your station (once)

1. **DNS.** Your station needs to use `192.168.20.5` as DNS. If you can't, the backup is the file `/etc/hosts` (see the [HomeLab Summary](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md)).

1. **Trust the HomeLab CA.** You need `kubectl` access to the cluster (`export KUBECONFIG=~/.kube/homelab-k3s.yaml`). Export the root certificate:

   ```
   kubectl -n cert-manager get secret homelab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > homelab-ca.crt
   ```

   Install it as a trusted certificate:

   ```
   sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain homelab-ca.crt
   ```

   ```
   sudo cp homelab-ca.crt /usr/local/share/ca-certificates/homelab-ca.crt
   sudo update-ca-certificates
   ```

1. **Install the client.** On macOS, `brew install opkssh`. On other systems, the [opkssh releases page](https://github.com/openpubkey/opkssh/releases).

1. **Define the provider** in `~/.opk/config.yml`:

   ```
   ---
   default_provider: homelab

   providers:
     - alias: homelab
       issuer: https://argocd.homelab.local/api/dex
       client_id: opkssh
       scopes: openid email profile groups
       redirect_uris:
         - http://localhost:3000/login-callback
         - http://localhost:10001/login-callback
         - http://localhost:11110/login-callback
   ```

   The list goes under the key `providers:`. With `default_provider` is enough `opkssh login`.

1. **Check** (in a new terminal):

   ```
   curl -s -o /dev/null -w "%{http_code} ssl_verify=%{ssl_verify_result}\n" https://argocd.homelab.local/api/dex/.well-known/openid-configuration
   ```

   It should give `200 ssl_verify=0`. With `ssl_verify=20`, Dex works but your station doesn't know the CA.

## Enter

The key lasts 24 hours. Every day:

```
opkssh login
ssh -o IdentitiesOnly=yes -i ~/.ssh/id_ecdsa devops@oliver.homelab.local
```

`opkssh login` opens the browser: you log in with GitHub and return to the terminal. `IdentitiesOnly=yes` makes SSH offer only the opkssh key; without it, your client tries other keys first and the server cuts with `Too many authentication failures`.

To avoid typing the flag every time, add to `~/.ssh/config`:

```
Host invincible oliver deborah
    HostName %h.homelab.local
    User devops
    IdentityFile ~/.ssh/id_ecdsa
    IdentitiesOnly yes
```

and enter with `ssh oliver`.

Shared account with `sudo`

Everyone logs in as `devops`, with `sudo` without a password. Whoever is in the team is an administrator of the three servers.

## If something fails

| Symptom                                                             | What it means                                                                | What to do                                                                                                                                                                                                                                                                       |
| ------------------------------------------------------------------- | ---------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `opkssh login`: `cannot unmarshal !!seq into config.ClientConfig`   | `~/.opk/config.yml` without the key `providers:`                             | Use the format from step 4                                                                                                                                                                                                                                                       |
| `opkssh login`: `invalid provider config string`                    | You used `--provider=homelab`: that flag only accepts `<emisor>,<client_id>` | Use `opkssh login` or `opkssh login homelab`                                                                                                                                                                                                                                     |
| `opkssh login` fails with `redirect_uri` not valid                  | Missing one of the three callback addresses                                  | Check `redirect_uris` from step 4                                                                                                                                                                                                                                                |
| `x509: certificate signed by unknown authority`, or `ssl_verify=20` | Your station doesn't trust the CA                                            | Step 2                                                                                                                                                                                                                                                                           |
| `Too many authentication failures`                                  | SSH offers other keys first                                                  | Use `-o IdentitiesOnly=yes`                                                                                                                                                                                                                                                      |
| `Permission denied (publickey)`                                     | The server doesn't recognize you                                             | Check with `opkssh inspect ~/.ssh/id_ecdsa-cert.pub` that the token lists the group `maceosoft:devops`. If it does, alert the administrator: the problem is on the server ([4.12](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#412-ssh-to-hosts-with-dex-optional)) |
| Doesn't resolve `oliver.homelab.local`                              | Your station doesn't use the HomeLab DNS                                     | Step 1                                                                                                                                                                                                                                                                           |

If Dex or the cluster are down, SSO doesn't work. The administrator has an emergency key on each server.

# New version notices (Renovate)

Every day, [Renovate](https://docs.renovatebot.com/) checks whether a new version of the products installed in the cluster has been released and opens a PR (pull request) in [`maceosoft/gitops`](https://github.com/maceosoft/gitops) for each one. At the end of each run, the workflow itself posts a summary in the `actualizaciones` Discord channel. **It does not update anything on its own**: you review the PR, test it, and merge it, and only then does Argo CD deploy.

```
flowchart LR
  GHA["GitHub Actions<br/>renovate.yml, cada día<br/>(runner arc-runners)"] -->|"abre PRs"| PR[("PR en maceosoft/gitops")]
  GHA -->|"resumen de los PR nuevos<br/>(y aviso si falla)"| DC["Discord<br/>#actualizaciones"]
  PR -->|"revisas, pruebas, mergeas"| ARGO["ArgoCD despliega"]
```

Menu names are in English

This guide uses the texts exactly as they appear in the English interface of GitHub and Discord (**Settings**, **Webhooks**…). These platforms change their UI from time to time: if a button doesn't match, look for the equivalent on that same screen.

## What it detects and what it doesn't

| Detects                                                                                                                                                                  | Where it looks in `gitops`               |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------- |
| Helm charts of Argo CD Applications (Kong, VictoriaMetrics, cert-manager, MetalLB, OpenEBS, Sealed Secrets, Longhorn, ARC…), including ranges like `0.95.x` or `v1.21.x` | `argocd/apps/*.yaml`                     |
| Calico (tigera-operator and CRDs)                                                                                                                                        | `cni/calico-operator/kustomization.yaml` |
| Gateway API CRDs                                                                                                                                                         | `gateway-api/kustomization.yaml`         |
| Images with fixed version (e.g. `busybox`)                                                                                                                               | `arc/values-scaleset.yaml`               |
| GitHub Actions actions (`actions/checkout@v7`…)                                                                                                                          | `.github/workflows/`                     |

| Doesn't detect                                                                                                 | Why                                                                                                                                            |
| -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| **Debian patches on nodes**                                                                                    | Already covered by OS automatic updates and monitoring alerts                                                                                  |
| **K3s**                                                                                                        | Updated by SUC plans via the `stable` channel ([Update K3s](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md)); no version in Git |
| **Argo CD**                                                                                                    | Follows the `stable` branch of the official manifest ([Update Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-argocd/index.md))      |
| **Images with `latest` tag** (the ARC runner)                                                                  | No version to compare                                                                                                                          |
| Optional apps **not active** in `bootstrap/root-appset.yaml` (Traefik, NGINX Gateway, vCluster, Cilium, Canal) | Ignored to avoid noise. If you enable one, remove its path from the corresponding rule in `renovate.json5`                                     |
| `homelab` (this repo)                                                                                          | Its Python dependencies are still watched by Dependabot (see [below](#and-dependabot-of-homelab))                                              |

### What stays as notice only

These components have their own procedure. Renovate **opens the PR as a draft** with the label `procedimiento-manual` and **does not merge** until the guide is followed:

| Component | Procedure                                                                                                                                               |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Calico    | [Calico (CNI) with Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-calico/index.md): manual sync, and only from one minor version to the next |
| Longhorn  | From one minor version to the next, with healthy and backed-up volumes, reading the Longhorn release notes first                                        |

Additionally, **every major version** arrives in a separate PR, as a draft and with the label `mayor`. No PR merges automatically (no automerge).

## The credential: a custom GitHub App

Renovate needs to write to `gitops`: create branches, open PRs, and touch `.github/workflows` files. There are two reasonable options:

| Option                                                         | Verdict                                                                                                                                                                                                                                             |
| -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Reuse the ARC App (`maceosoft-arc-runners`)                    | **No.** It has organization permissions (*Administration*, *Self-hosted runners*: Read and write) that Renovate doesn't need, and its key lives in the cluster, not in GitHub Actions. Using it otherwise expands what anyone who reaches it can do |
| **A new `maceosoft-renovate` App**, installed only on `gitops` | **Chosen.** Minimal permissions, no organization permissions, revocable without touching anything else                                                                                                                                              |

An App is used instead of a personal token (PAT, *Personal Access Token*) because it doesn't depend on a person, expires on its own (the installation token lasts 1 hour), and leaves PRs authored by the bot. Moreover, PRs opened by an App **do trigger** the `gitops` pipeline, so each one arrives with its validation and its Argo CD diff.

## Create the GitHub App

Do this with an **organization owner** account `maceosoft`.

[maceosoft → Settings → Developer settings → GitHub Apps → New GitHub App](https://github.com/organizations/maceosoft/settings/apps/new)

| Field                                   | Value                          |
| --------------------------------------- | ------------------------------ |
| GitHub App name                         | `maceosoft-renovate`           |
| Homepage URL                            | `https://github.com/maceosoft` |
| Webhook → Active                        | unchecked                      |
| Where can this GitHub App be installed? | Only on this account           |

The rest of the form is left blank or unchecked.

| Repository permission | Access         | Purpose                                                                  |
| --------------------- | -------------- | ------------------------------------------------------------------------ |
| Contents              | Read and write | Create branches and commits for each update                              |
| Pull requests         | Read and write | Open and maintain PRs                                                    |
| Workflows             | Read and write | Update `.github/workflows` files (without it, GitHub rejects the push)   |
| Issues                | Read and write | The *Dependency Dashboard*: an issue listing what's detected and pending |
| Checks                | Read and write | Publish and read branch check status                                     |
| Commit statuses       | Read and write | Publish and read commit statuses used by Renovate                        |
| Administration        | Read-only      | Read repo configuration (branch protection)                              |
| Metadata              | Read-only      | Required                                                                 |

No organization permissions. All others remain *No access*. These are the ones [Renovate documentation](https://docs.renovatebot.com/modules/platform/github/) requests for an App, without the optional ones (*Dependabot alerts*, *Merge queues*, *Members*), which aren't used here.

After *Create GitHub App* (creating the App, generating its key, and installing it is done **only from the web**):

1. On the App page, under *Private keys* → **Generate a private key**. A file `.pem` downloads.

1. Left menu → **Install App** → *Install* next to `maceosoft` → **Only select repositories** → choose **`gitops`** → *Install*. Install it **only on `gitops`**.

1. Save the App in 1Password (vault `HomeLab`, Secure Note `maceosoft-renovate`, as in [maceosoft Platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields)). The *Installation ID* isn't needed: the workflow resolves it automatically.

   | 1Password Field | Type     | Data                                                                  |
   | --------------- | -------- | --------------------------------------------------------------------- |
   | `app_id`        | Text     | **App ID**, on the App's *General* tab                                |
   | `private_key`   | Password | The full content of the `.pem`, including the `BEGIN` and `END` lines |

   With 1Password unlocked (if you have more than one 1Password account, export `OP_ACCOUNT` first; `gh` needs `gh auth refresh -s admin:read:org` once). The `.pem` is read from the file:

   ```
   APP=maceosoft-renovate
   APP_ID=$(gh api orgs/maceosoft/installations --jq ".installations[] | select(.app_slug==\"$APP\") | .app_id")
   op item create --category "Secure Note" --vault HomeLab --title "$APP" \
     "app_id[text]=$APP_ID" "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
   ```

1. Create the **GitHub Actions secrets** in repo `gitops`, one per row:

   | Secret                     | Value                                                                    |
   | -------------------------- | ------------------------------------------------------------------------ |
   | `RENOVATE_APP_ID`          | The `app_id`                                                             |
   | `RENOVATE_APP_PRIVATE_KEY` | The full content of the `.pem` (multiple lines: GitHub accepts it as-is) |

   *Settings → Secrets and variables → Actions → New repository secret*, one per table row.

   Read from the 1Password item via pipe:

   ```
   APP=maceosoft-renovate
   op read "op://HomeLab/$APP/app_id" | gh secret set RENOVATE_APP_ID --repo maceosoft/gitops
   op read "op://HomeLab/$APP/private_key" | gh secret set RENOVATE_APP_PRIVATE_KEY --repo maceosoft/gitops
   gh secret list --repo maceosoft/gitops
   ```

1. **Delete the `.pem`** from Downloads. Via console: `shred -u ~/Downloads/maceosoft-renovate.*.private-key.pem` (on macOS use `rm -P` instead of `shred -u`).

This doesn't go through `playbook-platform-secrets.yml`: it's not a cluster Secret, GitHub Actions reads it. To rotate the key: generate another in the App, update `private_key` in 1Password and secret `RENOVATE_APP_PRIVATE_KEY`, and delete the old one in GitHub.

## Create the Discord channel and webhook

There's no Discord bot: notifications arrive via a **webhook** (a URL the workflow uses to post messages to a channel; it doesn't allow reading anything). You create the channel manually, once. You need to be a server admin (or have the **Manage Webhooks** permission).

Inside the private category

Notifications include product names and infrastructure versions, and `gitops` is a private repo. Create the channel **inside the private category `🖥️ Infraestructura`**, the same one as `alertas-criticas` and `alertas-avisos`, visible only to the `devops` role. Don't put it in the server's public channels.

### 1. The channel

1. Hover over the `🖥️ Infraestructura` category, click **+** (**Create Channel**) and choose **Text**.
1. **Channel Name:** `actualizaciones`. Since it's in the private category it inherits its permissions: confirm in *Edit Channel → Permissions* that only the `devops` role sees it and that `@everyone` doesn't.
1. Right-click the channel → **Notification Settings** → **Only @mentions** (or muted): check it when you want.

### 2. The webhook

1. Next to the channel click the gear (**Edit Channel**) → **Integrations** → **Webhooks** → **New Webhook**.
1. Open it and change the **Name** to `GitHub gitops` and verify **Channel** is `actualizaciones`.
1. Click **Copy Webhook URL** and **Save Changes**.

The URL is `https://discord.com/api/webhooks/<id numérico>/<token>`. **Save it in 1Password**:

Secure Note `discord-webhooks`, field `updates`, type Password; if the item already exists from monitoring, just add the field.

The URL is read without echo, so it doesn't stay in history. If the item already exists from monitoring, `op item edit` adds the field:

```
# pega la URL del webhook y Enter
read -rs DC_UPD; op item edit discord-webhooks --vault HomeLab "updates[password]=$DC_UPD"; unset DC_UPD
```

If the item doesn't exist yet, create it with the field: `op item create --category "Secure Note" --vault HomeLab --title discord-webhooks "updates[password]=$DC_UPD"` (before the `unset`).

It's a secret: don't paste it in Git, chat, or a ticket.

### 3. Save it as an Actions secret

The workflow reads the URL from a repo secret, so it doesn't appear in Git or in the log:

1. In GitHub, repo **`maceosoft/gitops`** → **Settings** → **Secrets and variables** → **Actions** → **New repository secret**.
1. **Name:** `DISCORD_WEBHOOK_UPDATES`. **Secret:** the webhook URL, exactly as you copied it (**without** `/github` at the end).
1. **Add secret**.

Using the GitHub CLI, reading the URL without echo:

```
# pega la URL del webhook y Enter
read -rs DC_UPD; printf '%s' "$DC_UPD" | gh secret set DISCORD_WEBHOOK_UPDATES --repo maceosoft/gitops; unset DC_UPD
```

Without this secret the workflow works the same, but it doesn't notify in Discord (the step notes it in its log and continues).

Why `updates` doesn't go through the playbook

The webhook is used **from GitHub Actions**, not from the cluster, so there's no Kubernetes Secret to create and [`apply_platform_secrets.py`](https://github.com/maceosoft/homelab/blob/main/ansible/scripts/apply_platform_secrets.py) is not touched. The `updates` field in 1Password is just the webhook backup copy: that script doesn't read or validate it, and it doesn't interfere with monitoring.

## How notifications arrive

- **Discord, `actualizaciones`:** when the daily run finishes, **a single message** with the PRs Renovate created in the last 26 hours (linked title, and the *draft*, *major*, or *manual procedure* labels when applicable), plus the total of open update PRs. If there are no new PRs, **it sends nothing**. If the run fails (e.g., due to an expired token), a red message arrives with the link to the log. A simulation (*dry run*) notification is not sent.
- **GitHub:** the *Product Updates (Renovate)* issue (*Dependency Dashboard*) lists what was detected, what's waiting, and open PRs. From there you can request a PR that Renovate hasn't opened yet, due to the simultaneous PR limit (3).

## What to do when receiving an update PR

1. **Read the PR.** It brings the version notes (*Release notes*) and the version jump. If it's Calico or Longhorn (draft, `procedimiento-manual`), go to its guide first: [Calico](https://homelab.maceo.cl/en/operacion/actualizar-calico/index.md).

1. **Check the pipeline.** It must be green, and in the `argocd-diff` job summary check which resources change ([GitOps pipeline](https://homelab.maceo.cl/en/operacion/pipeline-gitops/index.md)).

1. **Decide when.** A storage or network update isn't done under load or on a Friday afternoon.

1. **Merge**. Argo CD syncs; Calico Applications don't, those are synced manually.

   If it's a draft, click **Ready for review** first; then the PR's merge button.

   With the PR number (`gh pr list --repo maceosoft/gitops --search "head:renovate/"` lists them):

   ```
   # solo si es borrador
   gh pr ready <número> --repo maceosoft/gitops
   gh pr merge <número> --repo maceosoft/gitops --squash
   ```

1. **Verify** in Argo CD that the Application becomes `Synced` and `Healthy`, and that monitoring alerts don't fire. To roll back, revert the PR.

   In the Argo CD UI, the Application card shows the two states.

   ```
   kubectl -n argocd get applications
   ```

   The `SYNC STATUS` and `HEALTH STATUS` columns must say `Synced` and `Healthy`.

1. **If you don't want it:** close the PR without merging. Renovate won't reopen that version (it will the next one). If you close it by mistake, mark it in the *Dependency Dashboard*. With the console: `gh pr close <número> --repo maceosoft/gitops`.

Ranges

Many Applications pin a range (`0.95.x`). A Renovate PR rewrites it to the new range (`0.96.x`) only when a version outside it is released; patches within the range are already picked up by Argo CD without a PR.

## Test without creating PRs

A simulation (*dry run*): Renovate goes through everything and only writes to the log what it would do.

*Actions → renovate → Run workflow*, with **Simulation (dry run)** checked.

With the GitHub CLI (`gh workflow view renovate.yml --repo maceosoft/gitops` shows its inputs):

```
gh workflow run renovate.yml --repo maceosoft/gitops --ref main -f dry_run=true
gh run watch --repo maceosoft/gitops
```

For the log level, add `-f log_level=debug`.

With `debug` in the log level, each detected dependency is shown.

If the workflow fails at the *GitHub App Token* step, one of the two secrets is missing or wrong, or the App isn't installed in `gitops`.

## Change the configuration

The configuration is in [`renovate.json5`](https://github.com/maceosoft/gitops/blob/main/renovate.json5) (which managers it uses, what only notifies, limits, groupings) and the program in [`.github/workflows/renovate.yml`](https://github.com/maceosoft/gitops/blob/main/.github/workflows/renovate.yml). The Renovate version is pinned in the workflow: bump the version manually from time to time.

## Disable it

| What                         | How                                                                                                                                                                                                                                                     |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pause                        | `gitops` → *Actions → renovate* → **⋯** → **Disable workflow** (with the CLI: `gh workflow disable renovate.yml --repo maceosoft/gitops`)                                                                                                               |
| Remove Discord notifications | Delete the secret `DISCORD_WEBHOOK_UPDATES` in `gitops` → **Settings → Secrets and variables → Actions** (or `gh secret delete DISCORD_WEBHOOK_UPDATES --repo maceosoft/gitops`), and the webhook in Discord (*Edit Channel → Integrations → Webhooks*) |
| Remove entirely              | Delete `renovate.json5` and `.github/workflows/renovate.yml` in a PR, the two Actions secrets, the App (*maceosoft → Settings → Developer settings → GitHub Apps → Edit → Advanced → Delete GitHub App*) and the 1Password items                        |

Closing open Renovate PRs and deleting their branches `renovate/*` is manual (with the CLI: `gh pr list --repo maceosoft/gitops --search "head:renovate/" --json number --jq '.[].number'` to list them and `gh pr close <número> --delete-branch --repo maceosoft/gitops` to close each one).

## And Dependabot of `homelab`

`homelab` has Dependabot enabled on pip (PRs for `black` and `certifi`). Renovate here only watches `gitops`, so **it doesn't replace it**: they coexist without stepping on each other. If later you prefer a single system, `homelab` can be added to Renovate (install the App in that repo too and add it to `RENOVATE_REPOSITORIES`) and then remove Dependabot.

# Update Argo CD

Argo CD updates **itself**, always to the latest stable version. The Application `argocd` from [`maceosoft/gitops`](https://github.com/maceosoft/gitops) installs the official manifest from the `stable` branch of the Argo CD repo and adds its own configuration (Dex, RBAC —role-based access control—, health check, `ci` account): when a new version comes out, Argo CD applies it to itself.

| Piece                              | What it does                                                                                                                                                                                                                                                                                   |
| ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `argocd/apps/argocd.yaml`          | The Application: follows `argocd/config` with auto-sync and `ServerSideApply`                                                                                                                                                                                                                  |
| `argocd/config/kustomization.yaml` | Base: [`https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml`](https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml). On top, `configMapGenerator` with `behavior: merge` for `argocd-cm`, `argocd-rbac-cm` and `argocd-cmd-params-cm` |
| Ansible role `k3s_argocd`          | First install and major version jumps (`version: latest` in `k3s_install.yml`)                                                                                                                                                                                                                 |

Unattended update

A new version is applied without asking you, and Argo CD is what deploys everything else. It already goes through the `gitops` pipeline (the manifests of the new version are validated when your repo changes), but **not** when the Argo CD repo changes. If a new version causes issues, see [If a new version fails](#if-a-new-version-fails).

## How long until it notices

The repo-server caches generated manifests for one hour (`reposerver.default.cache.expiration: 1h` in `argocd-cmd-params-cm`), so a new version is detected at most **one hour after** release. To speed it up:

```
kubectl -n argocd annotate application argocd argocd.argoproj.io/refresh=hard --overwrite
```

## The first time: update manually, then enable the app

Order matters

An old Argo CD (2.x) **cannot diff itself** on a cluster with Kubernetes 1.36 (`.status.terminatingReplicas: field not declared in schema`), so it cannot be the one to update itself. The first update is done by the Ansible role. **Do not `git push` the `gitops` change before** (replace `argocd-config` with `argocd` in the ApplicationSet).

1. **Check the current version and the one to be installed:**

   ```
   export KUBECONFIG=~/.kube/homelab-k3s.yaml
   kubectl -n argocd get deploy argocd-server -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
   curl -s https://raw.githubusercontent.com/argoproj/argo-cd/stable/VERSION
   ```

1. **Dry-run on the server, changing nothing** (must finish without errors; *failed to migrate last-applied-configuration* warnings are normal):

   ```
   curl -sL https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml \
     | kubectl apply -n argocd --server-side --force-conflicts --dry-run=server -f -
   ```

1. **Update with Ansible** (from `ansible/` in `homelab`):

   ```
   ansible-playbook -i inventory.ini playbook-k3s.yml --tags argocd
   ```

   The role resolves `version: latest` (latest stable) and applies the official manifest with `--server-side --force-conflicts` (mandatory since Argo CD 3.3, whose `ApplicationSet` CRD exceeds the normal apply limit).

1. **Verify** everything goes back to healthy. With the new version the app diff error disappears:

   ```
   kubectl -n argocd get pods
   kubectl -n argocd get deploy argocd-server -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
   kubectl -n argocd get applications
   ```

1. **Now `git push`** `gitops`. The ApplicationSet changes `argocd-config` to `argocd`: the old Application is deleted **without** deleting its resources (Dex and RBAC ConfigMaps remain), and the new one adopts them. Check again with `kubectl -n argocd get applications`.

## Pin a version (pause updates)

To pin a version, change the base of `argocd/config/kustomization.yaml`, in `gitops`, from `stable` to a tag:

```
resources:
  - https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.3/manifests/install.yaml
```

To resume automatic updates, set it back to `stable`. In the Ansible role, the equivalent is `k3s_install.gitops_argocd.version: v3.5.3`.

## If a new version fails

1. **Pin the previous version** in `gitops` (previous section) and wait for it to apply.

1. If Argo CD won't start and can't apply anything, reinstall the version that worked directly with `kubectl` (doesn't depend on Argo CD):

   ```
   kubectl apply -n argocd --server-side --force-conflicts \
     -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.3/manifests/install.yaml
   ```

1. The Argo CD [upgrade notes](https://argo-cd.readthedocs.io/en/stable/operator-manual/upgrading/overview/) explain what changes in each minor version.

## If it fails

| Symptom                                                                                                 | Check                                                                                                                                                                                                                                          |
| ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ComparisonError … terminatingReplicas: field not declared in schema`                                   | Argo CD is before 3.5: must do the first update with Ansible (above)                                                                                                                                                                           |
| `argocd` stays `OutOfSync` after enabling                                                               | Normal for a few minutes: Argo CD adds its tracking annotation to each resource                                                                                                                                                                |
| `The CustomResourceDefinition "applicationsets.argoproj.io" is invalid: metadata.annotations: Too long` | Applied without `--server-side`; use the commands on this page                                                                                                                                                                                 |
| GitHub login lost after update                                                                          | `argocd-cm` must have the `dex.config` key (`kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.dex\.config}'`) and the Secret `argocd-secret` its `dex.*` keys ([Dex](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)) |
| New version not detected                                                                                | Wait an hour, or force: `kubectl -n argocd annotate application argocd argocd.argoproj.io/refresh=hard --overwrite`                                                                                                                            |

# Calico (CNI) managed by Argo CD

Calico is the CNI (Container Network Interface, the pod network) of the cluster. It is installed in **two stages**:

1. **Bootstrap, with Ansible** (role `k3s_cni`): when creating the cluster, Argo CD does not yet exist and needs pod network to start, so Ansible installs Calico first.
1. **From then on, Argo CD** manages and updates it: two Applications of [`maceosoft/gitops`](https://github.com/maceosoft/gitops).

| Application       | What it manages                                                              | Where                                    |
| ----------------- | ---------------------------------------------------------------------------- | ---------------------------------------- |
| `calico-operator` | CRDs and the operator (`tigera-operator`) in the chosen version              | `cni/calico-operator/kustomization.yaml` |
| `calico-config`   | `Installation` (pod pool `10.42.0.0/16`, VXLAN cross-subnet) and `APIServer` | `cni/calico/custom-resources.yaml`       |

Manual sync, no prune

The two Applications **do not** sync automatically. A Git version change updates the pod network of the 3 nodes, so it must be applied manually (Argo CD UI → **Sync**). They also do not have `prune` or finalizer: if the Application is deleted, the CNI is not touched.

## What version is running

```
kubectl get installation default -o jsonpath='{.status.calicoVersion}{"\n"}'
kubectl -n tigera-operator get deploy tigera-operator -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
```

## Adopt and update: 3.29.1 → 3.31.7 → 3.33.0 (in two stages)

Calico was installed with **v3.29.1** (operator `v1.36.2`). Version 3.33 only supports upgrading from **v3.31 or v3.32** (Calico documentation), so the direct jump from 3.29 **is not supported**. It is done in two stages, each with its own Sync and verification:

| Stage | `cni/calico-operator/kustomization.yaml` | Operator   | What it does                                                         |
| ----- | ---------------------------------------- | ---------- | -------------------------------------------------------------------- |
| 1     | `v3.31.7`                                | `v1.40.15` | Adopts existing resources and upgrades the CNI from 3.29.1 to 3.31.7 |
| 2     | `v3.33.0`                                | `v1.44.0`  | Upgrades from 3.31.7 to 3.33.0                                       |

Since version 3.31, the CRDs come in a separate file (`operator-crds.yaml`), which already includes the kustomization.

The two Applications carry `argocd.argoproj.io/compare-options: ServerSideDiff=true`: without this, Argo CD shows `OutOfSync` phantom in the Calico CRDs and in the fields that the operator completes (such as `assignmentMode` in the pool, new in v3.31), even if `kubectl diff --server-side` comes up empty.

It is the pod network of the three nodes

If a stage fails, the pod network is compromised and **there is no simple rollback**. Do it with maintenance window and direct access to the nodes. It is the same error that broke Longhorn when jumping from 1.6 to 1.13: never skip more versions than the project documentation supports.

**Before:**

```
# calico y apiserver: AVAILABLE=True
kubectl get tigerastatus
# los 3 Ready
kubectl get nodes
# sin pods raros
kubectl get pods -A --no-headers | awk '$4!="Running" && $4!="Completed"'
```

**Preview of what would change** (read-only, against the server):

```
kustomize build gitops/cni/calico-operator | kubectl diff --server-side --force-conflicts -f - | grep -cE '^[+-] '
kubectl diff --server-side --force-conflicts -f gitops/cni/calico/custom-resources.yaml
```

The second one should come **empty**: the YAML of `calico-config` declares all the fields that the operator writes, so that Argo CD and the operator do not step on each other.

**Sync, in order:**

1. Activate the two Applications: they are already in `bootstrap/root-appset.yaml`; apply the ApplicationSet (`kubectl apply -f gitops/bootstrap/root-appset.yaml`).

1. **Sync** of `calico-operator` (with *Server-Side Apply*, already active). The operator updates and restarts the `calico-node` **node by node** (`maxUnavailable: 1`).

   In Argo CD, **Sync** of `calico-operator`.

   With the Argo CD CLI (the Application already brings `ServerSideApply=true`, so there is no need to pass it):

   ```
   argocd login argocd.homelab.local --grpc-web
   argocd app sync calico-operator
   ```

1. Monitor: `kubectl -n calico-system get pods -w` and `kubectl get tigerastatus`.

1. When everything is `Available`, **Sync** of `calico-config`.

   In Argo CD, **Sync** of `calico-config`.

   ```
   argocd app sync calico-config
   ```

1. Verify stage 1 (next block) and **repeat steps 2–4 for stage 2**, after changing the URLs to `v3.33.0` and pushing.

**After:**

```
# v3.31.7 tras la etapa 1; v3.33.0 tras la 2
kubectl get installation default -o jsonpath='{.status.calicoVersion}{"\n"}'
kubectl get tigerastatus
kubectl -n argocd run curl-test --rm -i --restart=Never --image=curlimages/curl --quiet -- \
  curl -s -o /dev/null -w "%{http_code}\n" http://kong-gateway-proxy.kong.svc:80/
```

The last one checks that a pod still reaches a service on another node.

## Post-upgrade changes

- **Upgrade:** change the two URLs of `cni/calico-operator/kustomization.yaml`, push and **Sync** manually. Upgrade one or two minor versions at a time and review the release notes.
- **Do not change** `cidr`, `encapsulation` or `blockSize` of the pool without a maintenance window: they change the network of all pods.
- **Do not run** `--tags cni` of Ansible on an already adopted cluster: the role detects that Argo CD follows the `Installation` and skips Calico, to avoid conflicts with it.

## If it fails

| Symptom                                         | What to check                                                                                                                   |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `calico-operator` is `OutOfSync` after the Sync | It is normal for a moment due to the fields the operator adds; if it persists, `kubectl diff --server-side` to see what differs |
| `tigerastatus calico` in `Degraded`             | `kubectl -n tigera-operator logs deploy/tigera-operator --tail=50`                                                              |
| New pods without network                        | `kubectl -n calico-system get pods`; `calico-node` should be `Running` on the 3 nodes                                           |
| Argo CD cannot sync (because the CNI fell)      | Fix the CNI with `kubectl` directly from the station, not through Argo CD                                                       |

# Update Debian on deborah (12 → 13)

`deborah` (Orange Pi 5 Plus, ARM64 —64-bit ARM architecture—) runs **Debian 12 (bookworm)**; the other two nodes are already on **Debian 13 (trixie)**. The playbook [`playbook-upgrade-debian13.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-upgrade-debian13.yml) updates it with backup, a prior simulation, and a **boot gate** that validates before rebooting.

It's the most delicate operation in the Homelab

`deborah` is the **K3s control plane**, the **LAN DNS** (BIND) and a member of **Incus**. During this process, the Kubernetes API and DNS do not respond (the pods on the other nodes continue running). If the machine does not boot after the reboot, physical access (HDMI —video output— or serial console) is required. Do this during maintenance windows.

The Kubernetes API goes down while `deborah` reboots

`deborah` is the control plane: while it reboots, `kubectl` and Argo CD don't respond, but the pods already running on the other nodes keep working. If the control plane were ever on another node ([Move the control plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md)), only `deborah`'s share of the load would go down (LAN DNS, Ollama and its pods).

## What Changes and What Doesn't

| Changes                                                                                        | Doesn't Change                                                                                                                              |
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| Debian packages move to trixie (around 600 are updated and about 40 old libraries are removed) | **The manufacturer's kernel** (`5.10.160-rockchip-rk3588`) and the boot with u-boot: they don't come from Debian and apt doesn't touch them |
| `systemd`, `glibc`, `openssh`, `bind9`… to their trixie versions                               | K3s (its binary is independent), Incus (keeps the same version), and the pods                                                               |
| `/tmp` becomes memory (`tmpfs`) after the reboot                                               | `docker-ce` and `containerd.io` (not from Debian; remain as they are)                                                                       |

Kernel 5.10 meets the minimum recommended by `systemd` 257 (5.4), but this is an unusual case: Debian documents the update for "pure" systems, not for manufacturer images. That's why the boot gate exists.

## Before You Start

- **Incus aligned on all three nodes.** The cluster requires the same version on all members. If `deborah` has a different Incus version than the repository offers, update Incus on all three nodes first with [`playbook-upgrade-incus.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-upgrade-incus.yml) (`-e incus_upgrade_confirm=true`; see [Update or Uninstall Incus](https://homelab.maceo.cl/en/incus/cluster-setup/#update-or-uninstall-incus)). The playbook stops if they don't match (`upgrade_skip_incus_check=true` skips it, only for testing).
- **Physical access** to `deborah` (HDMI or serial console) and, if possible, an **eMMC image** (`dd` from another machine). This is the only complete rollback.
- Healthy K3s: `kubectl get nodes` with the three `Ready`, and `incus cluster list` with the three `ONLINE`.
- No one will need the DNS or the Kubernetes API for ~30–60 minutes.

## Step 1 — Preflight and Simulation (Nothing Changes)

```
cd ansible
export KUBECONFIG=~/.kube/homelab-k3s.yaml
ansible-playbook -i inventory.ini playbook-upgrade-debian13.yml
```

Without `-e upgrade_confirm=true` it only runs `preflight`: checks that it's Debian 12 on ARM64, free space (6 GB in `/`, 300 MB in `/boot`), that `dpkg` doesn't have issues, **boot** (u-boot, `uInitrd` in gzip, UUID —unique identifier— of the root), and the Incus version. Then it downloads the trixie indexes to a separate directory (`/var/tmp`), **simulates** the update, and **fails if it would remove the kernel, boot, or a Homelab service** (`linux-image`, `orangepi`, `k3s`, `incus`, `bind9`, `openssh-server`, `netplan`…). Then it deletes that directory.

## Step 2 — Update

```
ansible-playbook -i inventory.ini playbook-upgrade-debian13.yml -e upgrade_confirm=true
```

| Phase (tag) | What It Does                                                                                                                                                                                                      |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `backup`    | Backs up `/etc`, `/boot`, the state of `dpkg`/`apt`, BIND, and K3s (SQLite, certificates, and token; **stops K3s** before). Brings the files to `ansible/backups/` (ignored by Git)                               |
| `upgrade`   | Pauses automatic updates, keeps Debian 12 up to date, changes the repositories to trixie (Debian and Zabbly), and does `upgrade` and `full-upgrade` while preserving your configuration files (`--force-confold`) |
| `gate`      | **Before rebooting:** regenerates and validates the `initrd`, checks `uInitrd`, `boot.scr`, the DTB, kernel modules, the UUID of the root, `sshd -t`, `netplan generate`, and the IP of `br0`                     |
| `reboot`    | Reboots and waits up to 20 minutes                                                                                                                                                                                |
| `verify`    | Confirms Debian 13 with the same kernel, K3s `Ready`, services, DNS, and Incus                                                                                                                                    |
| `cleanup`   | Reactivates apt timers and **lists** obsolete packages (doesn't remove them)                                                                                                                                      |

If something fails **before** the reboot, the playbook restarts K3s, leaves the machine powered on, and tells you where the logs (`/var/log/pre-trixie-upgrade-*.log`) and backup are. To test without rebooting: `-e upgrade_reboot=false`; the reboot and verification you do manually (`--tags reboot,verify`).

### Why the Boot Gate Exists

The board's boot depends on a u-boot script (`boot.scr`) and a `uInitrd` that the hook `/etc/initramfs/post-update.d/99-uboot` creates by declaring it **gzip**. If the update left the `initrd` compressed differently, u-boot couldn't decompress it, and the machine **wouldn't boot**. The playbook requires `COMPRESS=gzip` in `initramfs.conf` (it's preserved with `--force-confold`) and validates the result with `gzip -t` and `mkimage -l` before rebooting.

## If the Machine Doesn't Boot

1. Connect a screen or serial console and see where it stops (u-boot, kernel, or `systemd`).
1. If it's the boot: restore the eMMC image you made earlier. With the image, `deborah` returns to Debian 12.
1. Without an image: reinstall the board's system and restore the backup of `ansible/backups/deborah-<fecha>/`: `etc.tgz`, `boot.tgz`, `k3s.tgz` (certificates, token, and control plane database, while `deborah` is the control plane) and `bind.tgz`. Without the K3s token and certificates, you'd have to reinstall the cluster.
1. The other two nodes continue working, but without a control plane, there are no changes or synchronization.

## Next Steps

- `ansible-playbook -i inventory.ini playbook-upgrade-debian13.yml --tags verify` repeat the verification when you want to.
- **Do not blindly purge** the list of obsolete packages: it includes `linux-image-legacy-rockchip-rk3588`, `orangepi-*` and `linux-u-boot-*`, which are what make the board boot.
- With `deborah` in Debian 13, the versions of OVN and Open vSwitch now match those of the other nodes (a prerequisite for considering OVN: see [Load Balancing and OVN](https://homelab.maceo.cl/en/incus/cluster-setup/#load-balancing-and-ovn)).
- **Two units may end up in `failed` after a reboot** (`systemctl --failed`), without affecting the cluster: `smartmontools` (the eMMC does not have SMART and in Debian 13 the service is enabled by default) and `dnsmasq` (port 53 is already occupied by BIND, which is the LAN DNS). If you do not use those units, disable them with `systemctl disable --now dnsmasq smartmontools`.
- The package `wiringpi` is held back (`apt-mark hold`) and the `docker-ce` / `containerd.io` are not from Debian: they remain as they were; review them if they cause issues.

## Variables

Passed with `-e nombre=valor` (never with dots).

| Variable                   | Default                                      | Purpose                                                 |
| -------------------------- | -------------------------------------------- | ------------------------------------------------------- |
| `upgrade_confirm`          | `false`                                      | `true` runs for real; `false` only preflight            |
| `upgrade_reboot`           | `true`                                       | `false` stops before reboot                             |
| `upgrade_autoremove`       | `false`                                      | `true` runs `apt autoremove` at the end                 |
| `upgrade_skip_incus_check` | `false`                                      | `true` skips the Incus version check (for testing only) |
| `upgrade_debian_mirror`    | `http://ftp.cl.debian.org/debian`            | Debian mirror                                           |
| `upgrade_security_mirror`  | `http://security.debian.org/debian-security` | Security repository                                     |

# Upgrade K3s

K3s (lightweight Kubernetes distribution) upgrades **itself**: the SUC (System Upgrade Controller) watches the K3s `stable` channel and, when a new version comes out, upgrades one node at a time (first the control plane, the node with the `node-role.kubernetes.io/control-plane` label: `deborah`; then the workers), draining it first (*cordon* and *drain*). This procedure is for checking that this works, pinning a version when needed and keeping Ansible aligned.

The plans live in [`maceosoft/gitops` → `k3s-upgrade/`](https://github.com/maceosoft/gitops/tree/main/k3s-upgrade) and are applied by the `k3s-upgrade` Argo CD Application.

## 1. Check that the SUC is installed

The `k3s-upgrade` Application only applies the *plans*. The controller that runs them is installed by hand once ([K3s — Manual installation, 6.1](https://homelab.maceo.cl/en/k3s/index.md)):

```
kubectl -n system-upgrade get deploy system-upgrade-controller
kubectl -n system-upgrade get plans
```

The controller and the `k3s-server` and `k3s-agent` plans must show up.

## 2. Check the upgrade status

```
# columna VERSION de cada nodo
kubectl get nodes -o wide
# versión objetivo (LATESTVERSION)
kubectl -n system-upgrade get plans -o wide
# un job por nodo actualizado
kubectl -n system-upgrade get jobs
```

All nodes should end up with the same `VERSION`.

## 3. Pin a version (optional)

To stop following `stable` (for example, to wait before a minor version upgrade), replace `channel` with `version` in **both** plans of `k3s-upgrade/k3s-plans.yaml`, in the `gitops` repo:

```
spec:
  # channel: https://update.k3s.io/v1-release/channels/stable
  version: v1.36.5+k3s1
```

Merge the change: Argo CD applies it and the SUC keeps the nodes on that version.

## 4. Align Ansible with the actual version

Ansible installs **new** nodes with `k3s_install.core.version` (`group_vars/incus_cluster/k3s_install.yml` in `homelab`). The SUC does not touch that file, so after every upgrade it falls behind. Update it to the version shown by `kubectl get nodes`:

```
k3s_install:
  core:
    version: "v1.37.0+k3s1"   # la VERSION actual de los nodos
```

This is essential before [adding a node](https://homelab.maceo.cl/en/operacion/agregar-nodo/index.md): otherwise the new node joins with an older version.

## If it fails

| Symptom                                     | Check                                                                                                                                    |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `get plans` shows nothing                   | The `k3s-upgrade` Application does not sync: the controller or its CRDs are missing (step 1)                                             |
| A node stays `SchedulingDisabled`           | The upgrade stopped halfway: check the job with `kubectl -n system-upgrade logs job/<job>` and, once resolved, `kubectl uncordon <nodo>` |
| Nodes on different versions for a long time | The `drain` does not finish (PodDisruptionBudget or pods without a controller); check the jobs from step 2                               |

# Add a disk to a node

Format a free disk on a node, mount it, and deliver it to **Longhorn** (replicated storage for Kubernetes) or to **Incus** (a new pool for containers and VMs).

Destructive

The playbook **formats** the disk: it deletes any data on it. Confirm the exact device before running it. The playbook has a safety pause and does not reformat a disk that already has a file system.

In the examples: disk `/dev/nvme0n1` on the node **`deborah`**.

## 1. Identify the disk

Run the node discovery and read the *Storage Recommendations* section of `reports/<nodo>.md`:

```
cd ansible
ansible-playbook -i inventory.ini playbook-discovery.yml --limit deborah
```

Confirm it also on the node (`lsblk -f`): the disk must be empty and unmounted.

## 2. Choose who to deliver to

| Consumer   | When                                                 | What the playbook does                                                         |
| ---------- | ---------------------------------------------------- | ------------------------------------------------------------------------------ |
| `longhorn` | For Kubernetes volumes with node-to-node replication | Mounts the disk and marks the node so Longhorn can use it                      |
| `incus`    | For more space for Incus containers and VMs          | Mounts the disk and creates a new storage pool, separate from the pool `local` |

## 3. Declare it in `host_vars`

In `host_vars/deborah.yml`:

```
storage_provision_disks:
  - device: /dev/nvme0n1
    consumer: longhorn            # o incus
    mount_path: /var/lib/longhorn-nvme0n1
    filesystem: ext4
    # solo si consumer: incus
    # incus_pool_name: nvme-pool
```

Without this variable, the playbook does nothing on that node.

## 4. Apply

```
ansible-playbook -i inventory.ini playbook-storage-provision.yml --limit deborah
```

Confirm in the safety pause when the device shown is correct.

## Verify

The `ssh` logs in with [single sign-on](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md) (`opkssh login` before):

```
# montado, con la entrada en /etc/fstab
ssh devops@deborah.homelab.local findmnt /var/lib/longhorn-nvme0n1
```

- **Longhorn:**

  In the UI, *Node → deborah* shows the new disk with *Schedulable* enabled.

  The disks on the node are in the resource `nodes.longhorn.io`; the new disk should appear with the condition `Schedulable` in `True`:

  ```
  kubectl -n longhorn-system get nodes.longhorn.io deborah -o jsonpath='{.status.diskStatus}'
  ```

- **Incus:** `sudo incus storage list` shows the new pool.

## If it fails

| Symptom                                      | Check                                                                                             |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| The playbook does nothing                    | Missing `storage_provision_disks` in `host_vars` of the node, or `--limit` points to another node |
| "Already has filesystem" and does not format | By design: the disk is not empty. Check what it contains before manually deleting it              |
| Longhorn does not show the disk              | Wait a minute and check the node annotation: `kubectl get node deborah -o yaml \| grep longhorn`  |

More details about the role: [`storage_provision`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/storage_provision/README.md).

# Add a node to the Ollama cluster

How to add a machine (a Linux server, a Mac, another box) so it runs language models for the [local AI assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md). No central configuration changes needed: the new machine **announces itself** and the router starts sending it the models it has.

## How it works (in 2 minutes)

An **Ollama node** is any machine running Ollama (the program that runs the model) and announcing itself to the cluster. Clients (Open WebUI, this documentation's translation, alert summarization, K8sGPT) don't talk to nodes: they talk to a single **router**.

```
flowchart LR
    C["Clientes<br/>Open WebUI, traducción,<br/>alertas, K8sGPT"] -->|"Bearer token"| R["ollama-router<br/>ollama.homelab.local"]
    R -->|"elige según el modelo"| N1["deborah"]
    R --> N2["invincible"]
    R --> N3["oliver"]
    R -.->|"solo si está encendido"| M["Mac"]
    N1 -. "latido cada 60 s" .-> R
    N2 -. "latido" .-> R
    N3 -. "latido" .-> R
    M -. "latido" .-> R
```

- **Automatic registration.** Each node runs an agent (`ollama_node.py register`) that every 60 seconds sends the router a record with its name, address, OS, memory, priority, tags, and **the list of models it has downloaded**. The agent reads this from Ollama's model folders without waking it, so announcing costs no memory.
- **Expiry.** If the router doesn't hear from a node for 3 heartbeats (180 seconds), it marks it down and stops using it. When the node comes back, it reappears automatically. The router keeps the registry in memory: if it restarts, nodes re-register in under a minute.
- **Node selection.** The router looks at the `model` field of each request and picks among live nodes that have that model: first the one that used it last (to avoid reloading), then the highest priority, then the least busy. A model that only exists on the Mac goes to the Mac; common models prefer nodes that are always available.
- **No polling.** The router never "asks" a node if it's alive: it doesn't wake anyone up. If a node fails to connect, it avoids it for 30 seconds and tries another.

Why a custom router

No existing project combined self-announcement registration, zero polls that wake machines, the native Ollama API, and model-aware selection. The router is a small standard Python program; it's described in [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md).

## Registry rules

- **Who can register.** The router only accepts nodes whose IP is inside `ALLOWED_NODE_CIDRS` (`192.168.20.0/22`). If your node (e.g., the Mac) is **outside** that range, you must add its `/32` to that variable in the router Deployment and to `host_firewall_ollama_extra_cidrs`.
- **A live node isn't overwritten.** If an already-registered node tries to announce from **another IP**, the router replies `409` until it expires (180 seconds) or `DELETE` is done. If a node's IP changes, run `deregister` (`ollama_node.py deregister`, or `ollama-node off` on the Mac) **before** moving it.
- **Tokens.** By default all nodes use the shared `registry_token`. Optionally, the Secret `ai/ollama-router-secrets` can carry a `registry_token-<nombre-del-nodo>` key: if it exists, **only** that token can register (or delete) that node; if not, the shared one works. The router reloads tokens every 30 seconds. See [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md).
- **Limits.** Up to 64 nodes, 2000 models per node, priority 0–100, and model names in `^[A-Za-z0-9._:/@-]{1,128}$` format.

## What any node needs

| Requirement                          | Detail                                                                                      |
| ------------------------------------ | ------------------------------------------------------------------------------------------- |
| Ollama installed                     | On Linux the Ansible role installs it; on Mac, `brew install ollama`                        |
| Listen on its own IP                 | Never on `0.0.0.0` if there's another option: Ollama's API has no built-in auth             |
| Reach the router                     | Via HTTPS on `https://ollama.homelab.local` (the Kong ingress), trusting the HomeLab CA     |
| The registration token               | A token shared by nodes, stored in 1Password (item `ollama-router`, field `registry_token`) |
| K3s nodes can reach its port `11434` | The router runs in the cluster and is the only thing that talks to the node                 |

## Option A — A Linux server with Ansible (3 steps)

Works for `deborah`, `invincible`, `oliver` or any new Debian already in the Ansible inventory (if it's not yet part of the HomeLab, first [add it as a node](https://homelab.maceo.cl/en/operacion/agregar-nodo/index.md)).

1. **Inventory.** In [`ansible/inventory.ini`](https://github.com/maceosoft/homelab/blob/main/ansible/inventory.ini) add the machine name to the `[llm_nodes]` group.

1. **Node variables.** Create `ansible/host_vars/<nodo>.yml` with the models it should have, its priority and its tags. Variable names are in the [role README](https://github.com/maceosoft/homelab/blob/main/ansible/roles/ollama_ondemand/README.md); the main ones are:

   ```
   ollama_ondemand_models:        # modelos que se descargan en este nodo
     - qwen2.5:7b-instruct
   ollama_ondemand_memory_max: 3500M   # tope de memoria de Ollama en este nodo
   ollama_node_priority: 40       # más alto = se prefiere
   ollama_node_tags: [x86, always-on]  # etiquetas libres, solo informativas
   ```

1. **Apply.** Prerequisites: 1Password unlocked (with app integration enabled), 1Password SDK installed (`pip install onepassword-sdk`) and `KUBECONFIG` exported toward the cluster. From `ansible/`:

   ```
   # ensayo
   ansible-playbook -i inventory.ini playbook-ollama.yml -l <nodo> --check --diff
   ansible-playbook -i inventory.ini playbook-ollama.yml -l <nodo>
   ```

The playbook applies two roles. `ollama_ondemand` downloads the Ollama version matching the machine's architecture (ARM64 or x86_64, with its checksum) and creates the socket that wakes it only when a request arrives, listening on the node's IP. `ollama_node` installs the registration agent as a systemd service (`ollama-node-register`). The registry token is read from 1Password at apply time (`ansible/scripts/read_ollama_registry_token.py`) and ends up in `/etc/ollama-node/registry-token`; it never goes to Git. To reapply only the agent: `--tags register`. A couple of minutes later the node should appear in the router (see [Verify](#9-verificar-que-el-nodo-quedo-registrado)).

The first run takes a while

It downloads Ollama (1+ GB) and the models (several GB). To re-download only models: `-l <nodo> --tags models`.

## Option B — A Mac, step by step

The Mac is the "heavy" node: its unified memory allows large models that don't fit on the other machines. It is turned on and off **manually** with a switch, and while it's off the cluster doesn't know it exists.

How it works on macOS

`launchd` (the macOS service manager) is used to listen on the port and start Ollama only when a real request arrives. macOS prompts (Local Network permission, firewall) depend on the system version; menu texts may vary slightly.

### 1. Install Ollama with Homebrew

```
brew install ollama
```

Use the Homebrew **formula** and **not** the Ollama desktop app: the app starts automatically on login and would occupy memory all the time. If you already have it installed, close it and remove it from *System Settings → General → Login Items*.

Now download the large models you want to keep on the Mac:

```
# por ejemplo, uno de 30B o más
ollama pull <modelo>
ollama list
```

Models live in `~/.ollama/models`. If you prefer an external drive, define `OLLAMA_MODELS` pointing to that folder and pass it to the installer with `--models-dir`.

### 2. Have the HomeLab token and CA handy

- **Registration token:** the `registry_token` field of the `ollama-router` item in 1Password (the same one the other nodes use; if it doesn't exist yet, generate it with `openssl rand -hex 32`, see [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md)).

- **HomeLab CA (certificate authority)**, so the Mac trusts the registry's HTTPS. Export it from a machine with `kubectl`:

  ```
  kubectl -n cert-manager get secret homelab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > homelab-ca.crt
  ```

- **A static IP for the Mac** (see step 6).

### 3. Install the node bundle

The bundle is in [`ollama-node/`](https://github.com/maceosoft/homelab/tree/main/ollama-node) of the `homelab` repo. From that folder (with your Mac's IP instead of `192.168.20.X`):

```
./macos/install.sh --registry-url https://ollama.homelab.local --ca-file ./homelab-ca.crt \
  --url http://192.168.20.X:11434 --allowed-cidrs 192.168.20.5/32,192.168.20.6/32,192.168.20.7/32
```

It does not use `sudo`. The installer prompts for the token without echoing it on screen (or read it from a file with `--token-file`, or store it in the macOS Keychain with `--keychain`). It does three things:

1. Copies the agent to `~/Library/Application Support/ollama-node/` and writes `config.json` there.
1. Installs the LaunchAgents (your user's `launchd` tasks) in `~/Library/LaunchAgents`, **without loading them**.
1. Leaves the `ollama-node` command (in `~/.local/bin`; make sure it's in your `PATH`).

With the 1Password CLI, the token is downloaded to a temp file only you can read and passed with `--token-file` (if you have more than one 1Password account, export `OP_ACCOUNT` first); complete example:

```
umask 077
op read "op://HomeLab/ollama-router/registry_token" > "$HOME/ollama-registry-token"
./macos/install.sh --registry-url https://ollama.homelab.local --ca-file ./homelab-ca.crt \
  --url http://192.168.20.X:11434 --allowed-cidrs 192.168.20.5/32,192.168.20.6/32,192.168.20.7/32 \
  --token-file "$HOME/ollama-registry-token"
rm -P "$HOME/ollama-registry-token"
```

Useful options: `--name` (name in the registry, defaults to the Mac's short name), `--priority` (defaults to 10, so always-on servers go first), `--tags` (defaults to `mac,gpu,large`), `--models-dir` (models folder, if using an external drive) and `--uninstall`, which downloads the agents and deletes the plists and the `ollama-node` command, but **keeps** the agent folder (with `config.json` and the token) and the logs.

The `config.json` stores the name, node URL, registry URL, priority, labels and `allowed_cidrs`, the list of networks allowed to use the proxy: **only put the K3s node IPs** (`192.168.20.5`, `.6` and `.7`), because they are the ones running the router. The token **does not** go inside the `config.json`: it stays in the `token` file in the agent folder (permissions `600`) or, with `--keychain`, in the macOS Keychain.

The registry token is only for registration

With it you can announce a node to the router, nothing more. It does **not** grant inference access: clients use other tokens and the node itself is protected by the router and the firewall (see [Security](#security)). Even so, don't paste it in chats or Git. If you use a token file, it must have permissions `600`.

### 4. Trust the HomeLab CA (if needed)

The registry travels over HTTPS and the certificate is signed by the HomeLab CA (`homelab-ca`). The agent already uses it (`--ca-file`); so that `curl` and the browser also trust it, install it in the keychain (once):

```
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain homelab-ca.crt
```

Verify with:

```
curl -s -o /dev/null -w '%{http_code}\n' https://ollama.homelab.local/api/version
```

No certificate error, and with `401` (means it reached the router and asks for token).

### 5. Firewall and macOS "Local Network" permission

- **macOS Firewall.** If active (*Settings → Network → Firewall*), allow incoming connections for the `ollama` binary and for the Python that runs the agent. The first time, macOS asks only once.
- **Local Network.** macOS (Sequoia and later) asks permission for apps to talk to devices on your network. Accept the prompt when it appears, or enable it in *Settings → Privacy & Security → Local Network* for Terminal (or the program you run `ollama-node` from). This permission is granted **only from Settings**: macOS has no command to grant it.
- **Allowed subnets.** The `allowed_cidrs` list in the `config.json` makes the agent reject connections that don't come from the K3s nodes, even if the macOS firewall is open.

### 6. Reserve the Mac's IP

Also, the Mac must resolve `ollama.homelab.local` using the DNS (BIND) of `deborah`, and the router only has outbound permission toward `192.168.20.0/22` on port `11434`: the Mac must be inside that network.

The router and nodes need to find the Mac always at the same address. In your home router, create a **DHCP reservation** (*Dynamic Host Configuration Protocol*, the router service that hands out IPs; this way the Mac gets the same IP always, based on the Mac's MAC address) outside the assignable range, or configure a static IP on the Mac itself. Put that IP in the installer's `--url` option (or in the `url` of `config.json`). If the Mac uses WiFi and Ethernet, reserve the one for the interface you'll use for this.

### 7. Power and sleep

While the Mac sleeps, it stops beating: after 180 seconds the router considers it absent, and it reappears when it wakes (as long as the switch remains in `on`). For long jobs:

- Plug it into power.
- While there is activity, the agent keeps the Mac awake with `caffeinate` and releases it when done.
- If you want it to never sleep while plugged in, check `pmset -g` and adjust, for example, `sudo pmset -c sleep 0`. Optional: sleeping is fine, it just means the cluster won't use it.

### 8. The switch: `ollama-node on|off|status|logs`

| Command                               | What it does                                                                                                                                                                                  |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ollama-node on`                      | Loads the two LaunchAgents: the **socket** one (`launchd` listens on port `11434`) and the **registration** one (heartbeat every 60 seconds). From this moment the Mac appears in the cluster |
| `ollama-node off`                     | Removes both and sends a `DELETE` to the registry: the Mac leaves the list instantly and stops listening                                                                                      |
| `ollama-node status`                  | Tells whether it's on, whether Ollama is active, and whether the registry sees it                                                                                                             |
| `ollama-node logs [socket\|register]` | Shows the agent logs (both, or just the socket or just the registration)                                                                                                                      |
| `ollama-node config [edit]`           | Shows `config.json`; with `edit` opens it in your editor                                                                                                                                      |

Right after installing, the state is `off`: no agents loaded, the Mac isn't listening and isn't registered. With `on`, `launchd` opens the port **but doesn't start anything heavy**: only when a real request arrives does it launch the proxy, which starts `ollama serve` on `127.0.0.1:11435`, waits for it to respond, and forwards the traffic. While there is activity it maintains `caffeinate`. After **5 minutes without traffic** it stops Ollama and exits, and unified memory becomes free again. The registration agent keeps beating (with the model list) until you turn it off with `off`.

### 9. Verify the node was registered

With the switch in `on`, from any machine on the LAN (local network) that can reach the Mac. First load the tokens into your terminal variables, without displaying them (with 1Password unlocked; if you have more than one account, export `OP_ACCOUNT` first):

```
REGISTRY_TOKEN=$(op read "op://HomeLab/ollama-router/registry_token")
TOKEN=$(op read "op://HomeLab/ollama-router/reports")
```

And now the tests:

```
# 1. El Mac responde y carga Ollama bajo demanda (la primera vez tarda unos segundos):
curl http://<ip-del-mac>:11434/api/version
```

If you're not a K3s node, your machine isn't on the allowlist and this may be rejected: that's expected (see [Security](#security)). Try from a K3s node, or skip to the next step.

```
# 2. El registro lo ve (el listado exige el token de REGISTRO, no el de un cliente):
curl -H "Authorization: Bearer $REGISTRY_TOKEN" https://ollama.homelab.local/_registry/nodes

# 3. Los modelos que ve el router (token de un cliente cualquiera, p. ej. el de `reports`):
curl -H "Authorization: Bearer $TOKEN" https://ollama.homelab.local/api/tags

# 4. Una conversación real, que debe llegar al Mac si el modelo solo está allí:
curl -H "Authorization: Bearer $TOKEN" https://ollama.homelab.local/api/chat \
  -d '{"model":"<modelo-solo-del-mac>","messages":[{"role":"user","content":"hola"}],"stream":false}'

# 5. Apagarlo y comprobar que sale del listado:
ollama-node off
curl -H "Authorization: Bearer $REGISTRY_TOKEN" https://ollama.homelab.local/_registry/nodes
```

### Remove the Mac entirely

```
ollama-node off
```

and, if you no longer want it as a node, run `./macos/install.sh --uninstall`, which unloads the agents and deletes the plists, the command, and the agent folder. Since the registration token is shared, there's no need to delete it; if you're concerned about it remaining on the Mac, rotate it ([Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md)). The `~/.ollama` models stay on your disk; delete them yourself if you need the space.

## Option C — Another machine (another Linux without Ansible, Windows, etc.)

The contract is the same and doesn't depend on the system: the only thing the router needs is for someone to send it this notice every 60 seconds.

1. Install Ollama and make it listen only on the machine's IP.
1. Run the portable agent, which is a single standard Python program: [`ollama_node.py`](https://github.com/maceosoft/homelab/tree/main/ollama-node) with the command `register --loop` (periodic announcement; `deregister` to exit) and, if you want on-demand startup, `proxy` (designed for macOS).
1. Or do it yourself: a `PUT /_registry/nodes/<nombre>` with the header `Authorization: Bearer <token del registro>` and a JSON body with `name`, `url`, `os`, `arch`, `ram_gb`, `priority`, `tags`, `ollama_version`, `ts`, and the list `models`. To exit, `DELETE` on the same path.

## Security

- **The registry token is only for registration.** It doesn't grant access to inference. If it leaks, rotate it (and update it on every node).
- **Inference is protected by the router.** Every client needs its own Bearer token, and the router blocks verbs that manage models (`pull`, `create`, `delete`, `copy`, `push`): models are installed on each node, not over the network.
- **The firewall protects the node.** A node listens only on its IP and accepts traffic only from K3s nodes (where the router runs). No client talks directly to a node.
- **Documented exception.** The Ollama API has no built-in authentication; that's why its control is the network. It's in the [exception table](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/#network-controlled-exceptions).

## Troubleshooting

| Symptom                                      | What to check                                                                                                                                                                     |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node does not appear in `/_registry/nodes`   | Incorrect token (401 in agent logs), node cannot reach router (DNS, firewall, CA), or registration service not running (`systemctl status` on Linux, `ollama-node status` on Mac) |
| Appears and disappears                       | Network or Mac goes to sleep: heartbeat lost for more than 180 seconds. Check power or WiFi                                                                                       |
| Appears, but router does not send it a model | Model not in its list (`ollama list` on node) or registry has stale list: wait for a heartbeat                                                                                    |
| `404` with a model suggestion                | No live node has that model; `GET /api/tags` shows available ones                                                                                                                 |
| `401` from a client                          | Incorrect, expired, or unauthorized client token                                                                                                                                  |
| `409` on registration                        | Node already registered with another IP: run `deregister` or wait 180 seconds                                                                                                     |
| `400` «IP outside ALLOWED_NODE_CIDRS»        | Node IP not in router's allowed range (see [Registration Rules](#registry-rules))                                                                                                 |
| `403` in `/api/pull`                         | Expected: models are managed on each node                                                                                                                                         |
| Certificate error from Mac                   | Missing HomeLab CA install (step 4)                                                                                                                                               |
| Mac does not respond on `11434`              | Toggle in `off`, macOS firewall, missing Local Network permission, or IP different from reserved one                                                                              |
| First response takes a long time             | Cold start: Ollama starts up and loads the model. Expected once                                                                                                                   |
| Mac runs out of memory                       | Model too large for free memory; try a smaller one or close applications                                                                                                          |

For logs: on Linux, `journalctl -u ollama-node-register`; on Mac, `ollama-node logs`; in cluster, router logs in [Centralized Logs](https://homelab.maceo.cl/en/operacion/logs/index.md).

See also [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md) and [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md).

# Add a Node

Add a new machine to the HomeLab as a member of the Incus cluster and as a **K3s worker (lightweight Kubernetes distribution)**. It's the same sequence as the initial setup ([Stages 0 to 3](https://homelab.maceo.cl/en/implementacion/index.md)), but limited to a single node with `--limit`, without touching the ones that are already running.

Before you start

- Debian installed on the new machine, with your admin user created during installation (see [which user gets `sudo`](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/#day-0-freshly-installed-nodes-only-the-first-time-per-node)), and connected to the LAN (local network).
- A free IP address in `192.168.20.0/22` for the node.
- The current cluster running: Incus and K3s with the 3 healthy nodes.
- A name for the node (the current ones follow the series *Invincible*: `invincible`, `oliver`, `deborah`). In the examples: **`nolan`**.

All commands are run from `ansible/`.

## Day 0: python3, sudo, and your SSH key

A freshly installed Debian doesn't have what Ansible needs. Add the current IP address of the node (the one assigned by DHCP) to `bootstrap/inventory.ini`, in the group `servidores_debian`, and run:

```
ansible-playbook -i bootstrap/inventory.ini bootstrap/setup_sudo.yml --ask-pass --limit <ip-actual>
ssh-copy-id <usuario-admin>@<ip-actual>
```

Details: [Stage 0 — Day 0](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/#day-0-freshly-installed-nodes-only-the-first-time-per-node).

## Node Inventory and Variables

In `inventory.ini`, add the node to its architecture group (`x86_nodes` or `arm64_nodes`) and to the group `k3s_agent`:

```
[x86_nodes]
# ... nodos existentes ...
nolan ansible_host=<ip-actual> static_ip=192.168.20.8 ip_static=false bridge_iface=eno1 ram_gb=16 cpu_model="..." k3s_role=agent

[k3s_agent]
invincible
oliver
nolan
```

`bridge_iface` is the physical network interface: confirm it on the node with `ip -br link`. The groups `incus_cluster` and the K3s groups are separated on purpose: a new node is **agent**, never control plane, unless you add it to `k3s_server` intentionally.

Create `host_vars/nolan.yml` (copy an existing one as a base):

```
---
incus_role: "Miembro"
incus_cluster_role: member
incus_cluster_groups: [x86-nodes, default]   # arm64-nodes si es ARM64
k3s_role: agent
ip_static: true
k3s_node_labels:
  node.longhorn.io/create-default-disk: "false"   # "true" si aporta disco a Longhorn
ansible_python_interpreter: /usr/bin/python3
```

No playbook edits

The *join* to Incus, the cluster groups, the manual scheduler, the K3s labels, and the DNS are calculated from `inventory.ini` and `host_vars/`. No playbook edits are needed for a new node.

If you use automatic OS updates (`k3s_install.os_hardening`), also add its reboot schedule in `group_vars/incus_cluster/k3s_install.yml` (`reboot_schedule`), different from the others.

## Discovery (optional, recommended)

Measure the real hardware of the node (RAM, disks, network, temperature) before deciding its role:

```
ansible-playbook -i inventory.ini playbook-discovery.yml --limit nolan
```

The report is saved in `reports/nolan.md`.

## Fixed IP Address

First review what will change and then apply it. The playbook has a safety pause because it can cut the SSH connection:

```
ansible-playbook -i inventory.ini playbook-set-static-ip.yml --limit nolan --check --diff
ansible-playbook -i inventory.ini playbook-set-static-ip.yml --limit nolan
```

After that, in `inventory.ini`, change `ansible_host` to the fixed IP address and `ip_static=true`.

## Incus Package and Hardening

```
ansible-playbook -i inventory.ini playbook-bootstrap.yml --limit nolan
ansible-playbook -i inventory.ini playbook-hardening.yml --limit nolan --check --diff
ansible-playbook -i inventory.ini playbook-hardening.yml --limit nolan
```

The hardening disables password login via SSH: if step 1 didn't load your key, the role stops before leaving you out.

## Join the Node to the Incus Cluster

Include `invincible` in the `--limit`: it's the *leader* that generates the join token and assigns the cluster groups.

```
ansible-playbook -i inventory.ini playbook-incus-cluster.yml --limit invincible,nolan
```

## Join the Node to K3s

Include the control plane node (`deborah`) in the `--limit`: from there comes the token with which the agent joins. With only `--limit nolan` fails with *Missing k3s_token*.

K3s Version

The [SUC (System Upgrade Controller)](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md) updates the cluster only, but Ansible installs the fixed version of `k3s_install.core.version` (`group_vars/incus_cluster/k3s_install.yml`). Before this step, put there the version the nodes have today (`kubectl get nodes` → column `VERSION`); if not, the new node joins with an older version.

```
# el primero es el control plane
ansible-playbook -i inventory.ini playbook-k3s.yml --limit deborah,nolan
```

The same playbook applies the `k3s_node_labels` labels.

## DNS and Extras

Add the `nolan.mco.local` record to DNS (the zones are generated from the inventory):

```
ansible-playbook -i inventory.ini playbook-bind-dns.yml
```

If the node has WiFi (`has_wifi: true` and `wifi_iface` in its `host_vars`), configure the WiFi backup:

```
ansible-playbook -i inventory.ini playbook-wifi-failover.yml --limit nolan
```

## Verify

The `ssh` log in with [SSO access](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md): run `opkssh login` first (requires step [4.12](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#412-ssh-to-hosts-with-dex-optional)).

```
# nolan en Ready, misma VERSION que el resto
kubectl get nodes -o wide
# etiquetas de k3s_node_labels
kubectl get node nolan --show-labels
# nolan ONLINE
ssh devops@deborah.homelab.local sudo incus cluster list
ssh devops@deborah.homelab.local sudo incus cluster group show x86-nodes
# 192.168.20.8
dig +short nolan.mco.local @192.168.20.5
```

## If It Fails

| Symptom                                     | Check                                                                                              |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------- |
| `setup_sudo.yml` doesn't connect            | Current IP in `bootstrap/inventory.ini`; that Debian still allows root login via SSH with password |
| Lose SSH after fixed IP                     | Connect with the new IP; check `bridge_iface` with `ip -br link` on the machine console            |
| Join to Incus fails                         | `--limit` without `invincible`; free space (≥ 20 GB) on the node                                   |
| K3s: *Missing k3s_token*                    | `--limit` without the control plane node (`deborah`)                                               |
| The node joins with a different K3s version | `k3s_install.core.version` outdated (see the warning in step 7)                                    |

# Procedure — Add a repo

Routine **procedure**: repeated every time the org `maceosoft` needs a new repo. The OpenTofu pipeline of the repo [`maceosoft/infra`](https://github.com/maceosoft/infra) creates it; you add the entry in the repo map and, if the project deploys to cPanel via FTP (File Transfer Protocol), its credentials in 1Password.

Before you start

- The base platform is already running: [Phase 4 — 4.11](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#411-arc-and-opentofu-pipeline) completed (the pipeline `tofu` of `infra` runs green).
- The 1Password app is unlocked and `ONEPASSWORD_ACCOUNT_NAME` with your account name ([details](https://homelab.maceo.cl/en/maceosoft/github/#1password-items)).

## 1. Define the repo

| Data                              | Example             | Purpose                                                                 |
| --------------------------------- | ------------------- | ----------------------------------------------------------------------- |
| Name                              | `web-clientes`      | Name of the repo in `maceosoft`                                         |
| Description                       | `Sitio de clientes` | Description in GitHub                                                   |
| Stack                             | `node` or `php`     | Which CI workflow of `core-pipelines` it uses                           |
| Does it deploy via FTP to cPanel? | yes / no            | If yes, create the environments `dev` and `prod` with their credentials |

## 2. Only if it deploys: FTP credentials in 1Password

Create the item **`ftp-<repo>`** (for example `ftp-web-clientes`) in the vault `HomeLab`, type Secure Note, with these 8 fields ([how to create an item](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields)):

| Field             | Type     | Example                                   |
| ----------------- | -------- | ----------------------------------------- |
| `dev_host`        | Text     | `dev.tu-cpanel.example.com`               |
| `dev_user`        | Text     | `usuario_dev`                             |
| `dev_password`    | Password |                                           |
| `dev_remote_dir`  | Text     | `/home/usuario/dev.maceosoft.example.com` |
| `prod_host`       | Text     | `tu-cpanel.example.com`                   |
| `prod_user`       | Text     | `usuario_prod`                            |
| `prod_password`   | Password |                                           |
| `prod_remote_dir` | Text     | `/home/usuario/public_html`               |

Passwords are read without echo (`read -rs`) so they do not appear on screen or in history. Replace `web-clientes` with your repo and the example values with those of your cPanel. If you have more than one 1Password account, export `OP_ACCOUNT` first.

```
# pega cada contraseña y Enter
read -rs FTP_DEV_PASS;  read -rs FTP_PROD_PASS
op item create --category "Secure Note" --vault HomeLab --title ftp-web-clientes \
  "dev_host[text]=dev.tu-cpanel.example.com" "dev_user[text]=usuario_dev" \
  "dev_password[password]=$FTP_DEV_PASS" "dev_remote_dir[text]=/home/usuario/dev.maceosoft.example.com" \
  "prod_host[text]=tu-cpanel.example.com" "prod_user[text]=usuario_prod" \
  "prod_password[password]=$FTP_PROD_PASS" "prod_remote_dir[text]=/home/usuario/public_html"
unset FTP_DEV_PASS FTP_PROD_PASS
```

Check that it reads correctly and take it to the cluster, from the root of `homelab`. The OpenTofu runner receives these credentials as `TF_VAR_ftp` from the cluster, so this step goes **before** the PR (pull request): if not, the pipeline fails for lack of credentials.

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
.venv/bin/python ansible/scripts/apply_platform_secrets.py --check
cd ansible
ansible-playbook -i inventory.ini playbook-platform-secrets.yml
```

`--check` has to show the item `ftp-<repo>` with its 8 fields in ✓.

## 3. Add the repo to the map (PR in `infra`)

In the repo `infra`, add an entry to `local.repos` in [`github/repos.tf`](https://github.com/maceosoft/infra/blob/main/github/repos.tf):

```
locals {
  repos = {
    # ... repos existentes ...
    web-clientes = {
      description           = "Sitio de clientes"
      stack                 = "node"   # node | php
      deploy                = true     # false si no despliega por FTP
      required_status_check = "ci-cd"  # quitar si el repo no tiene CI
    }
  }
}
```

Optional fields: `ftp_protocol` (`ftp`, `ftps` or `sftp`; default `ftps`) and `visibility` (`private` or `internal`; default `private`).

Open a PR in `infra`: the pipeline `tofu` shows the `plan` with what it will create. Review it and merge; when it reaches `main`, the pipeline does the `apply`.

With the GitHub CLI (`gh`)

From your local copy of `infra`, with `gh auth login` done:

```
git switch -c agregar-web-clientes
git commit -am "Agregar repo web-clientes" && git push -u origin agregar-web-clientes
gh pr create --repo maceosoft/infra --fill
# espera al plan de tofu
gh pr checks --watch --repo maceosoft/infra
# el plan queda como comentario
gh pr view --comments --repo maceosoft/infra
# al mergear corre el apply
gh pr merge --squash --repo maceosoft/infra
# ver el apply
gh run list --repo maceosoft/infra --limit 3
```

## 4. Push the code

The repo is born with a README (`auto_init`). To push a project that you already have on your machine:

```
cd <carpeta-del-proyecto>
# si todavía no es un repo git
git init -b main
git add -A && git commit -m "Primer commit"
git remote add origin https://github.com/maceosoft/<repo>.git
git pull origin main --allow-unrelated-histories
# como owner de la org puedes pushear directo a main
git push -u origin main
```

## 5. Only if it has CI/CD: the workflow

Create `.github/workflows/ci-cd.yml` in the new repo calling the workflows of `core-pipelines` (same schema as the repo `api`). For a Node project that deploys (`develop` → dev, `main` → prod):

```
name: CI/CD

on:
  push:
    branches: [develop, main]
  pull_request:
    branches: [main]

jobs:
  ci:
    uses: maceosoft/core-pipelines/.github/workflows/ci-node.yml@main
    secrets: inherit

  deploy-dev:
    needs: ci
    if: github.ref == 'refs/heads/develop' && github.event_name == 'push'
    uses: maceosoft/core-pipelines/.github/workflows/deploy-ftp.yml@main
    with:
      stack: node
      environment: dev
    secrets: inherit

  deploy-prod:
    needs: ci
    if: github.ref == 'refs/heads/main' && github.event_name == 'push'
    uses: maceosoft/core-pipelines/.github/workflows/deploy-ftp.yml@main
    with:
      stack: node
      environment: prod
    secrets: inherit
```

For PHP use `ci-php.yml` and `stack: php`. The job name (`ci-cd`) is the check that enforces the protection of `main` if you set `required_status_check = "ci-cd"`. When `core-pipelines` starts using the tag `v1`, change `@main` to `@v1` ([versioning](https://github.com/maceosoft/core-pipelines#versionado)).

## 6. Verify

In GitHub, in the new repo:

- *Settings → Environments*: exist `dev` and `prod` (only if it deploys).
- In each environment, the secrets `FTP_HOST`, `FTP_USER`, `FTP_PASSWORD` and `FTP_REMOTE_DIR`, and the variables `PROJECT_STACK` and `FTP_PROTOCOL`.
- *Settings → Branches*: `main` protected.
- *Actions*: the first push runs the workflow on the runner `arc-runners`.

The same with `gh` (change `web-clientes` to your repo):

```
# dev, prod
gh api repos/maceosoft/web-clientes/environments --jq '.environments[].name'
# FTP_* (solo nombres)
gh secret list --env prod --repo maceosoft/web-clientes
# PROJECT_STACK, FTP_PROTOCOL
gh variable list --env prod --repo maceosoft/web-clientes
gh api repos/maceosoft/web-clientes/branches/main/protection --jq '.required_status_checks'
gh run list --repo maceosoft/web-clientes --limit 3
```

## If it fails

| Symptom                                                                   | To check                                                                                                                                    |
| ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| The `plan` of `infra` fails with *deploy = true requires FTP credentials* | Missing item `ftp-<repo>` or did not rerun `playbook-platform-secrets.yml` (step 2)                                                         |
| `--check` marks fields with ✗                                             | Exact names of the 8 fields, no section ([details](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields)) |
| The `git push` to `main` is rejected                                      | Only owners of the org can bypass the protection of `main`; otherwise, push the code via PR                                                 |
| The deploy fails to connect                                               | Data of the item `ftp-<repo>` and `ftp_protocol` in `repos.tf`; after fixing the item, repeat step 2 and relaunch the pipeline of `infra`   |

# Backups and snapshots in Incus

How to back up Incus instances (containers and VMs) and how to restore them. There are three levels, from fastest to safest:

| Level             | What it protects                                 | Where it stays           | Works if the disk is lost            |
| ----------------- | ------------------------------------------------ | ------------------------ | ------------------------------------ |
| **Snapshot**      | An instance, to roll back                        | In the same storage pool | No                                   |
| **Export**        | A complete instance, restorable on another node  | A file `.tar.gz`         | Yes, if you save it outside the node |
| **Database dump** | Incus cluster configuration (networks, profiles) | Files `.sql`             | Yes, if you save it outside the node |

The commands go on the node hosting the instance, with `sudo`.

**This decision in other environments**

Scheduled Incus snapshots for quick rollbacks, and `incus export` to a file that you copy outside the node. No licenses or backup server: enough for a few instances, as long as you test restoring.

- **Benefits:** a centralized backup system schedules, encrypts, and verifies copies of all machines, alerts if a failure occurs, and allows restoring a VM or a single file without depending on who created it.
- **Rule 3-2-1:** three copies of the data, on two different media and one off-site; many companies add an immutable or disconnected copy against ransomware.
- **RPO and RTO defined:** how much data can be lost (RPO) and how quickly recovery must occur (RTO) for each system; from there comes how often backups are made and restoration is tested, with evidence for auditing.
- **Products with support:** [Veeam Backup & Replication](https://www.veeam.com/products/veeam-data-platform/backup-recovery.html) for VMs on supported hypervisors, and for Kubernetes [Veeam Kasten](https://www.veeam.com/products/cloud/kubernetes-data-protection.html) or [Velero](https://velero.io/) (open source). Incus does not integrate with these products, so backups are made with its own commands or by files.
- **When the same applies:** in small or disconnected sites, with few instances, where a script with snapshots and exports to another disk covers the requested RPO.

| Cloud              | Equivalent service                                                                                                                                                                                                      |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCP                | [Backup and DR Service](https://cloud.google.com/backup-disaster-recovery) (VMs) and [Backup for GKE](https://docs.cloud.google.com/kubernetes-engine/docs/add-on/backup-for-gke/concepts/backup-for-gke)               |
| AWS                | [AWS Backup](https://aws.amazon.com/backup/) (EC2, EBS, EKS, and more)                                                                                                                                                  |
| Azure              | [Azure Backup](https://learn.microsoft.com/en-us/azure/backup/backup-overview) (VMs, disks, and AKS)                                                                                                                    |
| Oracle Cloud (OCI) | [Block Volume Backups](https://docs.oracle.com/en-us/iaas/Content/Block/Concepts/blockvolumebackups.htm), with [scheduled policies](https://docs.oracle.com/en-us/iaas/Content/Block/Tasks/schedulingvolumebackups.htm) |

Gain managed backup policies, copies in another region, and configurable retention or immutability without managing a backup server. Lose portability (each cloud restores only in its own) and pay for each GB stored and for restorations between regions.

## Manual snapshot (before a risky change)

```
sudo incus snapshot create <instancia> antes-de-cambio
# ... haces el cambio ...
# si salió mal
sudo incus snapshot restore <instancia> antes-de-cambio
```

## Scheduled snapshots

A daily snapshot at 06:00, which is deleted after 7 days:

```
sudo incus config set <instancia> snapshots.schedule="0 6 * * *"
sudo incus config set <instancia> snapshots.expiry=7d
sudo incus config set <instancia> snapshots.pattern="{{ creation_date|date:'2006-01-02' }}"
```

Check the existing ones with `sudo incus snapshot list <instancia>`.

## Export (backup outside the node)

```
sudo incus export <instancia> /backup/<instancia>.tar.gz
```

Copy the file to another machine or disk: an export that stays on the same node does not work if the node is lost. To restore, on any node in the cluster:

```
sudo incus import /backup/<instancia>.tar.gz
```

## Cluster configuration

```
sudo incus admin sql local .dump > /backup/incus-local.sql
sudo incus admin sql global .dump > /backup/incus-global.sql
```

## Verify

- `sudo incus snapshot list <instancia>` shows scheduled snapshots with a recent date.
- Occasionally test a `incus import` of an export on another node: a backup that has never been restored is not tested.

More detail and official links: [Incus — Backup and snapshots](https://homelab.maceo.cl/en/incus/cluster-setup/#backup-and-snapshots-optional).

# ARC runner cache

ARC (Actions Runner Controller) runners are ephemeral pods: every job starts empty. Without a cache, each job re-downloads Python or Node, pip/npm packages, Helm charts, OpenTofu providers, and the `gitops` pipeline tools. To avoid this, runners mount a shared volume at `/cache` that persists across jobs, except for the `setup-python`/`setup-node` *tool cache* (see below).

One cache per organization

Each scale set has its own cache: `arc-cache` of 20 GiB at `arc-runners` (org `maceosoft`) and another 10 GiB at `arc-runners-symintel` (legacy org). Everything that follows applies to both; commands run in the corresponding namespace.

## How it is built

| Piece             | Where                                                                                                               | What it does                                                                                                                     |
| ----------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| PVC `arc-cache`   | [`gitops/arc/cache/pvc.yaml`](https://github.com/maceosoft/gitops/blob/main/arc/cache/pvc.yaml)                     | 20 Gi, `ReadWriteMany` (RWX: multiple pods at once, on different nodes), StorageClass `longhorn` (served via NFS)                |
| Mount & variables | [`gitops/arc/values-scaleset.yaml`](https://github.com/maceosoft/gitops/blob/main/arc/values-scaleset.yaml)         | Mounts the volume at `/cache` and points each tool's cache there; an `initContainer` leaves the folders owned by the runner user |
| Cleanup           | [`gitops/arc/cache/prune-cronjob.yaml`](https://github.com/maceosoft/gitops/blob/main/arc/cache/prune-cronjob.yaml) | Weekly CronJob (Sunday 04:00) that deletes files untouched for more than 30 days from pip, npm, Helm, and XDG, and shows usage   |

What is stored and where:

| Folder                                        | Content                                                                                   | Variable                                                  |
| --------------------------------------------- | ----------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `/cache/pip`, `/cache/npm`, `/cache/composer` | Downloaded packages                                                                       | `PIP_CACHE_DIR`, `npm_config_cache`, `COMPOSER_CACHE_DIR` |
| `/cache/helm`, `/cache/xdg`                   | Helm indexes and charts, generic caches                                                   | `HELM_CACHE_HOME`, `XDG_CACHE_HOME`                       |
| `/cache/tofu-plugins`                         | OpenTofu providers (per OS and architecture)                                              | `TF_PLUGIN_CACHE_DIR`                                     |
| `/cache/dl`                                   | Tarballs of helm, kustomize, kubeconform, gitleaks, and argocd from the `gitops` pipeline | `RUNNER_CACHE`                                            |

The `setup-python` and `setup-node` tool cache is NOT shared

`setup-python` deletes and reinstalls the folder for the version it will use. With the tool cache on the shared volume, two parallel jobs (e.g., `render` and `security` from `gitops`) would stomp on the same folder and fail with `rm: cannot remove … Stale file handle`. That's why `RUNNER_TOOL_CACHE` stays at the default (`_work/_tool`, local to each runner) and Python/Node are downloaded on every job (~20 s). What *is* shared is parallel-safe: pip, npm, Helm, and the `/cache/dl` tarballs (written to a temp file and renamed).

The cluster mixes `amd64` and `arm64` nodes (`deborah`): that's why architecture-dependent content lives in separate folders by architecture (OpenTofu providers and tarball names) and is safe to share.

## How reuse is verified

The `gitops` pipeline binaries are not trusted blindly: in each job the project's published checksums file (a few KB) is downloaded, and the cached tarball is only used if its `sha256` matches; otherwise it is re-downloaded. The rest of the caches (tool cache, pip, npm) cannot be verified this way.

Risk: shared cache with a public repo

The scale set also serves public repos (e.g., `homelab`). A PR from a fork could leave a tampered package in the cache that a private repo job later uses. Mitigation: in org `maceosoft`, *Settings → Actions → General → Fork pull request workflows*, require **approval for all outside collaborators**. If you don't need `homelab` to be public, making it private eliminates the risk.

In org `maceosoft`: *Settings → Actions → General → Fork pull request workflows*, approval option for **all outside collaborators**.

With `gh` as org owner (the `admin:org` permission is granted once via `gh auth refresh -s admin:org`):

```
gh api -X PUT orgs/maceosoft/actions/permissions/fork-pr-contributor-approval \
  -f approval_policy=all_external_contributors
gh api orgs/maceosoft/actions/permissions/fork-pr-contributor-approval
```

## Verify it works

1. The volume is bound and runners mount it:

   ```
   # STATUS: Bound
   kubectl -n arc-runners get pvc arc-cache
   kubectl -n arc-runners get cronjob arc-cache-prune
   ```

1. Run the same workflow twice in a row. On the second run, `pip install` says `Using cached …` and the `gitops` tools step does not download tarballs (takes seconds).

1. Check how much space it uses (with a temp pod, the runner is gone once the job finishes):

   ```
   kubectl -n arc-runners run cache-du --rm -it --restart=Never --image=busybox:1.37 \
     --overrides='{"spec":{"containers":[{"name":"cache-du","image":"busybox:1.37","command":["sh","-c","du -sh /cache/*; df -h /cache | tail -1"],"volumeMounts":[{"name":"c","mountPath":"/cache"}]}],"volumes":[{"name":"c","persistentVolumeClaim":{"claimName":"arc-cache"}}]}}'
   ```

## Clear the cache

It's safe: jobs will re-download what's missing. With a temp pod like the one above but using `rm -rf /cache/*`, or by deleting the PVC (Argo CD recreates it):

```
kubectl -n arc-runners delete pvc arc-cache
```

Do this if you suspect a tampered or corrupted cache.

## If it fails

| Symptom                                               | Check                                                                                                                                                    |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Runners won't start (`Pending`)                       | `kubectl -n arc-runners describe pvc arc-cache`: Longhorn must be `Healthy` and nodes must have `nfs-common` (role `k3s_prereqs`)                        |
| Job says `Permission denied` in `/cache`              | The `initContainer cache-perms` couldn't `chown` (NFS with root squash): `kubectl -n arc-runners logs <pod> -c cache-perms`                              |
| `rm: cannot remove … Stale file handle` in `/cache/…` | Two jobs writing the same volume folder: don't share caches that delete and reinstall (like the tool cache) there; if data is corrupt, clear that folder |
| `Checksum incorrecto` in the `gitops` pipeline        | A corrupt download or a tampered cache: clear `/cache/dl` and retry                                                                                      |
| Volume fills up                                       | `df -h /cache` from the temp pod; increase PVC size (Longhorn allows expansion) or clear `/cache/xdg` or `/cache/pip`                                    |

# Change the Gateway Controller

The HomeLab publishes its services with [Gateway API](https://gateway-api.sigs.k8s.io/) (`Gateway` and `HTTPRoute`), not with `Ingress`. There is a single `Gateway`, called `homelab` (namespace `gateway`), with an HTTP listener (80) and one HTTPS (443) for `*.homelab.local`. Each app hooks up with its own `HTTPRoute` (`argocd/route`, `vcluster/route`).

Who handles that Gateway is **one controller at a time**; the three cannot coexist on the same IP 80/443 from MetalLB:

| Application          | Controller                                                                                   | Notes                                                            |
| -------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| `kong` (**default**) | [Kong Ingress Controller](https://developer.konghq.com/kubernetes-ingress-controller/) (KIC) | Uses the Kong installed by the chart (unmanaged mode)            |
| `traefik`            | [Traefik](https://doc.traefik.io/traefik/routing/providers/kubernetes-gateway/)              | Only Gateway API: no `Ingress` or `IngressRoute`                 |
| `nginx-gateway`      | [NGINX Gateway Fabric](https://docs.nginx.com/nginx-gateway-fabric/)                         | Also needs `nginx-gateway-helm-repo` (OCI registry of the chart) |

The `Gateway` of each is in `gateway/<kong|traefik|nginx>/` of the repo [`gitops`](https://github.com/maceosoft/gitops); `GatewayClass`, ports, and annotations differ from one to another, the `HTTPRoute` do not.

## How TLS Works

- **cert-manager** (`cert-manager`) and a custom CA (`cert-manager-config`, `ClusterIssuer homelab-ca`) issue the certificate. The `Gateway` carries the annotation `cert-manager.io/cluster-issuer: homelab-ca` and cert-manager creates only the Secret `homelab-tls`.

- The Gateway **terminates TLS** (Transport Layer Security) and speaks HTTP with the services. That's why `argocd-server` runs with `server.insecure` (in `argocd/config`) and the `HTTPRoute` points to its port 80. The diff of the `gitops` pipeline uses `--plaintext` for the same reason.

- **Kong is the exception.** KIC (Kong Ingress Controller) 3.5 with Kong 3.9 rejects (`value must be null`) the certificate issued for a Gateway API listener, and Kong never gets `Ready`. That's why Kong's Gateway does not use `certificateRefs`: cert-manager issues `kong-default-tls` (`gateway/kong/certificate.yaml`), the chart mounts it in the pod and Kong loads it as the default certificate with `KONG_SSL_CERT` and `KONG_SSL_CERT_KEY`. Kong reads it only on startup: **after each renewal (every ~60 days) restart the pod**: `kubectl -n kong rollout restart deploy/kong-gateway`. Traefik and NGINX Gateway Fabric use the certificate issued by cert-manager from the Gateway (`homelab-tls`) and update automatically.

- To make the browser trust the CA, import its root certificate:

  ```
  kubectl -n cert-manager get secret homelab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d > homelab-ca.crt
  ```

  - **macOS:** open `homelab-ca.crt` with a double-click (added to *Keychain Access*), open it in the list and in **Trust** choose *Always Trust*.
  - **Firefox** (has its own store; if it does not trust the system CA): *Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import*, and mark it to trust for identifying websites.

  ```
  # macOS (llavero del sistema)
  sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain homelab-ca.crt
  # Debian/Ubuntu
  sudo cp homelab-ca.crt /usr/local/share/ca-certificates/ && sudo update-ca-certificates
  ```

  If Firefox still does not trust it (Linux does not read the system store), import it from its interface.

## Why the Service Uses `externalTrafficPolicy: Local`

With the default policy (`Cluster`), traffic arriving from the LAN to a node that **does not** have the controller pod is redirected to another node, and in this cluster that connection does not respond: `curl` to `192.168.23.200` expired even though DNS resolved correctly. It only happened with external traffic entering via `deborah` or `oliver`; pods, services, and connections from the nodes themselves worked fine. With `Local`, MetalLB advertises the IP from the node where the controller runs and the traffic goes directly to the pod (and also preserves the client IP). All three controllers bring it in `gitops/argocd/apps/`.

Underlying Cause Not Resolved

The reason why inter-node traffic redirection fails for external traffic (the Calico overlay between pods works) is unclear. It affects any other Service `LoadBalancer` or `NodePort` with policy `Cluster`. To investigate, packet captures are needed on the nodes (`tcpdump` is not installed).

## Change Controller

1. In `bootstrap/root-appset.yaml` (repo `gitops`), **comment out** the current controller and **uncomment** the new one (and `nginx-gateway-helm-repo` if it's NGINX). Push the changes.

1. Reapply the ApplicationSet (Argo CD does not manage it):

   ```
   kubectl apply -f gitops/bootstrap/root-appset.yaml
   ```

1. The Applications of the controllers carry the Argo CD finalizer: when commenting out the app, `homelab-root` deletes the Application **and** its resources (the Gateway and the IP are freed for the new one). Wait until it disappears:

   ```
   kubectl -n argocd get applications
   ```

1. Verify that the Gateway is scheduled and has an IP:

   ```
   kubectl get gateway homelab -n gateway
   kubectl get httproute -A
   ```

   It should show `PROGRAMMED=True`, an address between `192.168.23.200` and `.220`, and the `HTTPRoute` with `Accepted`.

## First Time (Coming from ingress-nginx)

`ingress-nginx` and the `Ingress` of Argo CD no longer exist in `gitops`. Their Applications disappear when reapplying the ApplicationSet, but what they created remains in the cluster:

```
kubectl -n argocd delete ingress argocd-server
kubectl delete namespace ingress-nginx
kubectl delete clusterrole,clusterrolebinding,ingressclass,validatingwebhookconfiguration \
  -l app.kubernetes.io/instance=ingress-nginx
```

Apply the change of `argocd` first (activate `server.insecure`): until the Gateway is `Programmed`, log in to Argo CD with `kubectl port-forward svc/argocd-server -n argocd 8080:80` ([`http://localhost:8080`](http://localhost:8080)).

## If it fails

| Symptom                                                                                                      | Check                                                                                                                                                                                                                |
| ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Gateway` without address or `PROGRAMMED=False`                                                              | `kubectl describe gateway homelab -n gateway`: GatewayClass missing (controller not active) or Gateway API CRDs absent (`gateway-api` not synced)                                                                    |
| `curl https://argocd.homelab.local` expires even though DNS resolves to `192.168.23.200`                     | Check that the controller Service has `externalTrafficPolicy: Local` (`kubectl -n kong get svc kong-gateway-proxy -o jsonpath='{.spec.externalTrafficPolicy}'`) and that the pod runs on a node MetalLB can announce |
| Two active controllers                                                                                       | Comment out one: they fight for MetalLB IPs                                                                                                                                                                          |
| The Gateway has no certificate (`homelab-tls` does not exist)                                                | `kubectl describe certificate -n gateway`; `cert-manager-config` must be `Healthy`                                                                                                                                   |
| `HTTPRoute` without `Accepted`                                                                               | `kubectl describe httproute -n argocd argocd-server`: `parentRefs` must point to `homelab` in `gateway`                                                                                                              |
| With Kong, pods do not reach `Ready` and the controller log says `failed posting new config to /config`      | See the Kong point in *How it works with TLS*; confirm that the Secret `kong-default-tls` (`kubectl -n kong get certificate,secret`) exists                                                                          |
| With Kong, the listener remains unprogrammed                                                                 | KIC links the Gateway to the Kong chart ports; check `kubectl describe gateway` and the Service ports `kong-gateway-proxy`                                                                                           |
| With Kong, `kubectl -n kong logs deploy/kong-controller` says `invalid config.location: missing host in url` | A `HTTPRoute` with filter `RequestRedirect` without `hostname`: Kong builds the plugin `redirect` with a full URL. Add `hostname` to the filter (already in `argocd/route` and `vcluster/route`)                     |
| Argo CD responds `307` to the same URL (loop) or `too many redirects`                                        | `server.insecure` is read only on startup: restart the pod (`kubectl -n argocd rollout restart deploy/argocd-server`) and check the value (`kubectl -n argocd get cm argocd-cmd-params-cm -o yaml`)                  |
| With Kong, `http://…` responds `426 Upgrade Required`                                                        | The HTTPS-only route wins over the redirect route: the annotation `konghq.com/https-redirect-status-code: "301"` of `HTTPRoute` makes Kong redirect (already in `argocd/route` and `vcluster/route`)                 |

# Incus Metrics Certificate

How to give the monitoring tool permission to **read** Incus metrics, and what to do if it stops working. If you just want to know what the certificate is, read the first section.

## What is a Client Certificate

A **client certificate** is a digital ID card. When a program connects to Incus over HTTPS (encrypted web connection), Incus asks it to identify itself and the program shows its certificate. If Incus has that card noted in its list of trusted certificates, it lets it in; if not, it closes the door. It's like an office key: whoever carries it doesn't matter, it only opens the doors it was made for.

Each card comes with a **private key**, which proves that the card is yours. The certificate can be shown without risk (it's the public part); the private key is never shared.

**What it's used for here.** Each Incus node publishes the CPU, memory, disk, and network usage of its instances in `https://<nodo>:8443/1.0/metrics`. The monitoring agent (`vmagent`, the one that collects metrics) uses this certificate to enter and read that page every 60 seconds.

**Why it's read-only.** Incus allows creating certificates of different types. The type `metrics` can only read metrics: it cannot create, delete, or change instances. So, if someone stole the monitoring key, they could see how much CPU an instance uses, but not manage it. That's why an administrator's certificate is not used for this.

**Who issues it.** It is issued by `cert-manager` (the Kubernetes component that creates and renews certificates), signed by the internal HomeLab CA (`homelab-ca`). It is declared in Git as a resource `Certificate` called `incus-metrics` in the namespace `monitoring` ([`incus-certificate.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/config/scrapes/incus-certificate.yaml) in the repo `gitops`). The key pair is stored in the Secret `monitoring/incus-metrics` and **the private key does not leave the cluster**. Incus does not validate the CA: it only accepts the certificates that are manually noted in its list, and that is handled by step 2.

## Procedure

It is done once, or every time the certificate is recreated. It requires that Argo CD has already synchronized the monitoring and that you have the HomeLab kubeconfig (`~/.kube/homelab-k3s.yaml`).

1. **Check that the certificate is issued.** It should appear `READY` in `True`:

   ```
   kubectl -n monitoring get certificate incus-metrics
   ```

1. **Register it in Incus** with the playbook `playbook-incus-metrics.yml` (from the folder `ansible/` of the repo `homelab`). It reads only the public part of the Secret and notes it in Incus's trusted list with the type `metrics`. If it was already noted, it does nothing:

   ```
   ansible-playbook -i inventory.ini playbook-incus-metrics.yml
   ```

1. **Verify.** In `invincible` there should be a client of type `metrics` called `vmagent`:

   ```
   sudo incus config trust list
   ```

   And this query should return a value for each node with instances: `count by (nodename) (incus_procs_total)`.

   In Grafana, *Explore*, run the query.

   Open a tunnel to VictoriaMetrics and query its API (leave the `port-forward` running in another terminal):

   ```
   kubectl -n monitoring port-forward svc/vmsingle-vm-victoria-metrics-k8s-stack 8428:8428
   curl -s http://127.0.0.1:8428/api/v1/query --data-urlencode 'query=count by (nodename) (incus_procs_total)'
   ```

## If it fails

| Symptom                                                                | Likely Cause                                                                     | What to Check                                                                                                        |
| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| The playbook says *Could not read monitoring/incus-metrics*            | Argo CD has not yet created the `Certificate`, or cert-manager has not issued it | `kubectl -n monitoring describe certificate incus-metrics`; that `homelab-ca` is ready (`kubectl get clusterissuer`) |
| The playbook completes successfully but there are no metrics `incus_*` | The agent does not yet have the Secret, or there is no client `metrics` in Incus | `kubectl -n monitoring get secret incus-metrics` and `sudo incus config trust list` in `invincible`                  |
| Incus metrics disappeared after recreating the `Certificate`           | The Secret has a new certificate that Incus does not know                        | Run step 2 again. To clean the old card: `sudo incus config trust remove <huella>`                                   |
| A specific node does not appear                                        | Incus on that node does not respond on port 8443                                 | `curl -k https://<nodo>:8443/` from another LAN machine                                                              |

To change the certificate, delete the Secret (`kubectl -n monitoring delete secret incus-metrics`): cert-manager issues a new one, and then steps 2 and 3 are repeated. Other key rotation procedures are in [Rotate Credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md).

# Local AI Assistant (Ollama + Open WebUI)

The HomeLab has its own artificial intelligence, free and without leaving the house: language models (LLM, *Large Language Model*) that run on the home machines and are used from a web page (Open WebUI) to ask about the HomeLab, and from the pipeline that translates this documentation to English. Nothing is sent to paid services or the internet.

There are **several machines running models** (called *Ollama nodes*) and **a single entry point**: a router that knows which models each node has and sends each request to the corresponding one. Nodes announce themselves, so adding one is easy ([guide](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md)). And nodes only turn on when someone writes to them: the rest of the time Ollama (the program that runs the model) is off and uses no memory.

**This decision in other environments**

Local AI with Ollama on multiple nodes that turn on on demand, a custom router (`ollama-router`) as the only gateway and Open WebUI as chat. It's free, leverages machines already in the house, and no data leaves for the internet.

- **Benefits:** prompts and documents never leave your network, you don't pay per query, and a router with a token per client tells you who uses the AI and lets you cut one off without affecting the rest.
- **Why a company also chooses local AI:** sensitive data (clients, code, contracts), regulation or data sovereignty that doesn't allow sending prompts to third parties, disconnected sites, and predictable cost when usage volume is very high.
- **Dedicated GPUs** instead of machines that turn on and off: models stay loaded all the time to respond quickly to many users at once.
- **Supported inference servers** instead of Ollama: [vLLM](https://docs.vllm.ai/) (open source), [NVIDIA NIM](https://developer.nvidia.com/nim) or [Red Hat AI Inference](https://www.redhat.com/en/products/ai/inference) (based on vLLM), with high availability and support contract.
- **Access control and auditing:** entry with corporate identity (SSO), permissions by team, and logging of who asked what, for compliance.
- **Model and license evaluation:** before using a model, its quality is measured and its license is reviewed to allow commercial use. And someone has to operate all this: it's personnel cost.

| Cloud              | Equivalent Service                                                                                                          |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------- |
| GCP                | [Gemini Enterprise Agent Platform](https://cloud.google.com/products/gemini-enterprise-agent-platform) (formerly Vertex AI) |
| AWS                | [Amazon Bedrock](https://aws.amazon.com/bedrock/)                                                                           |
| Azure              | [Microsoft Foundry](https://azure.microsoft.com/en-us/products/ai-foundry) (formerly Azure AI Foundry)                      |
| Oracle Cloud (OCI) | [OCI Generative AI](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)                                      |

You gain managed models, no GPUs to buy or maintain, with SLA and immediate scale. You lose control: prompts and data travel to the provider, you pay per use (expensive at high volume), and you depend on their models and their prices.

## What each piece does

| Piece                               | What it does                                                                            | Where it lives                                                                                    |
| ----------------------------------- | --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| **Ollama**                          | Runs the language models                                                                | On each node (`deborah`, `invincible`, `oliver` and, when turned on, the Mac), outside Kubernetes |
| **Socket and proxy**                | Listen on port `11434` of the node and wake Ollama only when a request arrives          | Each node (systemd on Linux, `launchd` on the Mac)                                                |
| **Registration agent**              | Every 60 seconds tells the router the node is alive and which models it has             | Each node                                                                                         |
| **Ollama Router** (`ollama-router`) | Single entry point: validates token, chooses node by model, and publishes metrics       | Cluster, namespace `ai`                                                                           |
| **Kong**                            | Receives external traffic on `https://ollama.homelab.local` and forwards to the router  | Cluster                                                                                           |
| **Open WebUI**                      | Chat page; login with Dex (GitHub)                                                      | Cluster, namespace `ai`, at `open-webui.homelab.local`                                            |
| **Open Terminal**                   | Terminal with code execution that the assistant uses as a tool; isolated and admin-only | Cluster, namespace `ai-sandbox`                                                                   |
| **Read-only tools**                 | Let the assistant query cluster state and metrics                                       | Cluster, namespace `ai`                                                                           |
| **Alert summary and K8sGPT**        | Explain cluster alerts and issues in plain language                                     | Cluster, namespace `ai`                                                                           |

## Architecture

```
flowchart LR
    subgraph Cluster["Clúster"]
        OW["Open WebUI<br/>open-webui.homelab.local"]
        TR["Traducción de la doc<br/>(runners de CI)"]
        AL["alert-triage<br/>K8sGPT · reportes"]
        K["Kong<br/>https://ollama.homelab.local"]
        R["ollama-router<br/>(namespace ai)"]
    end
    subgraph Nodos["Nodos de Ollama"]
        D["deborah"]
        I["invincible"]
        O["oliver"]
        M["Mac (si está en on)"]
    end
    OW -->|"Bearer"| R
    TR -->|"Bearer"| R
    AL -->|"Bearer"| R
    EXT["Equipos externos"] -->|"HTTPS + Bearer"| K --> R
    R -->|"según el modelo"| D
    R --> I
    R --> O
    R -.-> M
    D -. "registro cada 60 s" .-> R
    I -. "registro" .-> R
    O -. "registro" .-> R
    M -. "registro" .-> R
```

- **Single entry point.** Clients inside the cluster use the internal Service `http://ollama-router.ai.svc`; external ones use `https://ollama.homelab.local` (HTTPS, via Kong). There is no Service per node nor `/api/tags` cache in Kong: the router responds `/api/tags` from its registry, without waking anyone.
- **Mandatory authentication.** Each client has its own token (`Authorization: Bearer ...`). Without a token, the router responds `401`. See [Security](#security-and-firewall).
- **Node selection.** The router looks at the `model` of the request and picks among live nodes that have it: the one that last used it (to avoid reloading), then the highest priority, then the least busy. A model that is only on the Mac goes to the Mac. If a node fails to connect, it avoids it for 30 seconds.
- **How nodes announce themselves.** See [Add a node to the Ollama cluster](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md): the agent reads models from Ollama folders (without waking it), sends them to the router, and the router expires them if it stops hearing from the node for 180 seconds (3 heartbeats).

How the "sleep and wake" of each node works:

1. A **socket** (systemd on Linux, `launchd` on the Mac) always listens on `11434`. It costs almost nothing.
1. When a real request arrives, a **proxy** starts, which turns on Ollama and forwards traffic to it on `127.0.0.1:11435`.
1. After 5 minutes without traffic, the proxy exits on its own and Ollama stops: the machine recovers its memory.

## Create the Open WebUI key (once only)

Open WebUI signs its users' sessions with a secret key, `WEBUI_SECRET_KEY`. Without it the pod will not start. Like the rest of the platform secrets, it lives in 1Password and does **not** go through Git (`homelab` is public and `gitops`, although private, does not store secrets):

1. Generate the key. It must be random, 64 hexadecimal characters:

   ```
   openssl rand -hex 32
   ```

1. Create the item in 1Password (vault `HomeLab`, type *Secure Note*, named `open-webui`, with the field `WEBUI_SECRET_KEY`):

   From the 1Password app: the item name and field name must be exact and the field must not be inside a section.

   With the `op` CLI and 1Password unlocked, a single command generates the key and creates the item:

   ```
   op item create --category "Secure Note" --vault HomeLab --title open-webui "WEBUI_SECRET_KEY[password]=$(openssl rand -hex 32)"
   ```

   With `op` step 1 is not needed: the command already generates the value.

1. Verify the item without showing the value: `python3 ansible/scripts/apply_platform_secrets.py --check` must show `✓ WEBUI_SECRET_KEY` under `open-webui`.

1. Apply it (create the namespace `ai` and the Secret `ai/open-webui-secret`):

   ```
   cd ansible
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

Do this before Argo CD syncs the Application `open-webui`. To rotate the key, repeat steps 1, 2, and 4 and restart the Open WebUI pod; users will have to log in again. More on the flow in [Secret management](https://homelab.maceo.cl/en/secrets/index.md) and [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md).

## Models per node

Each node declares its models; the router joins them in `/api/tags` and uses them to choose. The initial assignment is (adjusted by measuring tokens per second and memory on each node):

| Node         | Models                                                                                                   | Purpose                                                                                                 |
| ------------ | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `deborah`    | `qwen2.5:7b-instruct` as fallback (and an optional 14B)                                                  | Default model when `invincible` does not respond: chat with the assistant and documentation translation |
| `invincible` | `qwen2.5-coder:7b` and `qwen2.5:7b-instruct` (preferred node for the 7B, priority 80 vs 60 of `deborah`) | Code help (Ansible, YAML, and Kubernetes manifests) and the default model                               |
| `oliver`     | `nomic-embed-text` and models up to 3B                                                                   | *Embeddings* (turn text into numbers to search the documentation) and lightweight tasks                 |
| Mac          | Large models                                                                                             | Heavy tasks, only while the switch is on `on`                                                           |

Each node handles **one request at a time** (`OLLAMA_NUM_PARALLEL=1`) and the router keeps a semaphore per node. The list of models on a Linux node is the variable `ollama_ondemand_models` of its `host_vars` and is applied with the tag `models` (see [Ansible](https://homelab.maceo.cl/en/ansible/index.md)); on the Mac they are managed with `ollama pull`. Model management is **not** done over the network: the router blocks `pull`, `create`, `delete`, `copy` and `push`.

## Metrics per model and node

Ollama does not publish metrics or annotate the model of each request, and Kong does not see it either (it goes in the body). That is why the router publishes them in `/metrics` (Prometheus format): requests by model, node, and code; duration; tokens (from the `prompt_eval_count` and `eval_count` of each response); in-flight requests; whether each node is alive and when it was last heard from. VictoriaMetrics scrapes them with a `VMServiceScrape` and they are visible in the **HomeLab / Ollama** dashboard in Grafana (nodes with heartbeat, in-flight requests, 5xx errors, tokens and latency by model and node). Two alerts fire if the service fails: `OllamaRouterCaido` and `OllamaRouterSinNodos`. They serve to know **which models are used and on which nodes**, without reading logs. The router also writes a JSON log per request that goes to [VictoriaLogs](https://homelab.maceo.cl/en/operacion/logs/index.md).

## Cold start

Waking Ollama and loading the model into memory is not instantaneous: the first response after a period of inactivity takes longer (and even more if the model is large or the machine is slow).

- On each node, the service waits for Ollama to listen before considering the startup ready.
- In Kong, the router route has long timeouts and **no retries** (a retry would repeat a prompt that is already running).
- Clients tolerate the wait: the documentation translation retries the connection for up to about 5 minutes.

If the first message fails or is cut off, wait a minute and try again.

## What wakes a node and what does not

A **real message** to the model wakes a node (a chat, a translation, an embedding). Nothing "decorative" should wake it: the router responds `/api/tags` and `/api/version` from its registry, it does not probe the nodes, there are no active health checks, `ollama.homelab.local` is excluded from monitoring probes and Open WebUI has automatic titles, tags, and autocomplete disabled.

## Check the router

```
# Con el token de un cliente, leído de 1Password sin mostrarlo (si tienes más de una cuenta, exporta OP_ACCOUNT antes)
TOKEN=$(op read "op://HomeLab/ollama-router/reports")

# Nodos vivos y sus modelos
curl -s -H "Authorization: Bearer $TOKEN" https://ollama.homelab.local/api/tags

# Sin token debe dar 401; la gestión de modelos, 403
curl -s -o /dev/null -w '%{http_code}\n' https://ollama.homelab.local/api/tags
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "Authorization: Bearer $TOKEN" https://ollama.homelab.local/api/pull
```

And that a node sleeps and wakes (by SSH to the node, see [SSH access](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md)):

```
# inactive en reposo
systemctl is-active ollama
# tras un chat real por el router:
# active
systemctl is-active ollama
# unos 5 minutos después:
# inactive otra vez
systemctl is-active ollama
```

## Open WebUI with Dex

Open WebUI has no accounts of its own: you sign in with **SSO** (Single Sign-On) (via GitHub) through Dex, just like in Grafana.

- Only the `devops` team of the `maceosoft` organization can sign in; Open WebUI administrators are the members of that same team (the role arrives in the `groups` claim).
- No one can register or sign in with a password: there is no local form (`ENABLE_LOGIN_FORM=False`), registration is closed (`ENABLE_SIGNUP=False`) and the API rejects password login (`ENABLE_PASSWORD_AUTH=False`). Not even the first administrator is created manually: it is the first person from the team who signs in via SSO.
- The Dex client is public and uses PKCE (*Proof Key for Code Exchange*, a protection for clients without a secret).
- Open WebUI runs unprivileged (non-root user) and with a read-only filesystem.
- MCP tools (*Model Context Protocol*, a standard for giving tools to a model) are only visible to administrators.
- The pod trusts the HomeLab CA (certificate authority), which is the one that signs the Dex certificate.

## Ask the Assistant about the HomeLab (RAG)

RAG (*Retrieval-Augmented Generation*) means giving the model your own documents so it answers with them instead of making things up. This site already publishes a file with all the documentation in plain text, `llms-full.txt` (at the site root), designed exactly for this. Loading it is **a manual step in the UI**:

1. Download `llms-full.txt` from the published site.
1. In Open WebUI go to **Workspace → Knowledge**, create a collection and upload the file (and, if you want, the runbooks from [Operations](https://homelab.maceo.cl/en/operacion/index.md)).
1. In a chat, type `#` and pick that collection so the assistant queries it.

Open WebUI uses `nomic-embed-text` as the embeddings model, so indexing the documentation also goes through Ollama (and can wake a node). When the documentation changes you have to re-upload the file.

## Read-only tools and the "propose only" policy

The assistant can **inspect** the HomeLab via MCP tools (*Model Context Protocol*, a standard for giving tools to a model), exposed to Open WebUI as tool servers:

- **Kubernetes**: read-only cluster state, using a service account with minimal permissions (no access to `secrets` and no write permissions at all) and a network policy that limits who can call it.
- **VictoriaMetrics and Alertmanager**: queries for metrics, active alerts, and rules.

Querying metrics or Kubernetes does not wake any node; only the message to the model does.

The assistant never applies changes

Its policy is **propose only**. If you ask for a change, it returns the diff or a draft pull request for the repo [`maceosoft/gitops`](https://github.com/maceosoft/gitops); **you review and approve it**. Even if it could write, Argo CD (with `selfHeal`) would revert any manual change.

To verify the service account permissions for the tools:

```
# sin verbos de escritura
kubectl auth can-i --list --as=system:serviceaccount:ai:<cuenta>
```

## Alert summary to Discord

Alertmanager has an extra receiver that sends each alert to a service in the `ai` namespace (with a Bearer token); this asks Ollama via the router for a plain-language summary and posts it to Discord.

The summary does NOT replace real alerts

The original alert still goes out through its usual channels (Discord and healthchecks.io), **without passing through the AI and even if nodes are asleep**. The summary arrives later and is an extra: if Ollama doesn't respond, nothing happens. To avoid waking a node unnecessarily, the service groups and deduplicates alerts and rate-limits calls.

## K8sGPT

[K8sGPT](https://github.com/k8sgpt-ai/k8sgpt) scans the cluster for problems and explains them with help from Ollama.

K8sGPT is staged: the decision is yours

Its line is commented out in `root-appset.yaml`. The operator chart ships a ClusterRole with `*/*/*` permissions (effectively, cluster admin) and offers no option to remove it; plus dynamic RBAC reads Secrets across the whole cluster. To enable it there are three paths, described in [`security/k8sgpt-rbac/`](https://github.com/maceosoft/gitops/tree/main/security/k8sgpt-rbac): accept the risk and uncomment the line; use the scoped role proposed there (untested); and enable the namespace NetworkPolicy together with the app. Until activated, the rest of this section describes how it works. It runs as an operator, with scheduled and infrequent analysis, and leaves its results as cluster objects:

```
kubectl get results.core.k8sgpt.ai
```

All analysis is local: nothing leaves the house.

## Open Terminal: terminal for the assistant

Open Terminal is a code-execution terminal that Open WebUI uses as an **agent tool**: the assistant can write and run scripts, process files, and return the result inside the chat. It runs in its own namespace, `ai-sandbox`, with the image `ghcr.io/open-webui/open-terminal` (`slim` variant), and is deployed from [`ai/open-terminal/`](https://github.com/maceosoft/gitops/tree/main/ai/open-terminal).

**What it's for:** running code (Python, shell), converting or analyzing files, testing a script before taking it to Ansible or a manifest. **What it doesn't do:** it doesn't manage the HomeLab; it has no cluster access.

### How to use it

1. Log in to Open WebUI via SSO (GitHub) and open a chat.
1. Enable the **Open Terminal** integration in the chat (only visible to admins and the `maceosoft:devops` group).
1. Ask for something that requires running code, e.g., "calculate the average of this CSV (comma-separated data file)". The assistant runs the command in the terminal and shows you the output.

### How it connects (admins)

In Open WebUI: *Admin Settings → Integrations →* *Open Terminal* **(not in "External Tools" or "Tool Servers"). Enter the internal URL of the Open Terminal Service, the key, and the authentication type** Bearer\*\*. The variable `ENABLE_TOOL_SERVERS=true` must be enabled in Open WebUI. Access is restricted to admins and the `maceosoft:devops` group.

This connection is configured **only from the Open WebUI UI**: there is no command to do it.

### Open Terminal limits

- **Scoped sandbox:** max 1 CPU (central processing unit) and 2 GB RAM (memory).
- **No cluster or internal network access.** A NetworkPolicy only allows ingress from Open WebUI, and egress is limited to internet on ports 80 and 443.
- **Small persistent volume:** 2 GB on Longhorn; files survive a restart, but space is limited.

### Risks

Runs arbitrary code

Everything the assistant (or whoever uses the chat) decides to run executes for real. **It is not a security boundary between users**: whoever uses it shares the same terminal and the same files. That's why it's for admins only, and **`docker.sock` must never be mounted** nor given credentials, kubeconfig, or Secrets.

### API key: generate and rotate

The key protects the Open Terminal API (Bearer). It is created and rotated the same way as the others:

1. Generate the key and store it in 1Password:

   1. `openssl rand -hex 32`
   1. In 1Password, vault `HomeLab`, item **`open-terminal`**, field **`api_key`**.

   The value is generated inside the command, so you never see or copy it. If the item already exists (when rotating):

   ```
   op item edit open-terminal "api_key[password]=$(openssl rand -hex 32)"
   ```

   The first time, if the item does not exist yet:

   ```
   op item create --category "Secure Note" --vault HomeLab --title open-terminal "api_key[password]=$(openssl rand -hex 32)"
   ```

1. `cd ansible && ansible-playbook -i inventory.ini playbook-platform-secrets.yml` creates the Secrets `ai-sandbox/open-terminal-api-key` and `ai/open-terminal-api-key`.

1. Restart the Open Terminal and Open WebUI pods so they read the new value, and update the key in *Admin Settings → Integrations → Open Terminal* if needed (this last step is only from the UI).

The key is mounted in the pod as a **file** (`OPEN_TERMINAL_API_KEY_FILE`, permissions `0440`), not as an environment variable. The `ai-sandbox` NetworkPolicy travels with the pod (`ai/open-terminal/networkpolicy.yaml`, in the order Namespace → NetworkPolicy → pod). If you deploy the app **before** creating the Secret, Open WebUI starts fine but the connection to Open Terminal responds `401` until you create the Secret and restart the pod.

See [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md).

## Limits

- **CPU only (central processing unit).** Linux nodes have no graphics card (GPU), only processor; a 7-billion-parameter model responds slowly (words per second, not instant). Translating all documentation for the first time takes hours. The Mac, with unified memory, is the exception for large models.
- **One request per node at a time.** For more parallelism you need more nodes.
- **The router is a single point.** It runs in a single replica and keeps the registry in memory: if it goes down, AI goes down (nodes re-register automatically in under a minute when it comes back). Real alerts do not depend on it.
- **A loaded model takes memory** away from other workloads on the node while it lasts.
- **The K3s control plane** is on `deborah`, the same node that loads models. That is why Ollama runs on demand: it only uses memory while responding.

## Security and firewall

- **Clients with token.** The router requires `Authorization: Bearer <token>` on `/api/*` and `/v1/*`. There is one token per client (`open-webui`, `k8sgpt`, `translate`, `reports`, `alert-triage`), all in the `ollama-router` item of 1Password; each is rotated separately ([Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md)). Open WebUI and K8sGPT (when active) send their key this way; scripts read `OLLAMA_TOKEN`.
- **Registration with a different token.** `/_registry` uses a token distinct from the client tokens (field `registry_token` of the same item). With it you cannot use inference.
- **Node Ollama does not authenticate.** Each node listens only on its IP and its firewall (`nftables`) allows port `11434` from K3s nodes **and from pods** (the router may be on the same node as the Ollama it uses). The real inference protection is the router Bearer token plus the `ai` namespace NetworkPolicy; clients do not talk to a node. It is a documented exception: see [Authentication model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md).
- **HTTPS only via Kong.** There is no HTTP listener for `ollama.homelab.local`. Inside the cluster, the internal Service `ollama-router.ai.svc` only allows ingress from allowed namespaces and labels when the `ai` namespace NetworkPolicy is active (it is enabled in stages; see [What is active](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/#whats-active-and-whats-pending-activation)).
- **Router limits.** Registers up to 64 nodes with 2000 models each, priorities 0–100, and model names with the format `^[A-Za-z0-9._:/@-]{1,128}$`; the `HEAD`, `OPTIONS`, and `PATCH` methods respond `405`; timeouts are 900 seconds. Only accepts node registrations from within `ALLOWED_NODE_CIDRS` (`192.168.20.0/22`), and a live node cannot be overwritten from another IP (responds `409` until it expires or `DELETE` is done). Kong only exposes `/api`, `/v1`, and `/_registry`: neither `/metrics` nor `/healthz` go out that way.
- **Model management blocked** (`pull`, `create`, `delete`, `copy`, `push`, and `blobs`).
- Open WebUI is exposed only via HTTPS and its internal key (`WEBUI_SECRET_KEY`) travels as an encrypted secret.

## Where everything is

| What                                         | Where                                                                                                                                                                  |
| -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Ansible role `ollama_ondemand` (Linux nodes) | [`ansible/roles/ollama_ondemand`](https://github.com/maceosoft/homelab/tree/main/ansible/roles/ollama_ondemand)                                                        |
| Playbook                                     | [`ansible/playbook-ollama.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-ollama.yml) (group `llm_nodes`)                                        |
| Node agent and macOS bundle                  | [`ollama-node/`](https://github.com/maceosoft/homelab/tree/main/ollama-node)                                                                                           |
| Router                                       | [`ai/ollama-router/`](https://github.com/maceosoft/gitops/tree/main/ai/ollama-router)                                                                                  |
| Argo CD Applications                         | `ollama-router`, `open-webui`, `open-terminal`, `ai-tools`, and `k8sgpt` in [`argocd/apps/`](https://github.com/maceosoft/gitops/tree/main/argocd/apps)                |
| Manifests                                    | Folder [`ai/`](https://github.com/maceosoft/gitops/tree/main/ai) (`ollama-router`, `open-webui`, `open-terminal`, `tools`, `alert-triage`, `k8sgpt`, and `log-report`) |

See also [Add a node to the Ollama cluster](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md), [Logs](https://homelab.maceo.cl/en/operacion/logs/index.md), [Ansible](https://homelab.maceo.cl/en/ansible/index.md), [GitOps](https://homelab.maceo.cl/en/gitops/index.md), [Network](https://homelab.maceo.cl/en/networking/index.md), and [URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md).

# Operations — routine procedures

The [implementation guide](https://homelab.maceo.cl/en/implementacion/index.md) (Phases 0 to 6) builds the HomeLab **once**. This section collects what you do **afterwards**, every time it is needed: adding pieces, maintaining them and rotating credentials.

Each procedure says when to use it, what must be ready beforehand, the steps in order, how to verify and what to check if something fails.

| Procedure                                                                                                       | When                                                                                               | Where it runs                      |
| --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------- |
| [Product URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md)                        | Know where each tool lives, how to sign in and whether it responds                                 | Browser, `kubectl`                 |
| [SSH into the servers](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md)                               | Sign in to `invincible`, `oliver` or `deborah` with your GitHub account                            | Your workstation (`opkssh`, `ssh`) |
| [Monitoring and alerts](https://homelab.maceo.cl/en/operacion/monitoreo/index.md)                               | Check node temperatures and usage, understand a Discord notice, silence an alert                   | Grafana, `kubectl`                 |
| [Add a repo](https://homelab.maceo.cl/en/operacion/agregar-repo/index.md)                                       | Every new project in the `maceosoft` org                                                           | 1Password + PR in `infra`          |
| [Add a node](https://homelab.maceo.cl/en/operacion/agregar-nodo/index.md)                                       | Add a machine to the Incus and K3s cluster                                                         | Ansible (`homelab`)                |
| [Remove or replace a node](https://homelab.maceo.cl/en/operacion/sacar-nodo/index.md)                           | Retire a machine or swap it for another                                                            | `kubectl`, Incus and Ansible       |
| [Add a disk to a node](https://homelab.maceo.cl/en/operacion/agregar-disco/index.md)                            | More storage for Longhorn or Incus                                                                 | Ansible (`homelab`)                |
| [Incus backups and snapshots](https://homelab.maceo.cl/en/operacion/backups-incus/index.md)                     | Schedule backups or recover an instance                                                            | Incus                              |
| [Upgrade K3s](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md)                                    | Review the automatic upgrades or pin a version                                                     | GitOps + Ansible                   |
| [Patch the nodes](https://homelab.maceo.cl/en/operacion/parchear-nodos/index.md)                                | Install Debian security patches when monitoring warns you                                          | Ansible + `apt`                    |
| [Upgrade Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-argocd/index.md)                             | It upgrades itself to the latest version; first upgrade, pinning a version, what to do if it fails | GitOps + Ansible                   |
| [Change the Gateway controller](https://homelab.maceo.cl/en/operacion/cambiar-gateway/index.md)                 | Switch between Kong, Traefik and NGINX; TLS for `*.homelab.local`                                  | GitOps + `kubectl`                 |
| [Upgrade Debian on deborah (12 → 13)](https://homelab.maceo.cl/en/operacion/actualizar-debian-deborah/index.md) | Move the control plane to Debian 13, with a backup and a boot gate                                 | Ansible (`homelab`)                |
| [Calico (CNI) with Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-calico/index.md)                   | How Argo CD manages and upgrades the pod network                                                   | GitOps + `kubectl`                 |
| [Runner cache (ARC)](https://homelab.maceo.cl/en/operacion/cache-arc/index.md)                                  | What is cached between jobs, how to check it, prune it or empty it                                 | `kubectl` + GitOps                 |
| [Disk and DNS health](https://homelab.maceo.cl/en/operacion/salud-de-discos/index.md)                           | What to do when a SMART, eMMC wear or BIND alert fires                                             | Ansible (`homelab`) + nodes        |
| [Incus metrics certificate](https://homelab.maceo.cl/en/operacion/certificado-metricas-incus/index.md)          | What the read-only client certificate used by monitoring is and how to register it in Incus        | `kubectl` + Ansible (`homelab`)    |
| [New version notices (Renovate)](https://homelab.maceo.cl/en/operacion/actualizaciones-de-productos/index.md)   | Which products have a new version, Discord notices and how to handle each PR                       | GitHub + Discord + GitOps          |
| [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md)                         | Periodically, or if a credential leaked                                                            | 1Password + playbooks              |
| [gitops pipeline](https://homelab.maceo.cl/en/operacion/pipeline-gitops/index.md)                               | What each `gitops` PR validates; enable the Argo CD diff (once)                                    | GitHub Actions + 1Password         |

Where the commands run

Unless the procedure says otherwise, Ansible commands run from the `ansible/` folder of the `homelab` repo, and `kubectl` commands use the HomeLab kubeconfig (`~/.kube/homelab-k3s.yaml`).

# Centralized Logs and AI Reports

The logs from the three nodes and all the pods are collected in one place and read from Grafana. Additionally, two automatic reports run on these logs: a **daily** summary of what happened and a **weekly** report with improvement suggestions, published in the `#ia` Discord channel.

## How a Log Travels

```
flowchart LR
    P["Pods<br/>(kubernetes_logs)"] --> V["Vector<br/>DaemonSet, uno por nodo"]
    J["Journal de systemd<br/>(journald) de cada nodo"] --> V
    V -->|"único escritor"| L["VictoriaLogs<br/>retención 30 días"]
    L -->|"lectura"| G["Grafana<br/>Explore + HomeLab / Logs"]
    L --> D["CronJob diario<br/>app=log-digest"]
    M["VictoriaMetrics<br/>métricas del router"] --> D
    D --> W["CronJob semanal<br/>(viernes 09:00)"]
    W --> O["Ollama<br/>vía el router"]
    W --> DC["Discord #ia"]
```

- **Vector** is the collector: it runs on each node as a DaemonSet (one pod per node) and reads two sources: the logs of the Kubernetes containers and the journal of systemd on the node itself (`ssh`, `k3s`, `ollama`, the firewall...). It adds labels (`node`, `unit`, `namespace`, `pod` and the level) and sends them.
- **VictoriaLogs** is the log database. It stores **30 days** in a Longhorn volume and is only accessible inside the cluster: Vector is the only one that writes; Grafana, the daily and weekly reports (`log-report`), and the VMAlert for logs read from it.
- **Grafana** queries VictoriaLogs with a pre-provisioned data source.

The journal of `deborah` is volatile

In `deborah`, `/var/log` is a temporary partition of 200 MB in memory: its journal is lost on node restart. That's why it's important for Vector to export it to VictoriaLogs while the node is running; what reaches VictoriaLogs is kept for the 30 days of retention even if `deborah` restarts.

**This decision in other environments**

Vector collects the logs, VictoriaLogs stores them for 30 days, and Grafana displays them; on top of that, daily and weekly reports with AI summarize what happened. It's low-cost and sufficient to investigate a failure from the last few weeks.

- **Benefits:** with all logs in one place, you can investigate an incident in minutes without entering server by server, and you have the evidence of **who did what and when** that audits require.
- **Retention for compliance:** regulations and contracts may require storing certain logs for months or years, without anyone being able to delete or change them. It's common to separate the hot (queryable) data from the archived (cheap) data.
- **SIEM:** security logs go to a SIEM (system that correlates events to detect attacks), such as [Splunk Enterprise Security](https://www.splunk.com/en_us/products/enterprise-security.html), [Elastic Security](https://www.elastic.co/security) or [Microsoft Sentinel](https://learn.microsoft.com/en-us/azure/sentinel/overview), monitored by a security team.
- **Access and privacy:** who can read what logs is controlled by corporate identity, and personal data is masked before storing.
- **When the HomeLab setup is useful:** when the volume of logs makes a service that charges by GB expensive, or the data cannot leave your infrastructure.

| Cloud              | Equivalent service                                                                                                  |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- |
| GCP                | [Cloud Logging](https://cloud.google.com/logging)                                                                   |
| AWS                | [Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html)        |
| Azure              | [Azure Monitor Logs (Log Analytics)](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/data-platform-logs) |
| Oracle Cloud (OCI) | [OCI Logging](https://docs.oracle.com/en-us/iaas/Content/Logging/home.htm)                                          |

You gain integrated collection with that cloud's services, configurable retention, and scale without worrying about disks. You lose control over the format and queries, pay by GB ingested and stored, and become more tied to that provider.

## What is Collected

| Source                   | Content                                                                                    | Main Labels                             |
| ------------------------ | ------------------------------------------------------------------------------------------ | --------------------------------------- |
| Pods (`kubernetes_logs`) | stdout and stderr of each container, including the log JSON requested by the Ollama router | `namespace`, `pod`, `container`, `node` |
| Journal (`journald`)     | Services on each node: `k3s`, `ollama`, `ssh`, firewall, `auditd`                          | `node`, `unit`                          |
| Reports                  | The summary left by the daily CronJob (`app=log-digest`)                                   | `app`                                   |

The level (error, warning, informational) is inferred from the message when the log does not carry it.

## Querying Logs in Grafana

1. Open Grafana ([URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md)) → **Explore**.
1. Choose the VictoriaLogs data source.
1. Write a query in **LogsQL** (VictoriaLogs' query language). You can combine filters with `AND` and `OR` and chain operations with `|`.

Examples:

```
# Todo lo que dice "error" en el namespace ai, en la última hora
_time:1h namespace:ai error

# Logs de Ollama en un nodo
_time:6h node:deborah unit:ollama.service

# Reinicios por falta de memoria en cualquier nodo
_time:24h "Out of memory" OR "oom-kill"

# Los 10 mensajes que más se repiten, con los números colapsados
_time:24h error | collapse_nums | stats by (_msg) count() as total | sort by (total) desc | limit 10

# El resumen de ayer
app:log-digest _time:2d
```

The exact names of the fields can be seen in Explore, in the list of fields of any log entry. The full language reference is in the [LogsQL documentation](https://docs.victoriametrics.com/victorialogs/logsql/).

## Dashboards

In Grafana, there are 12 HomeLab dashboards; the one for logs is called **HomeLab / Logs** and shows volume by level, errors by service and node, and restarts. It is defined as code in the repo [`gitops`](https://github.com/maceosoft/gitops/tree/main/monitoring/config/dashboards), not edited manually.

The Ollama router also publishes metrics by **model and node** (requests, tokens, latency, live nodes), visible in the **HomeLab / Ollama** dashboard (and queryable in Explore on `ollama_router_*`). See [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/#metrics-per-model-and-node).

## Retention and Space

- The retention is **30 days**. After that, VictoriaLogs deletes the oldest data.
- The logs occupy the Longhorn volume of VictoriaLogs; if it grows more than expected, lower the retention or filter noisy data in Vector. The memory limits of Vector are intentionally small.

## Alerts on Logs

A second VMAlert (of type `vlogs`) evaluates rules written in LogsQL and sends its alerts through Alertmanager, just like metric alerts ([Monitoring and alerts](https://homelab.maceo.cl/en/operacion/monitoreo/index.md)).

## Daily and Weekly Reports in `#ia`

Both are CronJobs in the namespace `monitoring` (the Application `log-report`). The daily analysis **does not use AI** and does not wake up any Ollama node.

| Report     | When                                                          | What It Does                                                                                                                                                                                                                                                                           |
| ---------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Daily**  | Every day at 06:30                                            | Queries VictoriaLogs (errors by level and service, repeated patterns, restarts, lack of memory, starts and stops of `ollama`) and VictoriaMetrics (requests, tokens and latency by model and node, live nodes, Kong, alerts). Saves the summary as a structured log (`app=log-digest`) |
| **Weekly** | Friday at 09:00 (`America/Santiago`), `schedule: "0 9 * * 5"` | Reads the daily summaries of the last 7 days, calculates **rule-based suggestions** and asks Ollama for a summary in Spanish. If Ollama fails, publishes the report based on rules anyway                                                                                              |

The rule-based suggestions are, for example: a model unused for several days, frequent cold starts (raise `KEEP_ALIVE`), a node with memory shortage (lower its `MemoryMax`), new errors, recurring alerts and disk issues. The AI text is marked as **AI suggestion, verify**: never apply changes.

The CronJobs are named `log-digest-daily` (daily, 06:30) and `log-report-weekly`. To run one manually:

```
kubectl -n monitoring create job --from=cronjob/log-digest-daily prueba-manual
kubectl -n monitoring logs job/prueba-manual
```

The manifests are in [`gitops/ai/log-report`](https://github.com/maceosoft/gitops/tree/main/ai/log-report). To change the day or time of the weekly report, edit its `schedule` field in the repo.

### Create the `#ia` Channel and Its Webhook (Once)

The reports are published via a Discord webhook, just like the alerts. The Discord steps are the same as in [Discord and healthchecks.io](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/index.md):

1. In the private category `🖥️ Infraestructura`, create the text channel **`ia`**.

1. *Server Settings → Integrations → Webhooks → New Webhook*; name it `Reportes IA` and choose the channel `ia`.

1. **Copy Webhook URL** and save it in 1Password:

   Paste it in the item `discord-webhooks` of 1Password, in a new field called **`ai`** (next to `critical` and `warnings`).

   The URL is read without echo, so it does not stay in the history (if it does not exist, `op item edit` creates the field):

   ```
   # pega la URL del webhook y Enter
   read -rs DC_AI; op item edit discord-webhooks --vault HomeLab "ai[password]=$DC_AI"; unset DC_AI
   ```

1. Validate and apply:

   ```
   cd ansible
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

The script checks that the URL has the format `https://discord.com/api/webhooks/<id>/<token>` and adds it to the Secret `monitoring/discord-webhooks`. To rotate it, delete the webhook in Discord, create a new one and repeat from step 2.

See also [Rotate Credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md) and [Secrets Management](https://homelab.maceo.cl/en/secrets/index.md).

## Troubleshooting

| Symptom                                   | What to Check                                                                                            |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| No logs from a node appear                | The Vector pod of that node (`kubectl -n monitoring get pods -o wide`) and its logs; the node's journal  |
| Missing old logs of `deborah`             | Its journal is volatile: only what Vector managed to export before the restart is kept                   |
| Grafana does not see the data source      | The VictoriaLogs plugin in Grafana and the address of the VictoriaLogs Service                           |
| The weekly report arrives without AI text | Ollama did not respond: it's normal, the report based on rules is published anyway                       |
| Nothing arrives at `#ia`                  | Missing or invalid `ai` field of `discord-webhooks`; check the CronJob with `kubectl -n monitoring logs` |

# Monitoring and alerts

What is watched in the HomeLab, where to look, and what to do when a notice arrives in Discord. If you are the one **installing** it, go to [4.13 — Monitoring and alerts](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional).

In one line

Every 60 seconds metrics are scraped from the three nodes and the platform. If something crosses a threshold for a while, a message lands in Discord: **🔴 critical** in `alertas-criticas` (reaches your phone) and **🟡 warnings** in `alertas-avisos`. When it resolves, a **✅** arrives.

## General status: what to look at first

If you only check one dashboard, make it **HomeLab / General status**: it summarizes on one screen whether the HomeLab is healthy. It is designed for beginners: each box (*tile*) paints **green** if all is well, **amber** if it deserves a look, and **red** if action is needed, and **each one links to the detail dashboard** for that topic. Read top to bottom:

| Row                    | What it shows                                                                                                                                                    |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Summary**            | The green/amber/red tiles for each area: the first thing you check                                                                                               |
| **Active alerts**      | Alerts firing right now, with their severity                                                                                                                     |
| **Nodes**              | CPU, memory, disk, and temperature per node; pending security patches and whether any node needs a reboot                                                        |
| **Kubernetes**         | Pods with issues, recent restarts, workloads with fewer replicas than requested, and unbound volumes (PVC)                                                       |
| **Storage (Longhorn)** | Status of Longhorn volumes. A **detached** volume (no app using it) shows as `unknown`: this is normal, not a failure                                            |
| **Platform**           | ArgoCD (healthy and synced apps), certificates, web probes, Kong 5xx errors, and Incus                                                                           |
| **AI and logs**        | The Ollama router (live nodes, requests) and log ingestion. Their tiles show **"no data"** in neutral color until you deploy the router and Vector: not an error |

Problem tables (pods, workloads, alerts…) that appear **empty, with the text "No issues"**, are expected: it means there is nothing to attend to. A red tile **links to its detail dashboard**: click it to see what is failing. At the top, the **HomeLab Panels** menu leads to all dashboards tagged `homelab`.

What to do if a tile turns red:

| Red tile                      | Where to continue                                                                                                                                                                                            |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Node down or no metrics       | [SSH in](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md) and the alert row; if the node does not come back, [Remove or replace a node](https://homelab.maceo.cl/en/operacion/sacar-nodo/index.md) |
| Temperature                   | Check what is loading it (`kubectl top pods -A`) and ventilation; see the alert table below                                                                                                                  |
| Disk or DNS                   | [Disk and DNS health](https://homelab.maceo.cl/en/operacion/salud-de-discos/index.md)                                                                                                                        |
| Pending patches or reboot     | [Patch the nodes](https://homelab.maceo.cl/en/operacion/parchear-nodos/index.md)                                                                                                                             |
| Pods or workloads with issues | `kubectl get pods -A`, and the tile's detail dashboard                                                                                                                                                       |
| Longhorn degraded or failed   | Longhorn UI ([URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md)) and [Add a disk](https://homelab.maceo.cl/en/operacion/agregar-disco/index.md) if it fills up                 |
| ArgoCD                        | `kubectl get applications -n argocd` and the ArgoCD UI; to update it, [Update ArgoCD](https://homelab.maceo.cl/en/operacion/actualizar-argocd/index.md)                                                      |
| Certificates                  | `kubectl describe certificate -A`; if it is an expired credential, [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md)                                                   |
| Calico                        | [Calico (CNI) with ArgoCD](https://homelab.maceo.cl/en/operacion/actualizar-calico/index.md)                                                                                                                 |
| Kong 5xx or unreachable webs  | Service and Kong logs, and [URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md)                                                                                                  |
| AI (router without nodes)     | [Local AI assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md) and [Add an Ollama node](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md)                                  |
| No log ingestion              | [Centralized logs and AI reports](https://homelab.maceo.cl/en/operacion/logs/index.md)                                                                                                                       |

This is the **Grafana home dashboard**: it is pinned by the repo `gitops` (`default_home_dashboard_path` in [`values-vm-stack.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/values-vm-stack.yaml)), so it opens on entry. If you want a different home just for your user, pick it in *Profile → Preferences → Home Dashboard*.

## View temperatures and usage

Go to [`https://grafana.homelab.local`](https://grafana.homelab.local) (local network only) and click **Sign in with SSO (GitHub)**: it is the same login as Argo CD, and only members of the `devops` team in the `maceosoft` organization get in, receiving the **Admin** role. Anyone else is rejected. Node and Kubernetes panels come preloaded.

If SSO fails: log in with the local user

The local admin remains active as a fallback, because SSO depends on Dex, which lives in Argo CD, and on the cluster resolving DNS. Log in with user `admin` and the password from item `grafana-admin` in 1Password (on the same screen's form, without pressing the SSO button). If the button fails with *Failed to get token from provider*, check the Grafana log: `kubectl -n monitoring logs deploy/vm-grafana -c grafana | grep oauth`.

For a one-off query use *Explore* with these expressions:

| What                 | Query                                                                                                  |
| -------------------- | ------------------------------------------------------------------------------------------------------ |
| Temperature per node | `max by (nodename) (node_thermal_zone_temp)`                                                           |
| Available memory (%) | `100 * node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes`                                    |
| Root disk used (%)   | `100 * (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"})` |
| CPU usage (%)        | `100 * (1 - avg by (nodename) (rate(node_cpu_seconds_total{mode="idle"}[5m])))`                        |

## Grafana Dashboards

There are **13 dashboards**, own or from the community (those with the tag **`homelab`**), versioned as code in [`monitoring/config/dashboards`](https://github.com/maceosoft/gitops/tree/main/monitoring/config/dashboards) of repo `gitops`: they are not edited manually in Grafana. Those from grafana.com are saved with the revision pinned.

| Dashboard                                 | Purpose                                                                                                                                                                          | Source                    |
| ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| **HomeLab / General Status**              | The home screen: the summary from [above](#general-status-what-to-look-at-first)                                                                                                 | own                       |
| **HomeLab / Temperatures**                | Temperature of `deborah` and the two Intels, now and over 24 hours, with alert thresholds                                                                                        | own                       |
| **HomeLab / Sealed Secrets**              | The Sealed Secrets controller: decryptions, errors, CPU and memory                                                                                                               | own (none on grafana.com) |
| **HomeLab / Logs**                        | Log volume by level, errors by service and by node, restarts ([Logs](https://homelab.maceo.cl/en/operacion/logs/index.md))                                                       | own                       |
| **HomeLab / Ollama**                      | Ollama Router: nodes with heartbeat, in-flight requests, 5xx errors, tokens and latency by model and node ([Local AI](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)) | own                       |
| **ArgoCD**                                | Sync and health of apps, sync operations, repo-server usage                                                                                                                      | grafana.com 14584         |
| **Kong (official)**                       | Requests, HTTP codes and latency by service and route                                                                                                                            | grafana.com 7424          |
| **Longhorn Example v1.1.0**               | Volumes, replicas, usage and disk health                                                                                                                                         | grafana.com 13032         |
| **Calico: Felix**                         | The Calico agent on each node: applied rules, times and dataplane errors                                                                                                         | grafana.com 12175         |
| **Cert-manager-Kubernetes**               | Certificates, their expiration and their status                                                                                                                                  | grafana.com 20842         |
| **Metallb**                               | MetalLB Speakers, L2 advertisements and IP pool usage                                                                                                                            | grafana.com 20162         |
| **Incus**                                 | CPU, memory, disk and network of each project and instance (without the Loki logs row, which is not used)                                                                        | grafana.com 19727         |
| **GitHub Actions ARC - Runner Scale Set** | CI Runners: queued, busy, jobs and times                                                                                                                                         | grafana.com 25015         |

Additionally, the VictoriaMetrics chart brings **41 own dashboards**: *Node Exporter Full* (CPU, memory, disk and network of each node), *Kubernetes / Views* (global, namespaces, pods, nodes), *Alertmanager / Overview* and VictoriaMetrics' own (single node, vmagent, vmalert, operator), among others.

If a panel comes up empty

Check in *Explore* if the base metric exists (for example `argocd_app_info`, `longhorn_volume_robustness`, `kong_http_requests_total` or `certmanager_certificate_expiration_timestamp_seconds`): the scrape may be missing or the labels may not match. The Longhorn one is made for version 1.1 and with the current one some panels come up empty. The AI and logs ones show «no data» until deploying the router and Vector.

The reference temperatures, read from the hardware itself: `deborah` (SoC, system on chip, the chip that integrates CPU and graphics) throttles at 75 °C and again at 85 °C; `invincible` and `oliver` have a manufacturer limit of 100 °C.

## What you might receive

| Alert                                                                                                                                             | Meaning                                                                                                                                       | Action                                                                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 🟡 `TemperaturaAlta`                                                                                                                              | CPU exceeded 80 °C (75 °C on `deborah`) for several minutes                                                                                   | Check what's loading it (`kubectl top pods -A`) and the machine's ventilation                                                                                                                                    |
| 🔴 `TemperaturaCritica`                                                                                                                           | Exceeded 90 °C (85 °C on `deborah`)                                                                                                           | Check ventilation and load immediately; if it keeps rising, shut down heavy workloads (e.g., ARC runners)                                                                                                        |
| 🔴 `NodoNoListo`                                                                                                                                  | A node is not `Ready` in Kubernetes                                                                                                           | Verify it's powered on and on the network, access via `ssh` ([access](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md)) and check `systemctl status k3s` or `k3s-agent`                                |
| 🔴 `ServicioBaseSinReplicas`                                                                                                                      | ArgoCD, Kong, or cert-manager with zero replicas                                                                                              | `kubectl -n <namespace> get pods`; without them there's no GitOps, Gateway, or certificates                                                                                                                      |
| 🟡/🔴 `CertificadoPorVencer`                                                                                                                      | A certificate expires in less than 14 / 3 days                                                                                                | `kubectl describe certificate -A`; cert-manager should renew it automatically, so if it doesn't, investigate why                                                                                                 |
| 🟡 `CertificadoNoListo`                                                                                                                           | A certificate failed to be issued                                                                                                             | `kubectl describe certificate -A` and cert-manager logs                                                                                                                                                          |
| 🔴 `LonghornVolumenDegradado`                                                                                                                     | A volume lost a replica: only one copy remains                                                                                                | Open the Longhorn UI (see [URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md)) and verify both disks are healthy                                                                    |
| 🔴 `LonghornVolumenFallido`                                                                                                                       | A volume with no usable replicas                                                                                                              | Urgent: the app using it isn't working; check Longhorn and the nodes                                                                                                                                             |
| 🟡 `LonghornDiscoLlenandose`                                                                                                                      | Longhorn disk exceeded 80 %                                                                                                                   | Delete unused volumes or snapshots, or add a disk ([add disk](https://homelab.maceo.cl/en/operacion/agregar-disco/index.md))                                                                                     |
| 🟡 `ArgoCDAppNoSana` / `Desincronizada`                                                                                                           | An app has been out of `Healthy` / `Synced` for more than 15 / 30 min                                                                         | `kubectl get applications -n argocd` and the ArgoCD UI                                                                                                                                                           |
| 🟡 `NodosSinMetricas`                                                                                                                             | Some node isn't sending metrics                                                                                                               | `kubectl -n monitoring get pods -o wide` (missing a `node-exporter`?)                                                                                                                                            |
| 🟡 `SensoresTemperaturaSinDatos`                                                                                                                  | Metrics are received from a node, but no temperature                                                                                          | Temperature alerts for that node won't work: check `node-exporter`                                                                                                                                               |
| 🔴 `WebInaccesible`                                                                                                                               | ArgoCD, Dex, or Grafana don't respond `200` via their URL (what the user sees, even if pods are healthy)                                      | Open the URL; `kubectl -n kong get pods`, the Gateway, and the route (`kubectl get httproute -A`)                                                                                                                |
| 🟡 `RedirectHttpRoto` / `DnsInternoFalla` / `CertificadoWebPorVencer`                                                                             | The `http://` doesn't redirect to `https://`; a name doesn't resolve inside the cluster (CoreDNS or BIND); a Gateway certificate expires soon | Check the Gateway, `deborah`'s BIND, and certificates (`kubectl describe certificate -A`)                                                                                                                        |
| 🔴/🟡 `KongErrores5xx`                                                                                                                            | More than 5 % of a service's requests return 5xx (critical if ArgoCD, Grafana, or Dex and exceeds 25 %)                                       | Check the service logs and Kong                                                                                                                                                                                  |
| 🟡 `KongConfiguracionNoSeAplica` / `KongRecursosInvalidos`                                                                                        | Kong controller fails to apply a route or there are broken resources                                                                          | `kubectl -n kong logs deploy/kong-controller` and the affected `HTTPRoute`                                                                                                                                       |
| 🔴 `CalicoNodoCaido`                                                                                                                              | A node lacks the Calico pod: its pods are left without network                                                                                | `kubectl -n calico-system get pods -o wide`                                                                                                                                                                      |
| 🟡 Calico, MetalLB, Sealed Secrets, ARC, and ArgoCD (`CalicoFelix…`, `MetalLB…`, `SealedSecretsError…`, `ARC…`, `ArgoCDApplicationSetConErrores`) | A platform component fails, has no replicas, or has no metrics                                                                                | The alert name tells which one; `kubectl -n <namespace> get pods` and its logs                                                                                                                                   |
| 🔴/🟡 Disks and BIND (`DiscoSmartFalla`, `EmmcDesgastada`, `BindCaido`…)                                                                          | SMART failure, worn SSD or eMMC, or LAN DNS not responding                                                                                    | [Disk and DNS health](https://homelab.maceo.cl/en/operacion/salud-de-discos/index.md)                                                                                                                            |
| 🟡 `ParchesDeSeguridadPendientes`                                                                                                                 | A node has gone more than 3 days with uninstalled security patches                                                                            | Follow [Patch the nodes](https://homelab.maceo.cl/en/operacion/parchear-nodos/index.md)                                                                                                                          |
| 🟡 `ReinicioPendiente`                                                                                                                            | An installed update requires a node reboot, pending for more than 7 days                                                                      | Reboot the node using the same procedure                                                                                                                                                                         |
| 🟡 `ListasDePaquetesViejas`                                                                                                                       | apt lists haven't been updated for more than 7 days: patch notices may not be real                                                            | Check `systemctl status node-apt-metrics.timer` on the node                                                                                                                                                      |
| 🟡 `SinDatosDePaquetes`                                                                                                                           | `node-exporter` responds but doesn't publish pending packages                                                                                 | Missing `playbook-apt-metrics.yml` apply on that node                                                                                                                                                            |
| 🟡 `KongLatenciaUpstreamAlta` / `KongLatenciaPropiaAlta` / `KongControladorConErrores`                                                            | A service responds slowly, Kong adds too much latency, or its controller accumulates errors                                                   | *Kong (official)* dashboard and `kubectl -n kong logs deploy/kong-controller`                                                                                                                                    |
| 🟡 `PlataformaMetricasCaidas` / `PlataformaComponenteSinReplicas`                                                                                 | A platform component stopped publishing metrics or ran out of replicas                                                                        | `kubectl -n <namespace> get pods` of the component named in the alert                                                                                                                                            |
| 🟡 `ActualizacionK3sFallida`                                                                                                                      | A K3s upgrade job from SUC (*System Upgrade Controller*) failed                                                                               | [Upgrade K3s](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md)                                                                                                                                     |
| 🟡 `SondasSinDatos`                                                                                                                               | No metrics from external probes: the web could be broken without any warning                                                                  | `kubectl -n monitoring get pods` (blackbox-exporter)                                                                                                                                                             |
| 🟡 `RafagaDeErroresEnLogs`                                                                                                                        | A namespace logged more than 200 errors in 5 minutes                                                                                          | *HomeLab / Logs* dashboard; example query in [Logs](https://homelab.maceo.cl/en/operacion/logs/index.md)                                                                                                         |
| 🟡 `OOMKillerEnNodo`                                                                                                                              | Kernel killed processes due to OOM                                                                                                            | Node kernel logs ([Logs](https://homelab.maceo.cl/en/operacion/logs/index.md)) and what's consuming memory (`kubectl top pods -A`); if it's Ollama, lower its `MemoryMax`                                        |
| 🟡 `LogsSinIngestaEnNodo` / `VectorCaido`                                                                                                         | Vector isn't shipping logs from a node, or its metrics aren't being scraped                                                                   | `kubectl -n monitoring get pods -o wide` (that node's Vector pod) and [Logs](https://homelab.maceo.cl/en/operacion/logs/index.md)                                                                                |
| 🟡 `OllamaRouterCaido` / `OllamaRouterSinNodos`                                                                                                   | Ollama router doesn't respond, or has no nodes with heartbeat                                                                                 | `kubectl -n ai get pods` and [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md); for nodes, [Add an Ollama node](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md) |

You also get the standard Kubernetes and node alerts that come with the chart (disk filling up, high memory, clock not synced via NTP or Network Time Protocol, pods restarting endlessly or with missing replicas). Their description appears in the message itself. After the first deployment, the full list is in Grafana's *Alerting* tab.

Messages include the node name and a summary, **no IP addresses**, because the channel may be viewed by more people. A critical alert repeats every 4 hours while active; a warning, every 12 hours.

## What reaches the deployments channel

Besides alerts, ArgoCD writes to your `despliegues` channel every time an app finishes a successful sync ([4.14](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#414-argocd-notifications-to-discord-optional)):

- **🚀 `<app>` deployed**: with the revision, status, and images the app uses. If ArgoCD, Longhorn, or another product updated itself, the new version appears in those images (for example `quay.io/argoproj/argocd:v3.5.6`).
- **🟡 `<app>`: sync failed**: arrives at `alertas-avisos` with the error message. Check `kubectl describe application -n argocd <app>`.

This is a **record of what happened**, not something to act on: no need to reply to a 🚀.

## Silence an alert

To stop an alert from firing for a while (e.g., during maintenance), create a silence in Alertmanager:

```
kubectl -n monitoring port-forward svc/vmalertmanager-vm-victoria-metrics-k8s-stack 9093:9093
```

Open [`http://localhost:9093`](http://localhost:9093), click **Silences → New Silence**, enter `alertname` and the alert name (plus `nodename` if it's for a single node), and choose the duration. If the Service isn't named that, look up its name with `kubectl -n monitoring get svc | grep alertmanager`. A silence must never be left without an end date.

## If alerts stop arriving

Monitoring lives inside the cluster, so a severe failure could leave it mute. That's why there's a **dead man's switch**: every 5 minutes Alertmanager sends a ping to [healthchecks.io](https://healthchecks.io/), and if it stops arriving, that tool notifies via Discord in `alertas-criticas` that the check `homelab-monitoreo` (or whatever name you gave it) is down. When you receive it:

1. Check if the cluster responds: `kubectl get nodes`.
1. If it responds, check the monitoring pods: `kubectl -n monitoring get pods`.
1. If it doesn't respond, check the control plane node (currently `deborah`, which is also the DNS) and emergency key access, because Dex SSO also lives in the cluster.

## What still isn't covered

- What each login in Dex does (it only publishes gRPC telemetry), failed ArgoCD notification deliveries, and the status of K3s upgrade plans.
- OpenEBS, which doesn't publish useful metrics; only replica alerts cover it.
- Silencing alerts or checking status from Discord (currently done with `kubectl` and the Alertmanager UI).

# Move the K3s Control Plane to Another Node (deborah → oliver)

Procedure to transfer the **control plane** (the K3s server that stores the cluster state and serves the API) from `deborah` to `oliver`, and leave `deborah` as a **worker**. This is done during a **maintenance window**, with one person at the keyboard: the playbook pauses at several **checkpoint points** for you to confirm each stage.

Not planned

The control plane stays on `deborah`. This procedure is documented in case it ever needs to change nodes.

## Why It Could Be Moved

`deborah` is the machine with the most memory (31 GB), but it also runs Ollama, serves as the LAN (local network) DNS, and has a slow eMMC (onboard flash memory). A loaded model competes with the control plane for memory. `oliver` is a lightweight node (label `workload-tier=light`) with a stable load, and `deborah` is freed up for models and heavy loads.

The cluster is not rebuilt

Longhorn (the storage) stores its metadata within the cluster state: if it is lost, the volumes become orphaned. That's why this procedure **transfers the existing state**; it does not reinstall from scratch. Without a verified backup, do not start.

## How the Transfer Is Done

Today, K3s runs with a single server and SQLite (a file). The official K3s documentation does not describe how to move that file to another node (and the token includes the CA hash, the cluster's certificate authority). What is documented is converting the data store to **embedded etcd** (Kubernetes' distributed database, integrated into K3s), adding another server with the same token, and ending up with a single server. The playbook [`playbook-k3s-move-controlplane.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-k3s-move-controlplane.yml) does exactly that, in stages, each with its label (`tag`):

| Stage | Label       | What It Does                                                                                                                                                        | API              |
| ----- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| 1     | `preflight` | Checks: nodes `Ready`, Longhorn without degraded volumes, Argo CD synchronized                                                                                      | up               |
| 2     | `backup`    | Backup **encrypted** of the state, saved outside the node (stops K3s for a few minutes at the origin) and generates a different *agent-token* from the server token | down for a while |
| 3     | `convert`   | Converts `deborah` to etcd (`--cluster-init`) with security flags and restarts agents to the name `k3s.homelab.local`                                               | down for a while |
| 4     | `join`      | Drains `oliver`, removes its agent, and joins it as a second server                                                                                                 | up               |
| 5     | `dns`       | `k3s.homelab.local` points to `oliver`                                                                                                                              | up               |
| 6     | `retire`    | Leaves `oliver` as the sole etcd member and `deborah` as an agent                                                                                                   | down for a while |
| 7     | `verify`    | Final verification, summary of manual steps and secure deletion (`shred`) of the clear backup and `/root/k3s-move-*` on the nodes                                   | up               |

The rollback (`--tags rollback`) **never runs alone**.

During the window, there are two etcd members

Between stages 4 and 6, the cluster has two etcd members (`deborah` and `oliver`). With two members, there is no fault tolerance: if one goes down, the other loses quorum and the API stops responding. Do not touch the nodes outside the playbook during this interval.

## Prerequisites

- A window in which the API can be unresponsive for several minutes during stages 2, 3, and 6: Argo CD does not synchronize and `kubectl` does not respond; running loads continue, except for the `deborah` pods, which are drained in stage 6.
- SSH access to `deborah` and `oliver` ([SSH in](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md)) and physical access to `deborah`.
- Healthy Longhorn and synchronized Argo CD (the `preflight` checks this; you can skip Argo CD with `k3s_move_allow_unsynced`, but it is not recommended).
- `playbook-bind-dns.yml` already applied, with the registry `k3s` in the zone `homelab.local`: the variable `k3s_api_endpoint` (`k3s.homelab.local`) replaces the fixed IP of the control plane in the agents, the kubeconfig, and the SAN (alternative names of the API certificate).
- The HomeLab CA on the nodes (role `opkssh`), because new servers rely on Dex for user login.
- An encryption method for the backup (contains the token and cluster certificates): a public key of `age` (`k3s_move_backup_age_recipient`) or a password for `gpg` (`k3s_move_backup_passphrase`, better via environment variable). Without either, the playbook aborts, except if forced with `-e k3s_move_backup_unencrypted=true` (not recommended).
- `oliver` with about 5 GB of free memory. If it has heavy loads, move them beforehand.

## Step by Step

From `ansible/`:

1. **Dry run, no changes made:**

   ```
   ansible-playbook -i inventory.ini playbook-k3s-move-controlplane.yml --check --diff
   ```

1. **Actual execution.** With pauses enabled by default, the playbook asks for confirmation before the backup, before converting to etcd, before joining `oliver`, before restarting DNS and before removing the source:

   ```
   ansible-playbook -i inventory.ini playbook-k3s-move-controlplane.yml
   ```

   Useful variables: `k3s_move_target` (the destination, default `oliver`), `k3s_move_backup_dir` (where the backup is saved, default `~/k3s-move-backups` on your machine), `k3s_move_backup_age_recipient` or `k3s_move_backup_passphrase` (encryption), `k3s_move_agent_token` (if you want to set the *agent-token* instead of generating it), `k3s_move_drain_source`, `k3s_move_rotate_secrets`, `k3s_move_allow_unsynced` and `k3s_move_confirm=false` for non-interactive (not recommended).

1. **Pause points.**

   - After the backup: open the file and check its checksum before proceeding.
   - Before converting: this is the step that **cannot be undone without rollback**.
   - After joining `oliver`: `kubectl get nodes` should show two servers.
   - After restarting DNS: `dig +short k3s.homelab.local @192.168.20.5` should return the IP of `oliver`.

If something doesn't convince you, stop

Each pause allows aborting with `Ctrl+C`. If the playbook is interrupted after stage 3, do not resume with `--tags` loose: stages 4 and 6 need the token obtained in stage 3, within the same execution.

Upon completion, the playbook **does not modify the inventory**: you do it as explained below. The agents stop using the server token and switch to their own **agent-token**, which is stored in `<k3s_move_backup_dir>/agent-token`.

With the move, the new server's hardening measures are also applied: flags `oidc-*` towards Dex (user login, with `--oidc-username-prefix=oidc:` and `--oidc-groups-prefix=oidc:`), `--secrets-encryption` (encryption of Secrets at rest) and the K3s installer verified by checksum instead of `curl | sh`. The admin certificate is kept only for emergencies. See [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md).

## Verification

| What     | How                                                                                                                     |
| -------- | ----------------------------------------------------------------------------------------------------------------------- |
| Nodes    | `kubectl get nodes -o wide`: `oliver` with role `control-plane`, and the three in `Ready`                               |
| Argo CD  | `kubectl -n argocd get applications`: all `Synced` and `Healthy`                                                        |
| Longhorn | UI or `kubectl -n longhorn-system get volumes.longhorn.io`: no degraded volumes                                         |
| Kong     | `curl -kI https://argocd.homelab.local` responds                                                                        |
| DNS      | `dig +short k3s.homelab.local @192.168.20.5` returns the IP of `oliver`                                                 |
| Pods     | `kubectl get pods -A` without stuck pods (`Pending`, `CrashLoopBackOff`); those that lived in `deborah` are rescheduled |

## What Changes in the Inventory After

The playbook does not touch these files; edit them manually after finishing (and commit your changes):

- [`inventory.ini`](https://github.com/maceosoft/homelab/blob/main/ansible/inventory.ini): `deborah` changes from `[k3s_server]` to `[k3s_agent]` and `oliver` changes from `[k3s_agent]` to `[k3s_server]` (and its `k3s_role`).
- `group_vars/all.yml`: `k3s_api_ip` changes to `192.168.20.7` (the IP of `oliver`) and `k3s_move_done` changes to `true`.
- `host_vars/deborah.yml` and `host_vars/oliver.yml`: K3s labels and options follow `k3s_move_done`; `deborah` keeps `workload-tier=heavy` and `oliver` remains `light`.
- Reapply `playbook-bind-dns.yml` to align the zone with `k3s_api_ip`.
- **SUC Plans** (*System Upgrade Controller*, the automatic K3s updater; [`k3s-upgrade/`](https://github.com/maceosoft/gitops/tree/main/k3s-upgrade)): select the control plane by the `node-role.kubernetes.io/control-plane` label, so they do not change and now update first to `oliver`.

Other guides that discuss the control plane: [Remove or Replace a Node](https://homelab.maceo.cl/en/operacion/sacar-nodo/index.md), [Add a Node](https://homelab.maceo.cl/en/operacion/agregar-nodo/index.md), [Update K3s](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md) and the [Phase 3](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md).

## Rollback

Restore the original SQLite in `deborah` from the backup and return `oliver` to agent:

```
  # o -e k3s_move_backup_passphrase=...
ansible-playbook -i inventory.ini playbook-k3s-move-controlplane.yml --tags rollback \
  -e k3s_move_backup_stamp=<AAAAMMDDTHHMMSS> \
  -e k3s_move_backup_age_identity=<archivo de identidad age>
```

The mark is the date part of the backup file name in the backup directory. Rollback is four steps, in this order: **1)** DNS points back to `deborah` and waits about 300 seconds for propagation (if not, the agent reinstalled in `oliver` would try to join itself by `k3s.homelab.local`); **2)** the source is restored; **3)** `oliver` returns to agent; **4)** the other agents recover the original token. **All changes to the cluster after the backup are lost**: decide carefully before reverting. If you had already edited the inventory, revert it to its previous state.

## Manual Steps After Completion

1. **Kubeconfig.** Regenerate the one on your machine, which pointed to the old IP:

   ```
   ansible-playbook -i inventory.ini playbook-k3s.yml --tags kubeconfig
   ```

1. **Save the migration secrets in 1Password.** The *agent-token* (`<k3s_move_backup_dir>/agent-token`) and the key or password with which you encrypted the backup. The unencrypted backup and the `/root/k3s-move-*` files of the nodes are deleted automatically at the end (`shred`); the only thing left is the encrypted backup on your machine, which is what is used for rollback.

1. **Check Argo CD, Longhorn, and Kong** with the [Verification](#verification) table.

1. **Login permissions with Dex.** The repo `gitops` brings the ClusterRoleBinding [`oidc-devops-view`](https://github.com/maceosoft/gitops/blob/main/security/rbac/oidc-devops.yaml) (Application `security-rbac`, already active), which gives the ClusterRole `view` (read-only) to the group `oidc:maceosoft:devops`. It is harmless until there are OIDC tokens.

   Why read-only and not `cluster-admin`

   Logging in with Dex should not give full cluster control by default: state changes go through Git and Argo CD, and the admin certificate is kept for emergencies. If a user needs to write in a namespace, extend with a `RoleBinding` that binds the ClusterRole `edit` to the same group **only in that namespace**, instead of extending the ClusterRoleBinding.

1. **Optional, risk with Calico.** Check that a common service account cannot create global policies (see [Optional Checks](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/#optional-checks)).

### Logging in with Dex (kubelogin)

With the flags `oidc-*` enabled, you can log in to the API using your GitHub account instead of the admin certificate. Install the `kubelogin` plugin (`kubectl oidc-login`) and request a token:

```
kubectl oidc-login get-token \
  --oidc-issuer-url=https://argocd.homelab.local/api/dex \
  --oidc-client-id=kubernetes \
  --oidc-extra-scope=email,groups
```

A browser window opens; the Dex client `kubernetes` is public (PKCE) and receives the response at `http://localhost:8000` (or, as a fallback, `18000`): those ports must be free on your machine. To use it always, configure a user in your kubeconfig with `exec` pointing to that command.

## Risks

- It is a **single** control plane (no HA, high availability): if `oliver` goes down, the cluster loses its API until it is restored. Running workloads continue to run.
- Switching from SQLite to etcd is a data store change: that's why backup and rollback are part of the procedure, not an extra.

See also [Removing or Replacing a Node](https://homelab.maceo.cl/en/operacion/sacar-nodo/index.md), [Updating K3s](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md), and the [Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md).

# Patch the nodes

Debian patches **are not installed automatically**: monitoring only warns. Each node publishes how many patches are pending (and how many are security patches) and whether it needs a reboot; when something has gone unattended for too long, an alert arrives on Discord (`ParchesDeSeguridadPendientes`, `ReinicioPendiente`; see [Monitoring and alerts](https://homelab.maceo.cl/en/operacion/monitoreo/#what-you-might-receive)). This procedure is what you do when you receive it.

## Once: enable the notice

Apply the role that publishes the metrics on the three nodes ([`playbook-apt-metrics.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-apt-metrics.yml)) from `ansible/`:

```
ansible-playbook playbook-apt-metrics.yml
```

It installs a timer that refreshes the apt lists every 6 hours and writes the result where `node-exporter` reads it. It does not install any package. With monitoring already deployed, `SinDatosDePaquetes` reminds you of the nodes where it is missing.

## See what is pending

In Grafana, the *Node Exporter Full* dashboard does not show it; query `node_apt_upgrades_pending` in *Explore* (`origin` label: `security` or `other`). Or directly on the node:

```
sudo apt-get update && apt list --upgradable
```

## Recommended order

One node at a time, with `deborah` **last**: it is the LAN DNS server (BIND) and also the K3s control plane, so rebooting it cuts the DNS and the cluster API for a few minutes.

1. `oliver`
1. `invincible`
1. `deborah`

## With Ansible (recommended)

[`playbook-parchear.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-parchear.yml) (role [`node_patch`](https://github.com/maceosoft/homelab/tree/main/ansible/roles/node_patch)) runs the whole procedure in that order, one node at a time. It needs `kubectl` on your machine with `KUBECONFIG` pointing to the cluster (by default `~/.kube/homelab-k3s.yaml`) and a terminal where you can answer the confirmations. From `ansible/`:

First, a dry run. It does not drain or reboot: it shows which packages would be upgraded on each node and whether a reboot would be needed.

```
ansible-playbook -i inventory.ini playbook-parchear.yml --check
```

Then, node by node, or all in a row:

```
ansible-playbook -i inventory.ini playbook-parchear.yml -l oliver
ansible-playbook -i inventory.ini playbook-parchear.yml -l invincible
ansible-playbook -i inventory.ini playbook-parchear.yml -l deborah
```

```
ansible-playbook -i inventory.ini playbook-parchear.yml
```

On each node:

1. **Pre-check** (tag `precheck`): all nodes `Ready` and no Longhorn volume `degraded` or `faulted`. Otherwise it stops without touching anything. Disconnected volumes (`detached`) show up as `unknown` and do not count as a failure.
1. **Upgrade** (tag `upgrade`): lists the pending packages and runs `apt-get update` and `dist-upgrade`.
1. **Reboot** (tag `reboot`), only if `/var/run/reboot-required` exists: drains the node from your machine, reboots it, waits for it to come back `Ready` and runs `uncordon`. If no reboot is needed, it does not drain. If something fails with the node drained, it tries the `uncordon` and stops with a message.
1. **Post-check** (tag `postcheck`): waits, up to 15 minutes, for the node to be `Ready` and Longhorn to have no degraded volumes before moving on to the next one.

The order is set by the `node_patch_order` variable, not by the inventory. The K3s control plane and `deborah` are always moved to the end, even if written earlier, and **ask for confirmation twice**: before upgrading (if `bind9` is upgraded, `named` restarts and the DNS goes down for a few seconds) and before rebooting. On any failure, it does not continue with the next node.

Only part of it:

```
ansible-playbook -i inventory.ini playbook-parchear.yml --tags precheck
ansible-playbook -i inventory.ini playbook-parchear.yml -l oliver --tags upgrade,postcheck
```

Useful variables (with `-e`): `node_patch_order`, `node_patch_autoremove` (removes unused dependencies; `false` by default), `node_patch_drain_timeout`, `node_patch_reboot_timeout`, `node_patch_postcheck_minutes`.

## By hand

If you cannot use Ansible, for each node:

1. Check that the cluster is healthy: `kubectl get nodes` (all `Ready`) and that the Longhorn volumes are not degraded (Longhorn UI, see [URLs and access](https://homelab.maceo.cl/en/operacion/urls-y-accesos/index.md)). If a volume already has a single replica, do not reboot a node that holds the other one.
1. SSH in ([access](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md)) and install: `sudo apt-get update && sudo apt-get dist-upgrade`.
1. If a reboot is needed (`/var/run/reboot-required` exists, or the `ReinicioPendiente` alert): `kubectl drain <nodo> --ignore-daemonsets --delete-emptydir-data`, then `sudo reboot`.
1. When it comes back, `kubectl uncordon <nodo>` and wait for the node to be `Ready` and the Longhorn volumes healthy before moving on to the next one.

After a while (up to 6 hours, or run `sudo systemctl start node-apt-metrics.service`), that node's alerts resolve themselves.

# GitOps Pipeline

Each change in [`maceosoft/gitops`](https://github.com/maceosoft/gitops) is validated **before** Argo CD applies it to the cluster. This is done by the reusable workflow [`gitops.yml`](https://github.com/maceosoft/core-pipelines/blob/main/.github/workflows/gitops.yml) from `core-pipelines`, called from `gitops/.github/workflows/validate.yml`. It runs on the runner `arc-runners` of the HomeLab (all workflows of `core-pipelines` run on ARC). The tools it installs (helm, kustomize, kubeconform, gitleaks, argocd) are stored in the [runner cache](https://homelab.maceo.cl/en/operacion/cache-arc/index.md), verified by `sha256`.

## What It Validates

| Job           | When                  | What It Checks                                                                                                                                                       | Fails If…                                                                                                                                       |
| ------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `render`      | PR and push to `main` | yamllint; `kustomize build`; `helm template` of each Application with chart, with its values; the list of the ApplicationSet `bootstrap/root-appset.yaml`            | Invalid YAML, a chart or kustomize that does not render, an app in the list without a file (or vice versa)                                      |
| `schemas`     | PR and push to `main` | Each manifest, and what it renders, against the Kubernetes schema and CRD (Custom Resource Definition): Application, MetalLB, SUC (System Upgrade Controller) plans… | A field that does not exist or is misplaced, an incorrect type, a resource without a schema                                                     |
| `security`    | PR and push to `main` | gitleaks on the commits; repo rules                                                                                                                                  | A secret in the code; an Application of `gitops` with another `repoURL` or branch; an Application without `sync-wave`; a `Secret` in plain text |
| `argocd-diff` | Only on PR            | What Argo CD would change in the cluster with the PR commit, app by app                                                                                              | Only if Argo CD cannot calculate the diff. Having changes is not an error: that is what is being reviewed                                       |

The diff result appears in the **job summary** (*Actions → the run → Summary*): each app with "no changes" or a collapsible block with the diff. Multi-source apps (external charts with values from `gitops`, like `arc-runners`) are listed but not compared: their values are reviewed in the Argo CD UI.

## Enable the Cluster Diff (Once)

The diff uses a **read-only** Argo CD account, `ci`, which is already declared in `gitops/argocd/config` (`accounts.ci: apiKey` and `g, ci, role:readonly`). The token needs to be generated and brought to the runner. This must be done with Argo CD running and `argocd` synchronized. Until it exists, the job skips with a warning and the other jobs run as usual.

1. Generate the token with the Argo CD CLI, logged in as admin:

   ```
   argocd login argocd.homelab.local --grpc-web
   argocd account generate-token --account ci
   ```

1. Save the token in 1Password, vault `HomeLab`, item **`argocd-ci`**:

   In 1Password, vault `HomeLab`, create the item **`argocd-ci`** (Secure Note) with a field **`token`** (Password) and paste the token ([how to create an item](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields)).

   A single command generates the token and saves it, without showing it on screen (you need the login of `argocd` from step 1 and 1Password unlocked; if you have multiple 1Password accounts, export `OP_ACCOUNT` first):

   ```
   op item create --category "Secure Note" --vault HomeLab --title argocd-ci \
     "token[password]=$(argocd account generate-token --account ci)"
   ```

   To rotate it: `op item edit argocd-ci "token[password]=$(argocd account generate-token --account ci)"`.

1. Bring it to the cluster (create the Secret `arc-runners/argocd-ci`), from `ansible/` in `homelab`:

   ```
   # tu cuenta
   export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
   # tiene que mostrar: argocd-ci (opcional) ✓ token
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

New runners receive it as `ARGOCD_AUTH_TOKEN` (the scale set mounts it with `optional: true`). To rotate it, generate another token, replace the field, and rerun the playbook.

## Require the Pipeline Before Merging (Optional)

`gitops` is not managed by OpenTofu, so the protection of `main` is configured manually:

*gitops → Settings → Branches → main → Require status checks* and choose the checks `validate / render`, `validate / schemas` and `validate / security`.

With `gh auth login` done as the owner of the org:

```
gh api -X PUT repos/maceosoft/gitops/branches/main/protection --input - <<'JSON'
{
  "required_status_checks": {"strict": false,
    "contexts": ["validate / render", "validate / schemas", "validate / security"]},
  "enforce_admins": false,
  "required_pull_request_reviews": null,
  "restrictions": null
}
JSON
```

To see how it turned out: `gh api repos/maceosoft/gitops/branches/main/protection --jq '.required_status_checks.contexts'`. If the response is `403` (*Upgrade to GitHub Pro or make this repository public*), the organization plan does not allow protecting branches in private repos: in that case, only discipline of always opening PRs remains. PR checks are viewed with `gh pr checks <número> --repo maceosoft/gitops`.

## Adjustments

Inputs can be passed from the caller (`gitops/.github/workflows/validate.yml`):

| Input             | Default                                                                   | Purpose                                                        |
| ----------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------- |
| `argocd-diff`     | `true`                                                                    | Disable the diff                                               |
| `allowed-secrets` | `argocd/apps/arc-helm-repo.yaml argocd/apps/nginx-gateway-helm-repo.yaml` | Paths that can have a `Secret` in plain text (space-separated) |
| `no-wave-apps`    | `vcluster-platform capn-demo canal cilium`                                | Applications that can lack `sync-wave`                         |

In the repo `gitops` itself: `.yamllint.yaml` (yamllint rules) and `.gitleaks.toml` (gitleaks exceptions; today, the templates `*.example.yaml`).

## If It Fails

| Symptom                                           | To Check                                                                                                                          |
| ------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `render`: `helm … not a valid chart repository`   | The Helm repo URL for that Application changed or no longer exists (it happened to `sealed-secrets`)                              |
| `render`: app in the list without a file          | A misspelled name in `bootstrap/root-appset.yaml`, or a file of `argocd/apps/` that needs to be added (commented out) to the list |
| `schemas`: `additionalProperties '…' not allowed` | A field misspelled or in the wrong place in that manifest                                                                         |
| `schemas`: `could not find schema`                | A CRD not in the catalog; if intentional, add it to the `-skip` of the job                                                        |
| `security`: gitleaks found something              | If it's a real secret: rotate it and remove it from history. If it's an example with placeholders, name it `*.example.yaml`       |
| `argocd-diff` omitted                             | The token is missing (section *Enable the Diff*)                                                                                  |
| `argocd-diff`: error comparing                    | The account `ci` does not exist or does not have `role:readonly` (sync `argocd`), or the token expired                            |

# Rotate credentials

Replace a credential with a new one: periodically, or immediately if you suspect it was leaked. In all cases the new credential is saved first in 1Password (vault `HomeLab`) and then brought to the cluster.

Before you start

The 1Password app unlocked, with *Integrate with other apps*, and the account in the environment variable:

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
```

Ansible commands are run from `ansible/` in `homelab`.

## Summary

| Credential                                                                                       | Item in 1Password                                                                                         | Applied with                                                                                                                                   |
| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| Private key of a GitHub App (`maceosoft-terraform`, `maceosoft-arc-runners`, `maceosoft-argocd`) | the item named after the App                                                                              | `playbook-platform-secrets.yml`                                                                                                                |
| FTP credentials of a repo                                                                        | `ftp-<repo>`                                                                                              | `playbook-platform-secrets.yml` + `infra` pipeline                                                                                             |
| Dex client secret (GitHub login)                                                                 | `maceosoft-dex`                                                                                           | `playbook-dex-oauth-secrets.yml`                                                                                                               |
| Discord webhooks, healthchecks.io ping URL, and Grafana password (monitoring)                    | `discord-webhooks`, `healthchecks-ping`, `grafana-admin`                                                  | `playbook-platform-secrets.yml` ([see below](#monitoring-discord-webhooks-healthchecksio-ping-and-grafana))                                    |
| Open WebUI session key (AI assistant)                                                            | `open-webui` (field `WEBUI_SECRET_KEY`)                                                                   | `playbook-platform-secrets.yml` ([how to generate it](https://homelab.maceo.cl/en/operacion/ia-ollama/#create-the-open-webui-key-once-only))   |
| Ollama router tokens (one per client and the node registration token)                            | `ollama-router` (fields `registry_token`, `open-webui`, `k8sgpt`, `reports`, `alert-triage`, `translate`) | `playbook-platform-secrets.yml` ([see below](#ollama-and-internal-services-tokens))                                                            |
| `alert-triage` webhook token and `mcpo` API key                                                  | `alert-triage-webhook` (field `token`), `mcp-api-key` (field `key`)                                       | `playbook-platform-secrets.yml` ([see below](#ollama-and-internal-services-tokens))                                                            |
| Open Terminal API key                                                                            | `open-terminal` (field `api_key`)                                                                         | `playbook-platform-secrets.yml` ([see below](#ollama-and-internal-services-tokens))                                                            |
| `#ia` channel webhook (AI reports)                                                               | `discord-webhooks` (field `ai`)                                                                           | `playbook-platform-secrets.yml` ([create the channel](https://homelab.maceo.cl/en/operacion/logs/#create-the-ia-channel-and-its-webhook-once)) |
| Node root password                                                                               | `root@<nodo>` (created by the playbook)                                                                   | `playbook-rotate-root-passwords.yml`                                                                                                           |
| Sealed Secrets key                                                                               | `sealed-secrets`                                                                                          | `playbook-platform-secrets.yml` ([note: see below](#sealed-secrets-key))                                                                       |
| Argo CD token from the `gitops` pipeline (account `ci`)                                          | `argocd-ci`                                                                                               | `playbook-platform-secrets.yml` ([how to generate it](https://homelab.maceo.cl/en/operacion/pipeline-gitops/#enable-the-cluster-diff-once))    |

## Private key of a GitHub App

1. In GitHub: *maceosoft → Settings → Developer settings → GitHub Apps →* the App → *Private keys* → **Generate a private key**. This is done **only from the web UI**: neither `gh` nor the API allow generating or deleting keys of a GitHub App.

1. Save the new key in 1Password:

   In 1Password, replace the `private_key` field of that App's item with the full content of the new `.pem`.

   With 1Password unlocked (if you have more than one account, export `OP_ACCOUNT` first). The `.pem` is read from the file, so the key is not written in the command:

   ```
   op item edit maceosoft-terraform "private_key[password]=$(cat ~/Downloads/maceosoft-terraform.<fecha>.private-key.pem)"
   # comprueba que tiene contenido, sin mostrarlo
   op read "op://HomeLab/maceosoft-terraform/private_key" | wc -c
   ```

   Change `maceosoft-terraform` to the item of the App you are rotating (`maceosoft-arc-runners` or `maceosoft-argocd`).

1. Verify and apply:

   ```
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

1. If it is the ARC App (`maceosoft-arc-runners`), restart its *listener* so it picks up the new key: `kubectl -n arc-systems get pods` and delete the listener pod (it recreates itself). New runners already use the new key. In a single command: `kubectl -n arc-systems delete pod -l app.kubernetes.io/component=runner-scale-set-listener` (restarts all listeners in the namespace, which causes no issues).

1. Once everything works, delete the **old** key in GitHub (same page, *Delete* next to the previous key) and the downloaded `.pem`.

With the CLI

To delete the downloaded `.pem` and review the App ID and Installation ID:

```
# en macOS: rm -P
shred -u ~/Downloads/maceosoft-terraform.<fecha>.private-key.pem
# App ID e Installation ID
gh api orgs/maceosoft/installations --jq '.installations[] | [.app_slug, .app_id, .id] | @tsv'
```

`gh` requires the organization permission: `gh auth refresh -s admin:read:org`.

## FTP credentials of a repo

1. Change the password in cPanel and update the `ftp-<repo>` item:

   In the 1Password app, replace the `dev_password` field (or `prod_password` for production) of the item.

   Paste the password when prompted; it is not shown on screen nor kept in history:

   ```
   read -rs P; op item edit ftp-<repo> "dev_password[password]=$P"; unset P
   ```

   Use `prod_password` for production.

1. Apply and re-run the pipeline, which loads the new secrets into the repo environments:

   ```
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

   Then re-run the `maceosoft/infra` pipeline on `main`:

   In `maceosoft/infra`: *Actions → tofu → Run workflow* on `main`.

   ```
   gh workflow run tofu.yml --repo maceosoft/infra --ref main
   gh run watch --repo maceosoft/infra
   ```

## Dex client secret

1. In the *Maceosoft Dex* OAuth App: **Generate a new client secret**. GitHub shows it only once. Generating the secret in the OAuth App **is only done from the web UI**.

1. Save the new secret:

   Replace `GITHUB_CLIENT_SECRET` in the `maceosoft-dex` item.

   Paste the secret when prompted; it is not shown on screen nor kept in history:

   ```
   read -rs S; op item edit maceosoft-dex "GITHUB_CLIENT_SECRET[password]=$S"; unset S
   ```

1. Apply (updates `argocd-secret` and restarts Dex):

   ```
   ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml --tags argocd
   ```

1. Test GitHub login in Argo CD and then delete the old secret in the OAuth App.

## Monitoring: Discord webhooks, healthchecks.io ping, and Grafana

They are optional ([4.13](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional)). All three live in 1Password and the playbook places them in the `monitoring` namespace; they are never written to Git.

1. Generate the new value:

   - **Discord webhook:** Discord does not allow regenerating a webhook token. In *Server Settings → Integrations → Webhooks*, use **Delete Webhook** and create a new one ([how](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#4-one-webhook-per-channel)); copy its URL.

   - **healthchecks.io ping URL:** create a new check with the same configuration (**Period** of 5 minutes, **Grace Time** of 10 and its Discord integration), copy its ping URL and delete the old one. The steps are in [Discord and healthchecks.io](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#part-2-healthchecksio).

   - **Grafana password:** a new one in 1Password.

   - **Discord webhook:** deleting the old one can be done with its own URL (paste it when prompted, it is not shown); the new one is created **only from the Discord web UI or app** ([how](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#4-one-webhook-per-channel)):

     ```
     read -rs HOOK; curl -fsS -X DELETE "$HOOK"; unset HOOK
     ```

   - **healthchecks.io ping URL:** using the [API](https://healthchecks.io/docs/api/) and the project's API key (read-write). Create the new check with the same values (`timeout` is the *Period* and `grace` the *Grace Time*, in seconds; `"channels":"*"` connects the integrations you already have, such as Discord), save its ping URL in 1Password and delete the old one:

     ```
     # pega la API key y Enter (no se ve)
     read -rs HC_KEY
     OLD=$(curl -fsS -H "X-Api-Key: $HC_KEY" https://healthchecks.io/api/v3/checks/ \
       | jq -r '.checks[] | select(.name=="homelab-monitoreo") | .uuid')
     NEW_URL=$(curl -fsS -H "X-Api-Key: $HC_KEY" \
       -d '{"name":"homelab-monitoreo","timeout":300,"grace":600,"channels":"*"}' \
       https://healthchecks.io/api/v3/checks/ | jq -r .ping_url)
     op item edit healthchecks-ping "url[password]=$NEW_URL"
     curl -fsS -X DELETE -H "X-Api-Key: $HC_KEY" "https://healthchecks.io/api/v3/checks/$OLD"
     unset HC_KEY NEW_URL OLD
     ```

     With this the `url` field is already updated and you can skip step 2 for the ping URL.

   - **Grafana password:** generated and saved in step 2, with a single command.

1. Replace the field in its item (`critical`, `warnings` or `deployments` in `discord-webhooks`, `url` in `healthchecks-ping`, `password` in `grafana-admin`):

   In the 1Password app, open the item and paste the new value into its field.

   A webhook is pasted when prompted (it is not shown and does not remain in history); the Grafana password is generated inside the command:

   ```
   read -rs V; op item edit discord-webhooks "critical[password]=$V"; unset V
   op item edit grafana-admin "password[password]=$(openssl rand -base64 24)"
   ```

1. Verify and apply:

   ```
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

1. If after a few minutes Alertmanager is still using the previous value, restart its pod; for the Grafana password, restart its Deployment (the volume is ephemeral and reads it again on startup):

   ```
   kubectl -n monitoring get pods
   kubectl -n monitoring rollout restart deploy/vm-grafana
   ```

1. Test with the test alert from [step 7 of 4.13](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional).

## Ollama and internal services tokens

Clients of the [Ollama router](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md) authenticate with a Bearer token, one per client, so that one can be rotated without affecting the rest. All are generated the same way: 32 random bytes in hexadecimal (64 characters).

```
openssl rand -hex 32
```

| 1Password Item         | Fields                                                         | Used by                                                                                                                                                                                                    |
| ---------------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ollama-router`        | `registry_token`                                               | Ollama nodes, to register with the router                                                                                                                                                                  |
| `ollama-router`        | `open-webui`, `k8sgpt`, `reports`, `alert-triage`, `translate` | Each router client. `translate` is also the `OLLAMA_TOKEN` secret in GitHub Actions, used by the translation of this documentation                                                                         |
| `alert-triage-webhook` | `token`                                                        | Alertmanager, when calling `alert-triage`                                                                                                                                                                  |
| `mcp-api-key`          | `key`                                                          | Open WebUI, when calling MCP tools (`mcpo`)                                                                                                                                                                |
| `open-terminal`        | `api_key`                                                      | Open WebUI, when calling Open Terminal ([guide](https://homelab.maceo.cl/en/operacion/ia-ollama/#open-terminal-terminal-for-the-assistant)). After applying, restart the Open Terminal and Open WebUI pods |

To rotate one:

1. Generate the new value and replace **only that field** in 1Password (minimum 32 characters, no spaces):

   Generate the value with the command above (`openssl rand -hex 32`) and paste it into the item's field, in the 1Password app.

   The value is generated inside the command, so you never see it. For example, for the `reports` token and for `mcp-api-key`:

   ```
   op item edit ollama-router "reports[password]=$(openssl rand -hex 32)"
   op item edit mcp-api-key "key[password]=$(openssl rand -hex 32)"
   ```

   Other commands: [Secrets management](https://homelab.maceo.cl/en/secrets/#local-ai-create-tokens-with-op).

1. Verify and apply:

   ```
   cd ansible
   ../.venv/bin/python scripts/apply_platform_secrets.py --check
   ansible-playbook -i inventory.ini playbook-platform-secrets.yml
   ```

1. Restart the service that reads it so it picks up the new value (`kubectl -n <namespace> rollout restart deploy/<nombre>`): the router and the client whose token changed (the router also reloads tokens every 30 seconds). For `registry_token`, also update the token on each node (Linux: re-apply `playbook-ollama.yml --tags register`; Mac: replace the `token` file in the agent folder, or the one in the Keychain, and `ollama-node off` / `ollama-node on`).

1. **Per-node registration tokens (optional).** If you use a `registry_token-<nombre-del-nodo>` key in the `ai/ollama-router-secrets` Secret, only that token registers that node. To rotate it: change that value in the Secret, update the token on that node and verify it still appears in `/_registry/nodes`. The shared token still works for nodes that don't have their own, so to **fully retire** the shared one you must give each node its own first.

1. If you rotated `translate`, also update the `OLLAMA_TOKEN` secret in GitHub Actions:

   In *maceosoft/homelab → Settings → Secrets and variables → Actions*, edit `OLLAMA_TOKEN` and paste the value from the `translate` field.

   The value travels through a pipe, without being printed:

   ```
   op read "op://HomeLab/ollama-router/translate" | gh secret set OLLAMA_TOKEN --repo maceosoft/homelab
   ```

## Root password for nodes

Each run generates and applies a new password per node, and saves it in the `root@<nodo>` item. It does not affect SSH (Secure Shell): root login via SSH is already disabled; this password is for the console.

```
# todos
ansible-playbook -i inventory.ini playbook-rotate-root-passwords.yml
ansible-playbook -i inventory.ini playbook-rotate-root-passwords.yml --limit deborah
```

## Sealed Secrets Key

**Do not rotate lightly.** The `SealedSecret` already in Git were encrypted with the current key: if you change it, those files can no longer be decrypted. The key does not expire in practice (the certificate is valid for 10 years) and automatic renewal is disabled.

Only rotate it if the **private** key was compromised. In that case:

1. Generate a new key pair ([command](https://homelab.maceo.cl/en/maceosoft/github/#step-2-repos-argocd-app-and-ftp-manual)).

1. **Before replacing anything**, reseal each secret with the new key (`kubeseal --cert <tls.crt nuevo> …`) and commit the new `SealedSecret`.

1. Replace `certificate` and `private_key` in the `sealed-secrets` item, re-run `playbook-platform-secrets.yml` and restart the controller: `kubectl -n kube-system rollout restart deploy/sealed-secrets`.

   In the 1Password app, paste the contents of `tls.crt` into `certificate` and `tls.key` into `private_key`.

   The two values are read from the new files, leaving no trace in the command:

   ```
   op item edit sealed-secrets "certificate[text]=$(cat tls.crt)" "private_key[password]=$(cat tls.key)"
   ```

**Reinstalling K3s** is not a rotation: with the item in 1Password, the playbook recreates the same Secret and the `SealedSecret` in Git keep working. Do this **before** enabling `sealed-secrets` in `root-appset.yaml`.

## Verify

- `apply_platform_secrets.py --check` shows all ✓.
- Argo CD keeps syncing `gitops` (`kubectl get applications -n argocd`).
- The runner appears *Online* in *maceosoft → Settings → Actions → Runners* (via CLI: `gh api orgs/maceosoft/actions/runners --jq '.runners[] | [.name, .status] | @tsv'`).
- GitHub login works in Argo CD.

# Remove or Replace a Node

Remove a **worker** from K3s (a lightweight Kubernetes distribution) and an **Incus member**: first empty it, then remove it from each cluster, and finally from the inventory. To **replace** a node, remove the old one using this procedure and add the new one with [Add a Node](https://homelab.maceo.cl/en/operacion/agregar-nodo/index.md).

Nodes Not Removed by This Procedure

- **The K3s control plane node** (`deborah`): removing it will shut down the cluster. First, you need to [move the control plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md) to another node or rebuild it (see the [bootstrap runbook](https://github.com/maceosoft/homelab/blob/main/ansible/bootstrap/README.md)).
- **`deborah`** also serves as the DNS (BIND) for the LAN: move the DNS before removing it.
- **`invincible`** is the Incus *leader* and hosts the Incus UI: move those roles to another node before removing it.
- **Incus Quorum:** the cluster has 3 members. With 2, it still functions but can no longer tolerate another failure. Add the replacement soon.

In the examples, the node to be removed is **`nolan`**.

## 1. Empty the Node in K3s

```
kubectl cordon nolan
kubectl drain nolan --ignore-daemonsets --delete-emptydir-data
```

If the node provides disk space to Longhorn (`node.longhorn.io/create-default-disk: "true"`), disable its *Scheduling* and request that it be emptied:

In the Longhorn UI: *Node → nolan → Edit* → disable *Scheduling* and mark *Eviction Requested*.

Do the same on the resource `nodes.longhorn.io` of that node:

```
kubectl -n longhorn-system patch nodes.longhorn.io nolan --type merge \
  -p '{"spec":{"allowScheduling":false,"evictionRequested":true}}'
# réplicas que siguen en nolan (repite hasta que no salga ninguna)
kubectl -n longhorn-system get replicas.longhorn.io \
  -o custom-columns=NOMBRE:.metadata.name,NODO:.spec.nodeID | grep nolan
```

Wait for its replicas to move to other nodes before proceeding.

## 2. Empty the Node in Incus

From any node in the Incus cluster:

```
# instancias en nolan
sudo incus list --all-projects --format csv --columns nL | grep nolan
# las mueve o detiene
sudo incus cluster evacuate nolan
```

## 3. Remove It from K3s

```
kubectl delete node nolan
cd ansible
ansible-playbook -i inventory.ini playbook-k3s-uninstall.yml --limit nolan
```

On a worker, the playbook only removes the agent (it never touches the control plane).

## 4. Remove It from Incus

From the leader (`invincible`). The `ssh` logs in with [SSO access](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md) (`opkssh login` before):

```
ssh devops@invincible.homelab.local sudo incus cluster remove nolan
```

## 5. Inventory and DNS

1. Delete the node's line in `inventory.ini` (architecture group and `k3s_agent`) and its file `host_vars/nolan.yml`.
1. If it was in `reboot_schedule` (`group_vars/incus_cluster/k3s_install.yml`), remove it from there.
1. Regenerate the DNS to remove `nolan.mco.local`:

```
ansible-playbook -i inventory.ini playbook-bind-dns.yml
```

## Verify

```
# nolan ya no aparece
kubectl get nodes
# nolan ya no aparece
ssh devops@invincible.homelab.local sudo incus cluster list
# sin respuesta
dig +short nolan.mco.local @192.168.20.5
```

In Longhorn, the volumes must remain *Healthy* (without replicas on `nolan`).

## If It Fails

| Symptom                        | Check                                                                                                          |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------- |
| `drain` hangs                  | A pod with PodDisruptionBudget or without a controller; check with `kubectl get pods -A -o wide \| grep nolan` |
| Longhorn volumes *Degraded*    | You did not wait for replica evacuation (step 1); re-add the disk or wait for Longhorn to rebuild              |
| `incus cluster remove` refuses | Instances remain on the node: repeat `incus cluster evacuate nolan`                                            |
| The node no longer boots       | Skip steps that require connecting to it and use `sudo incus cluster remove --force nolan`                     |

# Disk and DNS Health

What is monitored on the hardware of nodes that previously had no alerts, and what to do when each one triggers. The notifications come through Discord just like the rest of the monitoring.

| What                                                                                                                             | Where                                         | How to Read                           |
| -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------- |
| Self-Monitoring, Analysis and Reporting Technology (SMART) diagnostics, damaged sectors, temperature, and remaining life of SSDs | `invincible`, `oliver` (SSD SATA, Serial ATA) | `smartctl` (package `smartmontools`)  |
| Wear and end-of-life warning for the eMMC (embedded MultiMediaCard)                                                              | `deborah`                                     | Files in `/sys/block/mmcblk*/device/` |
| Active and responding BIND (Berkeley Internet Name Domain, LAN DNS)                                                              | `deborah`                                     | A test query every minute             |

The eMMC of `deborah` **does not have SMART**: `smartctl` responds with "Unable to detect device type". What the driver does do is keep track of its wear, and the kernel publishes it in `life_time` and `pre_eol_info`. These are JEDEC (Joint Electron Device Engineering Council) standard fields, so they apply to any eMMC.

## How Metrics Are Collected

On each node, a systemd timer runs a script that writes a file `.prom` in `/var/lib/node_exporter/textfile`. node-exporter reads it with its *textfile collector* and VictoriaMetrics stores it. It is only **read**: no data is written to disks, no SMART autotests are launched, and BIND is not touched. The SMART autotest is launched manually, with Ansible (see [SMART Autotest](#autotest-smart)).

| Component   | Where                                                                                                                                                                                                                                       |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Disk role   | [`ansible/roles/node_smart_metrics`](https://github.com/maceosoft/homelab/tree/main/ansible/roles/node_smart_metrics) and [`playbook-smart-metrics.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-smart-metrics.yml) |
| BIND role   | [`ansible/roles/node_bind_metrics`](https://github.com/maceosoft/homelab/tree/main/ansible/roles/node_bind_metrics) and [`playbook-bind-metrics.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-bind-metrics.yml)     |
| Alerts      | [`gitops/monitoring/config/rules/homelab-hardware.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/config/rules/homelab-hardware.yaml)                                                                                       |
| Alert tests | [`gitops/monitoring/tests/rules-test.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/tests/rules-test.yaml)                                                                                                                 |

Requirement: textfile collector

node-exporter must mount `/var/lib/node_exporter/textfile` (configuration in [`gitops/monitoring/values-vm-stack.yaml`](https://github.com/maceosoft/gitops/blob/main/monitoring/values-vm-stack.yaml)). Without this, the files are written but no one reads them, and after 24 hours `SinDatosDeDiscos` triggers.

### Why a Script and Not `smartctl_exporter`

`prometheus-smartctl-exporter` is not packaged in Debian 13: it would require maintaining a separate binary or image, and in Kubernetes it would run as a privileged DaemonSet with access to `/dev`. Additionally, it would not cover the eMMC, which is the most important node disk. The script uses only `smartctl` (Debian package) and `/sys`, runs as `root` for a few seconds every 30 minutes, and node-exporter does not need elevated privileges.

### Why a Probe and Not `bind_exporter`

BIND does not have `statistics-channels` enabled, so `prometheus-bind-exporter` would have nothing to read. Enabling it would require editing BIND's configuration and restarting it, and the LAN DNS depends on that restart. To check if DNS is working, it is enough to **ask it**: the probe resolves a specific name (`deborah.mco.local`) every minute.

## Alerts and What to Do

**Imminent failure** alerts (`critical`) require action the same day; **warnings** (`warning`) require planning.

### SATA SSDs

| Alert                               | When It Triggers                                                    | What to Do                                                                                                      |
| ----------------------------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `DiscoSmartFalla` (critical)        | The SMART diagnostics report FAILED for 5 minutes                   | The disk predicts it will fail. Verify backups, copy data, and replace it. Details: `sudo smartctl -a /dev/sdX` |
| `DiscoAtributoSmartEnFallo`         | A SMART attribute is below its threshold (`FAILING_NOW`) for 1 hour | Check which one with `sudo smartctl -A /dev/sdX`; it usually precedes `DiscoSmartFalla`                         |
| `DiscoSectoresInestables`           | Unrecoverable or pending sectors greater than 0 for 1 hour          | The disk cannot reliably read some areas. Check backups and plan replacement                                    |
| `DiscoSectoresReasignadosCreciendo` | Reallocated sectors increased in the last 3 days                    | A fixed value is not a problem; an increase is. Plan replacement and monitor the trend                          |
| `SsdVidaBaja`                       | Remaining life less than 20% for 1 hour                             | Buy a replacement before it reaches 0                                                                           |
| `DiscoTemperaturaAlta`              | 60 °C or higher for 15 minutes                                      | The typical maximum for a SATA SSD is 70 °C. Check ventilation and dust on the node                             |

### eMMC of `deborah`

The field `life_time` has two areas (A and B) and the value ranges from 1 to 11 based on wear:

| Value   | Meaning                                                             |
| ------- | ------------------------------------------------------------------- |
| 1 to 10 | Between 0-10% and 90-100% of estimated life used (each step is 10%) |
| 11      | Exceeded estimated life                                             |

And `pre_eol_info` (end-of-life warning, EOL) is 1 if normal, 2 if 80% of the reserved blocks are used, and 3 if 90% are used.

| Alert                       | When It Triggers                                  | What to Do                                                    |
| --------------------------- | ------------------------------------------------- | ------------------------------------------------------------- |
| `EmmcDesgastada` (warning)  | Level 9 or higher (more than 80% used) for 1 hour | Back up (K3s control plane lives there) and plan replacement  |
| `EmmcDesgastada` (critical) | Level 11: exceeded estimated life                 | It may fail at any moment: back up immediately and replace it |
| `EmmcFinDeVida` (warning)   | `pre_eol_info` = 2                                | Plan backup and replacement                                   |
| `EmmcFinDeVida` (critical)  | `pre_eol_info` = 3                                | Back up immediately and replace it                            |

To check the values manually: `cat /sys/block/mmcblk1/device/life_time /sys/block/mmcblk1/device/pre_eol_info`.

How to Reduce eMMC Wear

This is the usual cause of reaching end-of-life prematurely: many small writes (logs, etcd, K3s database). If the level rises quickly, move those loads to another disk.

### Do Not Lose Monitoring

| Alert                 | When It Triggers                                               | What to Do                                                                                                                                    |
| --------------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `DiscoSmartNoLegible` | `smartctl` cannot read the SMART status of a disk for 2 hours  | Test `sudo smartctl -a /dev/sdX`. SMART may be off (`smartctl -s on`), the file may be missing (`smartmontools`), or the disk may not respond |
| `DatosDeDiscosViejos` | The metrics file has not been updated for more than 2 hours    | `systemctl status node-smart-metrics.timer node-smart-metrics.service` and `journalctl -u node-smart-metrics`                                 |
| `SinDatosDeDiscos`    | node-exporter responds but there are no disk metrics for 1 day | The role `playbook-smart-metrics.yml` is missing on that node, or node-exporter does not mount the textfile collector directory               |

### BIND (DNS)

| Alert                        | When triggered                                                 | What to do                                                                                                                                                                |
| ---------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `BindCaido` (critical)       | `named` is not active for 2 min                                | In `deborah`: `systemctl status named` and `journalctl -u named -e`. Check syntax with `named-checkconf` before restarting. Without DNS, internal LAN names won't resolve |
| `DnsSinRespuesta` (critical) | `named` is active but the test query fails or errors for 3 min | `dig @<IP del DNS> deborah.mco.local`. Check `rndc status`, disk space, and `named` logs                                                                                  |
| `DnsLento`                   | The query takes more than 0.5 s for 10 min                     | Normally takes milliseconds. Check node load and forwarders (*forwarders*)                                                                                                |
| `MetricasDeBindViejas`       | The probe has not run for more than 10 min                     | `systemctl status node-bind-metrics.timer`. Until then, it's unknown if DNS is responding                                                                                 |
| `SinDatosDeBind`             | node-exporter of `deborah` does not publish the probe for 1 h  | Missing `playbook-bind-metrics.yml` or the textfile collector is not mounted                                                                                              |

What it does not cover

Runs locally on the `deborah`: warns if BIND falls or does not respond, but not if the network between pods and DNS fails. For that, an external probe (blackbox-exporter module `dns` pointing to the LAN DNS) is needed.

## Apply the playbooks

From the `ansible/` folder of the `homelab` repo, and **after** `gitops` mounts the textfile collector:

```
ansible-playbook -i inventory.ini playbook-smart-metrics.yml --check --diff
ansible-playbook -i inventory.ini playbook-smart-metrics.yml
ansible-playbook -i inventory.ini playbook-bind-metrics.yml --check --diff
ansible-playbook -i inventory.ini playbook-bind-metrics.yml
```

Installs `smartmontools` only where SATA disks are present (`invincible` and `oliver`), creates the directory and timers, and does not restart anything. Verify on each node with `cat /var/lib/node_exporter/textfile/smart.prom` (or `bind.prom` in `deborah`).

Remaining SSD life

Each manufacturer publishes wear in a different SMART attribute. After the first run, check that `node_smart_ssd_life_remaining_percent` exists in `smart.prom`; if it's missing, that model uses another attribute and the script filter needs adjustment (see the role's README). Until it's fixed, `SsdVidaBaja` cannot warn.

## Autotest SMART

An autotest asks the disk itself to check its surface and electronics. It does not erase or write data: the disk does it in the background while the node continues to operate (it may slow down during the test). It confirms a warning like `DiscoSectoresInestables` or checks a disk occasionally. Only applies to SATA SSDs (`invincible`, `oliver`); the eMMC of `deborah` does not have SMART.

From `ansible/`, the simulation shows which disks would be tested; the second command runs it:

```
ansible-playbook -i inventory.ini playbook-smart-metrics.yml --tags smart_selftest --check
ansible-playbook -i inventory.ini playbook-smart-metrics.yml --tags smart_selftest
```

The output shows how many minutes it takes (a long SMART test on an SSD usually takes from minutes to an hour). After that time:

```
ansible-playbook -i inventory.ini playbook-smart-metrics.yml --tags smart_selftest_result
```

These tasks do not run in a normal playbook execution. For another disk or a short autotest: `-e '{"node_smart_metrics_selftest_devices": ["sda"], "node_smart_metrics_selftest_type": "short"}'`.

### How to read the result

In `autotests`, the disk log appears, most recent first (`# 1`):

| `Status`                                         | What it means                                                                                          |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| `Completed without error`                        | The disk did not find any issues                                                                       |
| `Self-test routine in progress`                  | Still running: check it later                                                                          |
| `Aborted by host` or `Interrupted`               | Interrupted (reboot or shutdown); run it again                                                         |
| `Completed: read failure` (or another `failure`) | There are unreadable sectors; the column `LBA_of_first_error` shows where. Backup and plan replacement |

In `atributos`, four counters are shown; look at the last column (`RAW_VALUE`):

| ID  | Attribute                | What it indicates                                  |
| --- | ------------------------ | -------------------------------------------------- |
| 5   | `Reallocated_Sector_Ct`  | Sectors the disk has already replaced with reserve |
| 187 | `Reported_Uncorrect`     | Errors the disk could not correct                  |
| 197 | `Current_Pending_Sector` | Suspect sectors, pending reassignment              |
| 198 | `Offline_Uncorrectable`  | Sectors that could not be read during disk checks  |

Normal is 0 in all four. A fixed value different from 0 is not urgent, but an increase between readings is: backup and plan replacement, as with the SATA disk alerts.

# Product URLs and Access

Where each HomeLab tool lives, how to reach it, and with what credentials. All addresses `*.homelab.local` resolve **only inside the LAN**, because they are served by the HomeLab's own DNS (Domain Name System): BIND on `deborah`, `192.168.20.5`. If your machine doesn't use that DNS, add the entries to your `hosts` file (see [HomeLab Overview](https://homelab.maceo.cl/en/implementacion/resumen-homelab/index.md)).

Main Access

Argo CD

GitOps · GitHub login

[argocd.homelab.local](https://argocd.homelab.local)

Incus UI

Instances, networks, and load balancers · SSO or TLS

[incus.homelab.local:8443](https://incus.homelab.local:8443/ui/)

Grafana Optional

Node temperatures and usage · GitHub SSO

[grafana.homelab.local](https://grafana.homelab.local)

Longhorn Via Tunnel

Replicated storage · no login

`kubectl port-forward` → `127.0.0.1:8080`

Documentation

This guide

[homelab.maceo.cl](https://homelab.maceo.cl)

## Catalog

Everything that has an address, grouped by purpose and ordered by priority within each group.

| Priority           | Meaning                                           |
| ------------------ | ------------------------------------------------- |
| ⭐ **Daily**       | You use it to operate the HomeLab regularly       |
| 🔧 **Occasional**  | You open it when performing a specific task       |
| 🔍 **Diagnostics** | Only when something breaks or a panel shows empty |

| Access Method     | Meaning                                                                                                   |
| ----------------- | --------------------------------------------------------------------------------------------------------- |
| 🌐 **Published**  | Has its own DNS name and Gateway route: opens in the browser                                              |
| 🔌 **Via Tunnel** | Not published: opens with `kubectl port-forward` from your machine ([see commands](#monitoring-consoles)) |
| 💻 **Server**     | Used with a command-line tool, not a browser                                                              |

### 1. Management

What you use to **operate and change** the HomeLab.

| Priority | Product                       | Address                                                                        | Access Method | Access                                                                                             | Purpose                                                                                                                                             |
| -------- | ----------------------------- | ------------------------------------------------------------------------------ | ------------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| ⭐       | **Argo CD**                   | [`https://argocd.homelab.local`](https://argocd.homelab.local)                 | 🌐            | GitHub SSO (team `devops`)                                                                         | See what's deployed, sync, and compare with git                                                                                                     |
| 🔧       | **Open WebUI** (AI assistant) | [`https://open-webui.homelab.local`](https://open-webui.homelab.local)         | 🌐            | GitHub SSO (team `devops`)                                                                         | Chat with the HomeLab's local assistant; a real message can wake an Ollama node ([guide](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)) |
| ⭐       | **Incus UI**                  | [`https://incus.homelab.local:8443/ui/`](https://incus.homelab.local:8443/ui/) | 🌐            | GitHub SSO or TLS certificate                                                                      | Virtual machines, containers, networks, and load balancers                                                                                          |
| ⭐       | **SSH to Servers**            | `ssh devops@<servidor>.homelab.local` (`invincible`, `oliver`, `deborah`)      | 💻            | GitHub identity with `opkssh` ([guide](https://homelab.maceo.cl/en/operacion/acceso-ssh/index.md)) | Access the nodes                                                                                                                                    |
| ⭐       | **Kubernetes API**            | `https://192.168.20.5:6443` (`deborah`)                                        | 💻            | `kubeconfig`                                                                                       | `kubectl` over the cluster                                                                                                                          |
| 🔧       | **Longhorn UI**               | [`http://127.0.0.1:8080`](http://127.0.0.1:8080)                               | 🔌            | No login                                                                                           | Volumes, replicas, and disks of the storage                                                                                                         |
| 🔧       | **vCluster Platform**         | [`https://vcluster.homelab.local`](https://vcluster.homelab.local)             | 🌐            | GitHub SSO                                                                                         | Virtual clusters. Enabled in Phase 5; without it the URL returns 404                                                                                |

### 2. Observability

To **see how** the HomeLab is doing and handle alerts. The main one is Grafana; the others are technical consoles for each component.

| Priority | Product                  | Address                                                          | Access Method | Access                                                                                   | Purpose                                                                                                                               |
| -------- | ------------------------ | ---------------------------------------------------------------- | ------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| ⭐       | **Grafana**              | [`https://grafana.homelab.local`](https://grafana.homelab.local) | 🌐            | GitHub SSO (team `devops`, Admin role); fallback: user `admin` from item `grafana-admin` | Node, temperature, Argo CD, Longhorn, Kong, Incus panels… ([what to watch](https://homelab.maceo.cl/en/operacion/monitoreo/index.md)) |
| 🔧       | **Alertmanager**         | [`http://127.0.0.1:9093`](http://127.0.0.1:9093)                 | 🔌            | No login                                                                                 | Active alerts and **silence** one for a while                                                                                         |
| 🔍       | **vmui** (on `vmsingle`) | [`http://127.0.0.1:8428/vmui/`](http://127.0.0.1:8428/vmui/)     | 🔌            | No login                                                                                 | Query metrics; *Cardinality explorer* to see what takes up the most                                                                   |
| 🔍       | **vmagent**              | [`http://127.0.0.1:8429/targets`](http://127.0.0.1:8429/targets) | 🔌            | No login                                                                                 | What's collected and what fails                                                                                                       |
| 🔍       | **vmalert**              | [`http://127.0.0.1:8880`](http://127.0.0.1:8880)                 | 🔌            | No login                                                                                 | Alert rules and their last error                                                                                                      |
| 🔍       | **blackbox-exporter**    | [`http://127.0.0.1:9115`](http://127.0.0.1:9115)                 | 🔌            | No login                                                                                 | External probe results                                                                                                                |

### 3. Platform (No Daily UI)

Pieces the others depend on. Used for diagnostics.

| Priority | Product             | Address                                                                                                         | Access Method | Access                              | Purpose                                                                                                                                                                                                                                                                                               |
| -------- | ------------------- | --------------------------------------------------------------------------------------------------------------- | ------------- | ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 🔍       | **Dex** (SSO)       | [`https://argocd.homelab.local/api/dex`](https://argocd.homelab.local/api/dex/.well-known/openid-configuration) | 🌐            | No UI of its own                    | OIDC issuer: used by Argo CD, Incus UI, vCluster, Grafana, and SSH with `opkssh`                                                                                                                                                                                                                      |
| 🔍       | **Gateway** (Kong)  | `192.168.23.200`                                                                                                | 💻            | —                                   | Receives traffic from `argocd`, `grafana`, `vcluster`, `api`, `open-webui`, and `ollama`                                                                                                                                                                                                              |
| 🔍       | **Ollama** (AI API) | `https://ollama.homelab.local`                                                                                  | 💻            | Bearer Token (one token per client) | Single router for Ollama nodes; HTTPS only. Inside cluster: `http://ollama-router.ai.svc`. Used by Open WebUI, K8sGPT, reports, and translation ([guide](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md), [add a node](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md)) |
| 🔍       | **DNS** (BIND)      | `192.168.20.5`                                                                                                  | 💻            | —                                   | Zones `homelab.local` and `mco.local`                                                                                                                                                                                                                                                                 |

### 4. Documentation and code

| Priority | What                                        | Address                                                                   |
| -------- | ------------------------------------------- | ------------------------------------------------------------------------- |
| 🔧       | **Documentation**                           | [`https://homelab.maceo.cl`](https://homelab.maceo.cl) (public)           |
| 🔧       | **Code** (Ansible, OpenTofu, documentation) | [`maceosoft/homelab`](https://github.com/maceosoft/homelab)               |
| 🔧       | **GitOps manifests**                        | [`maceosoft/gitops`](https://github.com/maceosoft/gitops)                 |
| 🔍       | **Reusable pipelines**                      | [`maceosoft/core-pipelines`](https://github.com/maceosoft/core-pipelines) |

Each name has its record in BIND

The zone `homelab.local` points `argocd`, `vcluster`, `api`, `grafana`, `open-webui`, and `ollama` to the Gateway IP (`192.168.23.200`), `incus` to the cluster leader (`invincible`, `192.168.20.6`), and each server (`invincible`, `oliver`, `deborah`) to its own IP. There is no wildcard: a new name does not resolve until its record is added. Only Argo CD, Grafana, Open WebUI, and Ollama have a route in the Gateway; that's why `vcluster.homelab.local` resolves but returns 404.

If you add a new interface

Document it here in the same change: a row in the corresponding group (with its priority and how to access it) and, if it's not published, its tunnel command in [Monitoring consoles](#monitoring-consoles) or in its own section.

## Argo CD

The GitOps interface. It prompts for login with GitHub (Dex); only members of the `devops` team in the `maceosoft` organization can enter.

View screenshot: Argo CD login

Inside, the root application `homelab-root` groups all the others (tree view):

View screenshot: homelab-root tree

More on the catalog and common issues: [GitOps — catalog and troubleshooting](https://homelab.maceo.cl/en/gitops/index.md).

## Incus UI

The Incus cluster web interface. It's on node `invincible` (the leader), port `8443`; each node also exposes its own API on that port. It offers two ways to log in:

View screenshot: Incus UI access

- **Login with SSO:** GitHub via Dex (OIDC, standard login). See [Incus Cluster — OIDC via Dex](https://homelab.maceo.cl/en/incus/cluster-setup/#oidc-via-dex-phase-4).
- **Login with TLS:** with a client certificate installed in the browser.

Network load balancers are in **Networks → `ovn-lan` → 'Load balancers' tab** (see [Load Balancing and OVN](https://homelab.maceo.cl/en/incus/cluster-setup/#load-balancing-and-ovn)).

## Longhorn UI

The replicated storage has its own dashboard, **with no login screen and no route in the Gateway**: it's an internal Service. Open it with a temporary tunnel from your machine:

```
export KUBECONFIG=~/.kube/homelab-k3s.yaml
kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80
# y abre http://127.0.0.1:8080  (Ctrl+C para cerrar el túnel)
```

View screenshot: Longhorn dashboard

More on storage classes: [Storage](https://homelab.maceo.cl/en/storage/#upgrade-and-reinstall-longhorn).

## Monitoring consoles

The main monitoring interface is [Grafana](https://grafana.homelab.local) (with SSO). Below are more technical consoles for each component, useful for diagnosis. **They have no login and no route in the Gateway**: they are internal Services in the `monitoring` namespace, so they open with a temporary tunnel from your machine (each command leaves the console at `127.0.0.1`; `Ctrl+C` closes the tunnel):

```
export KUBECONFIG=~/.kube/homelab-k3s.yaml
# vmui
kubectl -n monitoring port-forward svc/vmsingle-vm-victoria-metrics-k8s-stack 8428:8428
# vmagent
kubectl -n monitoring port-forward svc/vmagent-vm-victoria-metrics-k8s-stack 8429:8429
# vmalert
kubectl -n monitoring port-forward svc/vmalert-vm-victoria-metrics-k8s-stack 8880:8080
# Alertmanager
kubectl -n monitoring port-forward svc/vmalertmanager-vm-victoria-metrics-k8s-stack 9093:9093
# blackbox-exporter
kubectl -n monitoring port-forward svc/blackbox-exporter 9115:9115
```

Each needs its own terminal, and the `vmalert` port is `8880` to avoid clashing with Longhorn's `8080`.

| Console                    | Address                         | Purpose                                                                                                                                                                             |
| -------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **vmui** (VictoriaMetrics) | `http://127.0.0.1:8428/vmui/`   | Query and graph metrics without Grafana. The **Cardinality explorer** tab shows which metrics take up the most space (useful for trimming), and **Top queries** the slowest queries |
| **vmagent**                | `http://127.0.0.1:8429/targets` | What is being collected, which ones fail, and with what error. It's the first thing to check when a panel comes up empty                                                            |
| **vmalert**                | `http://127.0.0.1:8880`         | Alert rules, their status (`firing`, `pending`), and the last evaluation error                                                                                                      |
| **Alertmanager**           | `http://127.0.0.1:9093`         | Active alerts, silences (to mute an alert for a while), and Discord route configuration                                                                                             |
| **blackbox-exporter**      | `http://127.0.0.1:9115`         | Results of external probes (Argo CD, Dex, Grafana, redirects, and DNS) and their modules                                                                                            |

Why they are not published

None of these consoles ask for a user: anyone reaching the address could see all metrics or create silences in Alertmanager. Grafana and Argo CD do have SSO. If they are ever published, authentication must be placed in front (e.g., a proxy with Dex) and a BIND record ([each name has its own](#catalog)).

What is monitored and what to do for each alert: [Monitoring and alerts](https://homelab.maceo.cl/en/operacion/monitoreo/index.md).

## vCluster Platform

Its address is [`https://vcluster.homelab.local`](https://vcluster.homelab.local), but **it is not deployed by default**: its Application is commented out in `gitops/bootstrap/root-appset.yaml` and there is no `HTTPRoute`, so until it's enabled in [Phase 5](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md) the URL returns 404.

## No dedicated interface

These components have no UI; they are inspected with `kubectl` or with Argo CD:

| Product                                | How to inspect                                                                            |
| -------------------------------------- | ----------------------------------------------------------------------------------------- |
| **MetalLB** (IP of the `LoadBalancer`) | `kubectl get ipaddresspool,l2advertisement -n metallb-system`; pool `192.168.23.200–.220` |
| **Calico** (pod network)               | `kubectl get tigerastatus` and `kubectl get installation default`                         |
| **Kong** (Gateway)                     | `kubectl get gateway,httproute -A`                                                        |
| **cert-manager** (certificates)        | `kubectl get clusterissuer,certificate -A`                                                |
| **OVN** (Incus virtual network)        | `incus network list` and `ovn-nbctl lb-list` on a node                                    |
| **ARC** (GitHub Actions runners)       | `kubectl get pods -n arc-runners`                                                         |

## Other links

The repositories are in the [catalog](#4-documentation-and-code). Also:

| What                                      | Where                                                         |
| ----------------------------------------- | ------------------------------------------------------------- |
| Full documentation in plain text (for AI) | [`llms-full.txt`](https://homelab.maceo.cl/llms-full.txt)     |
| AI study notebook                         | [Learn with AI](https://homelab.maceo.cl/en/aprende/index.md) |

## Verify they respond

```
for u in https://argocd.homelab.local/ https://grafana.homelab.local/api/health https://incus.homelab.local:8443/ui/; do
  echo "$u -> $(curl -sk -o /dev/null -w '%{http_code}' --max-time 6 $u)"
done
# debe dar 192.168.23.200
dig @192.168.20.5 argocd.homelab.local +short
```

If a URL doesn't load, start with DNS (`dig`) and the Gateway (`curl -k https://192.168.23.200`); the [GitOps troubleshooting](https://homelab.maceo.cl/en/gitops/#quick-troubleshooting) table covers the most common cases.
# Arquitectura, plataforma y referencia

# C4 Level 3 — Components

Zoom into each **container** of the [Level 2](https://homelab.maceo.cl/en/architecture/overview/#level-2-containers): internal components, technologies, and relationships. For the physical node map, continue to **[Level 4 — Deployment](https://homelab.maceo.cl/en/architecture/nivel-4-despliegue/index.md)**.

Return to [Overview (Levels 1–2)](https://homelab.maceo.cl/en/architecture/overview/index.md).

## 3.1 — K3s Management

Breakdown of the bare-metal control cluster: GitOps (Argo CD), identity (Dex), provisioning CAPI/CAPN, service network (MetalLB, Gateway), storage operators (OpenEBS, Longhorn), secrets (Sealed Secrets), upgrades (SUC), and optional vCluster Platform.

Operational details: [Technology Stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md), [GitOps](https://homelab.maceo.cl/en/gitops/index.md).

```
---
title: Componentes — K3s Management (C4 Nivel 3)
config:
  flowchart:
    wrappingWidth: 280
---
flowchart LR
    operador(["<span style='color:#fff'><b>Operador HomeLab</b><br/><small>[Person]</small><br/>kubectl / ArgoCD UI</span>"])
    git["<span style='color:#fff'><b>Git remoto</b><br/><small>[Software System]</small><br/>maceosoft/gitops</span>"]

    subgraph k3s_mgmt["K3s Management"]
        direction LR
        argocd["<span style='color:#000'><b>Argo CD</b><br/><small>[Component: Helm]</small><br/>GitOps controller + server</span>"]
        dex["<span style='color:#000'><b>Dex</b><br/><small>[Component: OIDC]</small><br/>SSO ArgoCD / vCluster / Incus UI</span>"]
        subgraph apps["Applications de Argo CD"]
            direction TB
            sealed["<span style='color:#000'><b>Sealed Secrets</b><br/><small>[Component: controller]</small><br/>Descifra SealedSecret</span>"]
            metallb["<span style='color:#000'><b>MetalLB</b><br/><small>[Component: controller]</small><br/>LoadBalancer L2 en LAN</span>"]
            ingress["<span style='color:#000'><b>Gateway API (Kong)</b><br/><small>[Component: Helm]</small><br/>HTTP(S) hacia apps</span>"]
            openebs["<span style='color:#000'><b>OpenEBS</b><br/><small>[Component: Helm]</small><br/>LocalPV provisioner</span>"]
            longhorn["<span style='color:#000'><b>Longhorn</b><br/><small>[Component: Helm]</small><br/>Storage replicado v1</span>"]
            suc["<span style='color:#000'><b>SUC</b><br/><small>[Component: Helm]</small><br/>Upgrades K3s escalonados</span>"]
            vcluster["<span style='color:#000'><b>vCluster Platform</b><br/><small>[Component: Helm]</small><br/>Sync manual opcional</span>"]
        end
        capi["<span style='color:#000'><b>Cluster API</b><br/><small>[Component: controllers]</small><br/>Core CAPI controllers</span>"]
        capn["<span style='color:#000'><b>CAPN</b><br/><small>[Component: provider]</small><br/>Incus provider controller</span>"]
    end

    incus["<span style='color:#fff'><b>Incus Cluster</b><br/><small>[Container: Incus]</small><br/>Sustrato LXCMachine</span>"]

    operador -->|"Administra<br/><small>[HTTPS]</small>"| argocd
    operador -->|"Accede<br/><small>[HTTPS]</small>"| ingress
    git -->|"Sincroniza<br/><small>[GitOps]</small>"| argocd
    argocd -->|"Despliega config<br/><small>[Helm]</small>"| dex
    argocd -->|"Despliega<br/><small>[Application]</small>"| apps
    dex -.->|"Emite tokens<br/><small>[OIDC]</small>"| operador
    capi -->|"Orquesta<br/><small>[CAPI]</small>"| capn
    capn -->|"Provisiona<br/><small>[API REST]</small>"| incus

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class operador person
    class git external
    class incus container
    class argocd,dex,sealed,metallb,ingress,openebs,longhorn,suc,vcluster,capi,capn component
    style k3s_mgmt fill:none,stroke:#888,stroke-dasharray:6 4
    style apps fill:none,stroke:#aaa,stroke-dasharray:2 3
```

## 3.2 — Incus Cluster

Internals of the HA (high availability) Incus cluster on the 3 physical nodes: quorum dqlite, daemon per member, pool `local` (driver `dir`), cluster groups, manual scheduler in `oliver`, native UI in invincible and OIDC (OpenID Connect) Dex (Phase 4).

Details: [Incus Cluster](https://homelab.maceo.cl/en/incus/cluster-setup/index.md).

```
---
title: Componentes — Incus Cluster (C4 Nivel 3)
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    operador(["<span style='color:#fff'><b>Operador HomeLab</b><br/><small>[Person]</small><br/>incus CLI / UI</span>"])
    k3s_mgmt["<span style='color:#fff'><b>K3s Management</b><br/><small>[Container: Kubernetes]</small><br/>CAPN provider</span>"]
    subgraph incus["Incus Cluster"]
        dqlite["<span style='color:#000'><b>dqlite</b><br/><small>[Component: DB cluster]</small><br/>Quorum 3 miembros</span>"]
        incusd["<span style='color:#000'><b>incusd</b><br/><small>[Component: daemon]</small><br/>API REST en cada nodo</span>"]
        storage_pool["<span style='color:#000'><b>Pool local</b><br/><small>[Component: dir driver]</small><br/>Pool local por miembro</span>"]
        groups["<span style='color:#000'><b>Cluster groups</b><br/><small>[Component: Incus]</small><br/>x86-nodes / arm64-nodes</span>"]
        scheduler["<span style='color:#000'><b>Scheduler</b><br/><small>[Component: Incus]</small><br/>oliver manual</span>"]
        incus_ui["<span style='color:#000'><b>Incus UI</b><br/><small>[Component: HTTPS :8443]</small><br/>Solo en invincible</span>"]
        oidc["<span style='color:#000'><b>OIDC config</b><br/><small>[Component: Dex]</small><br/>Fase 4 SSO</span>"]
    end

    operador -->|"Administra<br/><small>[HTTPS]</small>"| incus_ui
    operador -->|"CLI<br/><small>[incus]</small>"| incusd
    k3s_mgmt -->|"CAPN API<br/><small>[REST]</small>"| incusd
    incusd -->|"Persiste estado<br/><small>[dqlite]</small>"| dqlite
    incusd -->|"Monta volúmenes<br/><small>[dir]</small>"| storage_pool
    groups -->|"Dirige instancias<br/><small>[cluster group]</small>"| scheduler
    incus_ui -->|"Proxy UI<br/><small>[local socket]</small>"| incusd
    oidc -->|"SSO<br/><small>[OIDC]</small>"| incus_ui

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class operador person
    class k3s_mgmt container
    class dqlite,incusd,storage_pool,groups,scheduler,incus_ui,oidc component
    style incus fill:none,stroke:#888,stroke-dasharray:6 4
```

## 3.3 — Workload Clusters

Internal structure of a **kubeadm** ephemeral cluster provisioned by CAPN (Cluster API Provider for Incus) (e.g., `demo`): LXCMachine in Incus, control plane, workers, and CNI (Container Network Interface) Flannel of the workload. Not a nested K3s (lightweight Kubernetes distribution).

Details: [CAPN](https://homelab.maceo.cl/en/capn/index.md), [Phase 6](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md).

```
---
title: Componentes — Workload Clusters (C4 Nivel 3)
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    k3s["<span style='color:#fff'><b>K3s Management</b><br/><small>[Container: Kubernetes]</small><br/>CAPN + CAPI</span>"]

    subgraph workload["Workload Cluster CAPN &nbsp;[kubeadm]"]
        direction TB
        machines["<span style='color:#000'><b>LXCMachine</b><br/><small>[Component: CAPN]</small><br/>Infra en Incus</span>"]
        cp["<span style='color:#000'><b>Control plane</b><br/><small>[Component: kubeadm]</small><br/>API + etcd + scheduler</span>"]
        workers["<span style='color:#000'><b>Workers</b><br/><small>[Component: kubelet]</small><br/>Nodos worker</span>"]
        cni["<span style='color:#000'><b>CNI</b><br/><small>[Component: Flannel]</small><br/>Red del workload cluster</span>"]
    end

    subgraph infra[" "]
        direction LR
        storage["<span style='color:#fff'><b>Storage</b><br/><small>[Container: OpenEBS + Longhorn]</small><br/>PVC del management</span>"]
        incus["<span style='color:#fff'><b>Incus Cluster</b><br/><small>[Container: Incus]</small><br/>Aloja LXCMachine</span>"]
    end

    k3s -->|"Provisiona<br/><small>[CAPN]</small>"| machines
    cp -->|"Orquesta<br/><small>[Kubernetes]</small>"| workers
    workers -->|"Red de pods<br/><small>[CNI]</small>"| cni
    workers -.->|"PVC opcional<br/><small>[CSI]</small>"| storage
    machines -->|"Instancia<br/><small>[LXC/VM]</small>"| incus

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class k3s,incus,storage container
    class machines,cp,workers,cni component
    style workload fill:none,stroke:#888,stroke-dasharray:6 4
    style infra fill:none,stroke:none
```

## 3.4 — Storage

Logical components of the hybrid tier **OpenEBS LocalPV** (default, 3 nodes) + **Longhorn v1** (cross-arch replica between `invincible` and `deborah`; see [Longhorn v1](https://homelab.maceo.cl/en/storage/#longhorn-v1-selective-ha-cross-arch)). The pods run in K3s; here they are modeled as a separate Level 2 C4 container for clarity.

Details: [Storage](https://homelab.maceo.cl/en/storage/index.md).

```
---
title: Componentes — Storage (C4 Nivel 3)
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    workload["<span style='color:#fff'><b>Workload Clusters</b><br/><small>[Container: kubeadm]</small><br/>Pods con PVC</span>"]
    k3s_mgmt["<span style='color:#fff'><b>K3s Management</b><br/><small>[Container: Kubernetes]</small><br/>Pods del management</span>"]
    subgraph storage["Storage tier"]
        sc["<span style='color:#000'><b>StorageClasses</b><br/><small>[Component: GitOps]</small><br/>openebs-hostpath + longhorn-mixto</span>"]
        openebs_prov["<span style='color:#000'><b>OpenEBS LocalPV</b><br/><small>[Component: DaemonSet]</small><br/>Default 3 nodos</span>"]
        longhorn_mgr["<span style='color:#000'><b>Longhorn manager</b><br/><small>[Component: Deployment]</small><br/>Control de volúmenes</span>"]
        longhorn_eng["<span style='color:#000'><b>Longhorn engine</b><br/><small>[Component: DaemonSet]</small><br/>Réplicas v1 cross-arch</span>"]
        csi["<span style='color:#000'><b>CSI drivers</b><br/><small>[Component: CSI]</small><br/>OpenEBS + Longhorn</span>"]
    end

    k3s_mgmt -->|"Aplica<br/><small>[homelab-storage]</small>"| sc
    workload -->|"Monta PVC<br/><small>[CSI]</small>"| csi
    k3s_mgmt -->|"Monta PVC<br/><small>[CSI]</small>"| csi
    csi -->|"LocalPV<br/><small>[openebs-hostpath]</small>"| openebs_prov
    csi -->|"Réplica HA<br/><small>[longhorn-mixto]</small>"| longhorn_eng
    longhorn_mgr -->|"Gestiona<br/><small>[Longhorn API]</small>"| longhorn_eng

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class workload,k3s_mgmt container
    class sc,openebs_prov,longhorn_mgr,longhorn_eng,csi component
    style storage fill:none,stroke:#888,stroke-dasharray:6 4
```

## 3.5 — Cinc Lab

Learning sandbox **Cinc/Chef** in Incus VMs (virtual machines) (Ubuntu 24.04), one in `@x86-nodes` and another in `@arm64-nodes`. Not a production piece of the HomeLab.

Details: [Cinc/Chef Lab](https://homelab.maceo.cl/en/labs/cinc/index.md).

```
---
title: Componentes — Cinc Lab (C4 Nivel 3)
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    operador(["<span style='color:#fff'><b>Operador HomeLab</b><br/><small>[Person]</small><br/>incus / chef</span>"])
    incus["<span style='color:#fff'><b>Incus Cluster</b><br/><small>[Container: Incus]</small><br/>Aloja VMs lab</span>"]
    omnitruck["<span style='color:#fff'><b>Omnitruck</b><br/><small>[Software System]</small><br/>Paquetes cinc-client</span>"]

    subgraph cinclab["Cinc Lab"]
        direction TB
        subgraph vms["VMs del lab"]
            direction LR
            vm_x86["<span style='color:#000'><b>cinc-lab-x86</b><br/><small>[Component: Incus VM]</small><br/>Ubuntu 24.04 @x86-nodes</span>"]
            vm_arm["<span style='color:#000'><b>cinc-lab-arm</b><br/><small>[Component: Incus VM]</small><br/>Ubuntu 24.04 @arm64-nodes</span>"]
        end
        cinc_client["<span style='color:#000'><b>cinc-client</b><br/><small>[Component: deb]</small><br/>Agente Cinc/Chef</span>"]
        cookbooks["<span style='color:#000'><b>Cookbooks</b><br/><small>[Component: Git local]</small><br/>cinc-lab/cookbooks</span>"]
    end

    operador -->|"SSH / chef<br/><small>[lab]</small>"| vms
    incus -->|"Aloja<br/><small>[VM]</small>"| vms
    omnitruck -->|"Instala<br/><small>[HTTPS]</small>"| cinc_client
    cookbooks -->|"Converge<br/><small>[chef-client]</small>"| cinc_client

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class operador person
    class omnitruck external
    class incus container
    class cookbooks,cinc_client,vm_x86,vm_arm component
    style cinclab fill:none,stroke:#888,stroke-dasharray:6 4
    style vms fill:none,stroke:#aaa,stroke-dasharray:2 3
```

## 3.6 — Monitoring

Observability stack in the namespace `monitoring`, deployed by Argo CD. **VictoriaMetrics** (Prometheus-compatible metrics database) stores the series; **vmagent** scrapes them every 60 seconds from the nodes, the platform, and Incus; **vmalert** evaluates the rules and **Alertmanager** sends the alerts to Discord. The **probes** are executed by **blackbox-exporter** (HTTP and DNS seen from inside the cluster). On each node, **node-exporter** publishes CPU, memory, and temperatures, and also reads the **textfile collectors**: files `.prom` that some timers (`systemd timers`) write to the node with pending apt patches and the DNS (BIND) status; the SMART role is prepared in the repo.

```
---
title: Componentes — Monitoreo (C4 Nivel 3)
config:
  flowchart:
    wrappingWidth: 280
---
flowchart LR
    operador(["<span style='color:#fff'><b>Operador HomeLab</b><br/><small>[Person]</small><br/>Grafana vía SSO (Dex)</span>"])
    discord["<span style='color:#fff'><b>Discord</b><br/><small>[Software System]</small><br/>alertas-criticas / alertas-avisos</span>"]
    healthchecks["<span style='color:#fff'><b>healthchecks.io</b><br/><small>[Software System]</small><br/>Dead man's switch</span>"]
    incus["<span style='color:#fff'><b>Incus Cluster</b><br/><small>[Container: Incus]</small><br/>/1.0/metrics en :8443 de cada nodo</span>"]
    plat["<span style='color:#fff'><b>Plataforma</b><br/><small>[Container: K3s Management]</small><br/>Argo CD, Longhorn, cert-manager, Kong, MetalLB, Calico</span>"]
    web["<span style='color:#fff'><b>Endpoints del HomeLab</b><br/><small>[Software System]</small><br/>argocd, grafana y DNS (BIND) en homelab.local</span>"]

    subgraph monitoreo["Monitoreo (namespace monitoring)"]
        direction TB
        vmagent["<span style='color:#000'><b>vmagent</b><br/><small>[Component: scraper]</small><br/>Recoge métricas cada 60 s</span>"]
        vmsingle["<span style='color:#000'><b>VictoriaMetrics (vmsingle)</b><br/><small>[Component: base de series temporales]</small><br/>Almacena las series (PVC Longhorn)</span>"]
        vmalert["<span style='color:#000'><b>vmalert</b><br/><small>[Component: rules]</small><br/>Evalúa reglas de alerta</span>"]
        alertmanager["<span style='color:#000'><b>Alertmanager</b><br/><small>[Component: notifier]</small><br/>Agrupa, enruta y repite avisos</span>"]
        grafana["<span style='color:#000'><b>Grafana</b><br/><small>[Component: UI]</small><br/>Paneles; SSO con Dex</span>"]
        blackbox["<span style='color:#000'><b>blackbox-exporter</b><br/><small>[Component: probes]</small><br/>Sondas HTTP, redirección y DNS (VMProbe)</span>"]
        subgraph nodos["En cada nodo (invincible, oliver, deborah)"]
            direction TB
            nodeexp["<span style='color:#000'><b>node-exporter</b><br/><small>[Component: DaemonSet]</small><br/>CPU, memoria, disco, temperaturas</span>"]
            textfile["<span style='color:#000'><b>Textfile collectors</b><br/><small>[Component: systemd timers]</small><br/>Parches de apt y estado del DNS (.prom)</span>"]
        end
    end

    textfile -->|"Escribe archivos .prom<br/><small>[/var/lib/node_exporter/textfile]</small>"| nodeexp
    vmagent -->|"Scrape<br/><small>[HTTP :9100]</small>"| nodeexp
    vmagent -->|"Scrape<br/><small>[ServiceMonitor]</small>"| plat
    vmagent -->|"Scrape<br/><small>[HTTPS :8443, certificado cliente metrics]</small>"| incus
    vmagent -->|"Scrape<br/><small>[/probe]</small>"| blackbox
    blackbox -->|"Sondea<br/><small>[HTTP / DNS]</small>"| web
    vmagent -->|"Escribe<br/><small>[remote write]</small>"| vmsingle
    vmalert -->|"Consulta<br/><small>[consulta de métricas]</small>"| vmsingle
    vmalert -->|"Dispara alertas<br/><small>[HTTP]</small>"| alertmanager
    grafana -->|"Consulta<br/><small>[consulta de métricas]</small>"| vmsingle
    alertmanager -->|"Avisos<br/><small>[webhook HTTPS]</small>"| discord
    alertmanager -->|"Ping Watchdog cada 5 min<br/><small>[HTTPS]</small>"| healthchecks
    healthchecks -.->|"Avisa si el ping se corta<br/><small>[Discord]</small>"| discord
    operador -->|"Consulta<br/><small>[HTTPS]</small>"| grafana

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class operador person
    class discord,healthchecks,web external
    class incus,plat container
    class vmagent,vmsingle,vmalert,alertmanager,grafana,blackbox,nodeexp,textfile component
    style monitoreo fill:none,stroke:#888,stroke-dasharray:6 4
    style nodos fill:none,stroke:#aaa,stroke-dasharray:2 3
```

## Next

**[→ Level 4 — Deployment](https://homelab.maceo.cl/en/architecture/nivel-4-despliegue/index.md)** — physical map of the 3 nodes.

# C4 Level 4 — Deployment

**Physical** map of the platform: the 3 nodes on the LAN (local network), Debian OS with `br0`, the Incus daemon and the K3s (lightweight Kubernetes distribution) role on each one. It complements the logical zoom of [Level 3 — Components](https://homelab.maceo.cl/en/architecture/nivel-3-componentes/index.md).

Back to [Overview (Level 1–2)](https://homelab.maceo.cl/en/architecture/overview/index.md).

## Nodes and roles

| Node         | Hardware               | RAM     | `br0` IP     | Incus                        | K3s                     |
| ------------ | ---------------------- | ------- | ------------ | ---------------------------- | ----------------------- |
| `invincible` | Lenovo M920q x86_64    | 15.5 GB | 192.168.20.6 | Leader + UI `:8443`          | agent; Longhorn disk    |
| `oliver`     | Lenovo M700 x86_64     | 7.7 GB  | 192.168.20.7 | Quorum; **manual** scheduler | agent                   |
| `deborah`    | Orange Pi 5 Plus ARM64 | 31 GB   | 192.168.20.5 | `arm64-nodes` member         | **server** (CP) `:6443` |

All three nodes also run on-demand Ollama (`llm_nodes` group, see [Local AI](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)), and `deborah` serves the LAN DNS (BIND, `homelab.local` zone).

RAM budget

`oliver` has a manual scheduler because it must not host Incus instances automatically (it saves RAM for the management cluster). `deborah` is the control plane because of its available RAM (31 GB), even though it is ARM64 (64-bit ARM architecture) — validate the CNI (Container Network Interface) and workloads before heavy production use.

Hardware details: [Inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md).

## Deployment diagram

```
---
title: Despliegue (C4 Nivel 4) - Plataforma HomeLab
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    subgraph lan["Red LAN &nbsp;[192.168.20.0/22 · br0] · Debian + netplan en los 3 nodos"]
        direction TB
        subgraph deborah["deborah · Orange Pi 5 Plus ARM64 · 31 GB · .5"]
            direction LR
            k3s_deb["<span style='color:#fff'><b>K3s</b><br/><small>[Container]</small><br/>server control-plane :6443</span>"]
            incus_deb["<span style='color:#fff'><b>Incus</b><br/><small>[Container]</small><br/>arm64-nodes member</span>"]
            bind_deb["<span style='color:#fff'><b>BIND</b><br/><small>[Container]</small><br/>DNS homelab.local :53</span>"]
            oll_deb["<span style='color:#fff'><b>Ollama</b><br/><small>[Container]</small><br/>bajo demanda :11434</span>"]
        end
        subgraph invincible["invincible · M920q x86_64 · 15.5 GB · .6"]
            direction LR
            k3s_inv["<span style='color:#fff'><b>K3s</b><br/><small>[Container]</small><br/>agent + Longhorn disk</span>"]
            incus_inv["<span style='color:#fff'><b>Incus</b><br/><small>[Container]</small><br/>leader + UI :8443</span>"]
            oll_inv["<span style='color:#fff'><b>Ollama</b><br/><small>[Container]</small><br/>bajo demanda :11434</span>"]
        end
        subgraph oliver["oliver · M700 x86_64 · 7.7 GB · .7"]
            direction LR
            incus_nol["<span style='color:#fff'><b>Incus</b><br/><small>[Container]</small><br/>quorum, scheduler manual</span>"]
            k3s_nol["<span style='color:#fff'><b>K3s</b><br/><small>[Container]</small><br/>agent</span>"]
            oll_nol["<span style='color:#fff'><b>Ollama</b><br/><small>[Container]</small><br/>bajo demanda :11434</span>"]
        end
    end

    k3s_deb -->|"Cluster<br/><small>[K3s agent]</small>"| k3s_inv
    k3s_deb -->|"Cluster<br/><small>[K3s agent]</small>"| k3s_nol
    incus_deb <-.->|"Quorum<br/><small>[dqlite]</small>"| incus_inv
    incus_deb <-.->|"Quorum<br/><small>[dqlite]</small>"| incus_nol
    incus_inv <-.->|"Quorum<br/><small>[dqlite]</small>"| incus_nol

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class incus_inv,k3s_inv,k3s_deb,incus_deb,incus_nol,k3s_nol,bind_deb,oll_deb,oll_inv,oll_nol container
    style lan fill:none,stroke:#888,stroke-dasharray:6 4
    style invincible fill:none,stroke:#888,stroke-dasharray:6 4
    style deborah fill:none,stroke:#888,stroke-dasharray:6 4
    style oliver fill:none,stroke:#888,stroke-dasharray:6 4
```

## Previous

**[← Level 3 — Components](https://homelab.maceo.cl/en/architecture/nivel-3-componentes/index.md)**

# Architecture

Design view of the HomeLab. For the **technology stack** with a layered diagram and links to official documentation of each product, see **[Technology Stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md)**.

Executable guide: **[Implementation](https://homelab.maceo.cl/en/implementacion/index.md)**.

## Layers

1. **Hardware** — 3 physical nodes (2x x86_64, 1x ARM64). See [inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md).
1. **Incus Cluster** — virtualization substrate, 3 members with dqlite quorum. Cluster groups `x86-nodes` / `arm64-nodes` to direct where each instance runs.
1. **K3s (lightweight Kubernetes distribution) Management** — the existing bare-metal K3s cluster, hosting:
   - Cluster API core + CAPN controller (Cluster API Provider for Incus)
   - Argo CD (GitOps)
   - Sealed Secrets controller
1. **Workload Clusters** — ephemeral kubeadm clusters provisioned by CAPN as Incus (LXC or VM) instances for dev/test/isolation.
1. **Storage** — OpenEBS LocalPV (default, on all 3 nodes) + Longhorn v1 (replicated between `invincible` and `deborah`; `oliver` stays out due to RAM if `createDefaultDiskLabeledNodes` is activated, see [Longhorn v1](https://homelab.maceo.cl/en/storage/#longhorn-v1-selective-ha-cross-arch)).

## Key Design Decision

The production K3s is not migrated to nested-K8s. The existing K3s cluster is kept as the *management cluster* for Cluster API; CAPN provisions additional clusters on demand over the same hardware without touching production.

## C4 Diagrams

### Level 1 — Context

High-level view: the **HomeLab Platform** as a single black box and the external actors/systems it interacts with (operator, Git, image registry, Omnitruck, LAN, and Discord and healthchecks.io alert services). Useful for understanding *what goes in and out* of the system without internal details.

```
---
title: Diagrama de Contexto (C4 Nivel 1) - Plataforma HomeLab
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    operador(["<span style='color:#fff'><b>Operador HomeLab</b><br/><small>[Person]</small><br/>Administra el clúster vía kubectl, SSH y la UI de ArgoCD</span>"])
    plataforma["<span style='color:#fff'><b>Plataforma HomeLab</b><br/><small>[Software System]</small><br/>K3s + Incus + CAPN — sistema en alcance</span>"]
    git["<span style='color:#fff'><b>Git remoto</b><br/><small>[Software System]</small><br/>GitHub / GitLab: fuente de verdad de los manifiestos</span>"]
    registro["<span style='color:#fff'><b>Registro de imágenes</b><br/><small>[Software System]</small><br/>images.linuxcontainers.org (simplestreams Incus/CAPN)</span>"]
    omnitruck["<span style='color:#fff'><b>Omnitruck Cinc/Chef</b><br/><small>[Software System]</small><br/>Distribuye los paquetes de cinc-client/cinc-auditor</span>"]
    lan["<span style='color:#fff'><b>Red LAN / Router</b><br/><small>[Software System]</small><br/>DHCP y asignación de IPs fijas para los 3 nodos</span>"]
    discord["<span style='color:#fff'><b>Discord</b><br/><small>[Software System]</small><br/>Canales de alertas críticas, avisos y despliegues</span>"]
    healthchecks["<span style='color:#fff'><b>healthchecks.io</b><br/><small>[Software System]</small><br/>Dead man's switch: avisa si el monitoreo deja de pingear</span>"]

    operador -->|"Administra<br/><small>[kubectl / SSH / ArgoCD UI]</small>"| plataforma
    plataforma -->|"Sincroniza manifiestos<br/><small>[GitOps (ArgoCD)]</small>"| git
    plataforma -->|"Descarga imágenes<br/><small>[kubeadm / ubuntu]</small>"| registro
    plataforma -->|"Instala paquetes<br/><small>[HTTPS]</small>"| omnitruck
    plataforma <-->|"Bridge L2, IP fija<br/><small>[netplan]</small>"| lan
    plataforma -->|"Envía alertas y avisos<br/><small>[webhook HTTPS]</small>"| discord
    plataforma -->|"Ping cada 5 min<br/><small>[HTTPS]</small>"| healthchecks
    healthchecks -.->|"Avisa si el ping se corta<br/><small>[integración Discord]</small>"| discord
    discord -.->|"Notifica<br/><small>[móvil / escritorio]</small>"| operador

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class operador person
    class plataforma system
    class git,registro,omnitruck,lan,discord,healthchecks external
```

### Level 2 — Containers

Open the box of the platform and show the **logical containers** (K3s Management, Incus Cluster, Workload Clusters, Storage, Monitoring, Logs, Local AI, Gateway and SSO, DNS, and Cinc Lab) with their technologies and how they relate to each other and to external systems.

```
---
title: Diagrama de Contenedores (C4 Nivel 2) - Plataforma HomeLab
config:
  flowchart:
    wrappingWidth: 280
---
flowchart TB
    operador(["<span style='color:#fff'><b>Operador HomeLab</b><br/><small>[Person]</small><br/>kubectl / SSH / ArgoCD UI</span>"])
    subgraph homelab["Plataforma HomeLab"]
        k3s_mgmt["<span style='color:#fff'><b>K3s Management</b><br/><small>[Container: Kubernetes bare-metal]</small><br/>Cluster API + CAPN controller + ArgoCD + Sealed Secrets</span>"]
        incus["<span style='color:#fff'><b>Incus Cluster</b><br/><small>[Container: Incus, dqlite HA]</small><br/>3 miembros: grupo x86 (M920q+M700) y grupo arm64 (Orange Pi)</span>"]
        workload["<span style='color:#fff'><b>Workload Clusters</b><br/><small>[Container: kubeadm sobre LXCMachine]</small><br/>Clusters K8s efímeros provisionados por CAPN (LXC o VM)</span>"]
        storage["<span style='color:#fff'><b>Storage</b><br/><small>[Container: OpenEBS LocalPV + Longhorn v1]</small><br/>Persistencia local y replicada entre x86/arm64</span>"]
        monitoreo["<span style='color:#fff'><b>Monitoreo</b><br/><small>[Container: VictoriaMetrics + Grafana]</small><br/>Métricas, sondas, alertas y paneles (namespace monitoring)</span>"]
        cinclab["<span style='color:#fff'><b>Cinc Lab VMs</b><br/><small>[Container: Incus VM (Ubuntu 24.04)]</small><br/>Sandbox para probar cookbooks de Cinc/Chef</span>"]
        acceso["<span style='color:#fff'><b>Gateway y SSO</b><br/><small>[Container: Kong Gateway + Dex]</small><br/>Entrada HTTPS de *.homelab.local y login con GitHub (team devops)</span>"]
        ia["<span style='color:#fff'><b>IA local</b><br/><small>[Container: Ollama router + Open WebUI]</small><br/>Chat y API de modelos; Ollama bajo demanda en los 3 nodos</span>"]
        logs["<span style='color:#fff'><b>Logs</b><br/><small>[Container: VictoriaLogs + Vector]</small><br/>Logs de pods y nodos (namespace monitoring)</span>"]
        dns["<span style='color:#fff'><b>DNS</b><br/><small>[Container: BIND en deborah]</small><br/>Zona homelab.local y reenvío al router</span>"]
    end
    git["<span style='color:#fff'><b>Git remoto</b><br/><small>[Software System]</small><br/>GitHub / GitLab</span>"]
    registro["<span style='color:#fff'><b>Registro de imágenes</b><br/><small>[Software System]</small><br/>images.linuxcontainers.org</span>"]
    discord["<span style='color:#fff'><b>Discord</b><br/><small>[Software System]</small><br/>Canales de alertas y despliegues</span>"]
    healthchecks["<span style='color:#fff'><b>healthchecks.io</b><br/><small>[Software System]</small><br/>Dead man's switch</span>"]

    operador -->|"Administra<br/><small>[kubectl / ArgoCD UI]</small>"| k3s_mgmt
    git -->|"Sincroniza manifiestos<br/><small>[GitOps]</small>"| k3s_mgmt
    k3s_mgmt -->|"Provisiona instancias<br/><small>[API REST Incus (LXCMachine)]</small>"| incus
    registro -->|"Sirve imágenes<br/><small>[simplestreams]</small>"| incus
    incus -->|"Aloja<br/><small>[cluster group @x86 / @arm64]</small>"| workload
    incus -->|"Aloja<br/><small>[target manual]</small>"| cinclab
    workload -->|"Monta volúmenes<br/><small>[PVC]</small>"| storage
    k3s_mgmt -->|"Despliega por GitOps<br/><small>[Argo CD]</small>"| monitoreo
    monitoreo -->|"Lee métricas de los nodos<br/><small>[HTTPS :8443, certificado metrics]</small>"| incus
    monitoreo -->|"Guarda métricas<br/><small>[PVC Longhorn]</small>"| storage
    monitoreo -->|"Alertas<br/><small>[webhook HTTPS]</small>"| discord
    monitoreo -->|"Ping cada 5 min<br/><small>[HTTPS]</small>"| healthchecks
    operador -->|"Consulta paneles<br/><small>[Grafana, SSO]</small>"| monitoreo
    operador -->|"Resuelve nombres<br/><small>[DNS :53]</small>"| dns
    operador -->|"Entra a las UIs<br/><small>[HTTPS + SSO]</small>"| acceso
    acceso -->|"Enruta<br/><small>[HTTPRoute]</small>"| ia
    acceso -->|"Enruta Grafana y Argo CD<br/><small>[HTTPRoute]</small>"| monitoreo
    logs -->|"Guarda logs<br/><small>[PVC Longhorn]</small>"| storage
    monitoreo -->|"Resume alertas<br/><small>[token Bearer]</small>"| ia

    classDef person fill:#08427B,stroke:#073B6F,color:#fff
    classDef system fill:#1168BD,stroke:#0B4884,color:#fff
    classDef container fill:#438DD5,stroke:#3C7FC0,color:#fff
    classDef component fill:#85BBF0,stroke:#78A8D8,color:#000
    classDef external fill:#999999,stroke:#8A8A8A,color:#fff
    class operador person
    class k3s_mgmt,incus,workload,storage,monitoreo,cinclab,acceso,ia,logs,dns container
    class git,registro,discord,healthchecks external
    style homelab fill:none,stroke:#888,stroke-dasharray:6 4
```

## Dive Deeper into C4

| Level | Page                                                                                    | What it shows                                                                                                                                                                                                                                                                                                                       |
| ----- | --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1–2   | *(this page)*                                                                           | Context + logical containers                                                                                                                                                                                                                                                                                                        |
| 3     | **[Components](https://homelab.maceo.cl/en/architecture/nivel-3-componentes/index.md)** | Zoom into container (K3s, Incus, workload, storage, monitoring, Cinc); AI, logs, and access in [Local AI](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md), [Logs](https://homelab.maceo.cl/en/operacion/logs/index.md), and [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md) |
| 4     | **[Deployment](https://homelab.maceo.cl/en/architecture/nivel-4-despliegue/index.md)**  | 3 physical nodes on the LAN                                                                                                                                                                                                                                                                                                         |

# Partitioning scheme for workload VMs (CAPN)

Convention for the VMs (virtual machines) that CAPN (Cluster API Provider for Incus) provisions as workload clusters (for example `demo`, see [Workload Clusters](https://homelab.maceo.cl/en/architecture/nivel-3-componentes/#33-workload-clusters)). It is a template for the base image of those VMs, not a measured state.

It comes from the partition review of the 3 bare-metal nodes (the [`playbook-discovery.yml`](https://homelab.maceo.cl/en/hardware/discovery/index.md) report, section "Recomendaciones — esquema de partición del SO"): no node has `/var` separate from `/`, and the swap of each one was sized with a different criterion (between 13 % and 94 % off from `min(RAM/2, 8GB)`). On an already installed node this cannot be fixed without repartitioning the mounted root; on a new VM it is defined from the first boot.

## Scheme

| Partition                          | Size                                | Filesystem | Notes                                                                                                |
| ---------------------------------- | ----------------------------------- | ---------- | ---------------------------------------------------------------------------------------------------- |
| `/boot` (or `/boot/efi` with UEFI) | 1 GB                                | vfat/ext4  | Same as the 3 bare-metal nodes                                                                       |
| swap                               | `min(RAM/2, 8GB)`                   | swap       | Fixed formula, the same on every VM                                                                  |
| `/var`                             | ≥20 % of the disk, or 20 GB minimum | ext4       | Separate from `/`: runaway growth of logs, containerd images or local volumes does not fill the root |
| `/`                                | Rest of the disk                    | ext4       | Base system                                                                                          |

## Why separate `/var`

K3s workloads (containerd, logs, image cache, `local-path-provisioner`/OpenEBS if used inside the VM) all write under `/var`. Without a separate partition, if that grows out of control it can fill `/` and bring down the whole node, not just the workload. On already installed bare metal, separating `/var` requires reinstalling; on a new CAPN VM it costs nothing to define it in the template from day one.

## Applying it

The scheme belongs in the base image or template that CAPN uses to provision the VMs, not in Ansible after installation. This repo does not have that template yet.

# Hardware discovery (Ansible)

Playbook that scans the 3 nodes and produces one Markdown sheet per node, to confirm (or contradict) with measured data the roles assigned in the [Hardware inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md) — in particular, whether `deborah` is still the best candidate for the K3s (lightweight Kubernetes distribution) control plane.

Code: [`ansible/playbook-discovery.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-discovery.yml) (two plays in the same file: per-node scan + summary).

## Usage

```
cd ansible/
ansible-playbook -i inventory.ini playbook-discovery.yml
```

The second play (summary) reads the facts collected by the first one from `hostvars`, which are only available in memory during the same `ansible-playbook` invocation -- that is why they live in the same file instead of in a separate playbook you would have to remember to pass along with the other one.

## What it collects per node

- **CPU** — architecture, model, physical/logical cores.
- **Memory** — total RAM and swap.
- **Storage** — each disk (rotational or not, size, model) + a 4k write benchmark (`fio`, or `dd` as a fallback) to estimate suitability for etcd.
- **Network** — interfaces with their negotiated speed.
- **Operating system** — distro, kernel, uptime, temperature (`vcgencmd` sensor on an SBC (Single Board Computer) or generic `thermal_zone0`).
- **Installed stack** — whether `/dev/kvm` is available, Incus and K3s versions.
- **Key packages** — version (or "no instalado") of docker, containerd, python3, git, qemu-img, virsh.
- **Incus instances** — name/type/state of each container or VM (virtual machine).
- **K3s** — `kubectl get nodes` and `get pods -A` (if the node has access to the API server).
- **Loose containers** — `crictl ps -a` / `docker ps -a`, to detect drift outside K3s/Incus.
- **systemd services** — active/enabled state of k3s, k3s-agent, incus, docker, containerd.
- **Listening ports** (`ss -tlnp`), **disk usage per filesystem** (`df -h`), **kernel modules** (overlay/br_netfilter/veth) and **firewall state** (ufw or nftables, whichever exists).

## Output

```
ansible/reports/            (gitignored -- datos de un escaneo en vivo)
├── invincible.md
├── oliver.md
├── deborah.md
└── summary.md
```

`summary.md` puts the 3 nodes side by side and lists the 4 criteria used to decide the control plane role:

1. Lowest write latency (fsync) — the decisive factor for etcd.
1. Stable power supply (NVMe + dedicated power, not USB).
1. RAM headroom above normal cluster usage (not the decisive factor on its own).
1. Temperature under sustained load — thermal throttling on an SBC.

## How to read a sheet

This is a **real** excerpt of the `invincible` sheet, trimmed to the hardware part. The full sheet also lists ports, services and containers, which is why it is not published.

```
## CPU
Arquitectura     x86_64
Modelo           Intel(R) Core(TM) i7-8700T CPU @ 2.40GHz
Núcleos físicos  6          Núcleos lógicos  12

## Memoria
RAM total        15.5 GB (15834 MB)          Swap total  0 MB

## Almacenamiento
sda   SSD   223.6GB   KINGSTON SA400S3   En uso
Benchmark I/O (escritura 4k): 40960000 bytes (41 MB) copied, 0.52 s, 78.4 MB/s
```

| Field                          | What it means and what it is for                                                                                                                                                                                                                         |
| ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Physical and logical cores** | Logical cores count threads (hyper-threading): 6 physical give 12 logical. Kubernetes and Incus schedule by logical cores.                                                                                                                               |
| **Total RAM**                  | Given in GB that are GiB (15834 MB ÷ 1024 = 15.5). With 0 MB of swap, if RAM runs out the kernel kills processes.                                                                                                                                        |
| **Type**                       | `HDD (rotacional)` is a mechanical disk: much slower and unsuitable for etcd. It also tells apart `SSD`, `NVMe`, `eMMC/SD` (the memory of ARM boards), `flash SPI` and `zram` (compressed RAM, which is not a disk). There are no mechanical disks here. |
| **4k write benchmark**         | Writes 40 MB in 4 KB blocks **directly to the disk**. It is for comparing nodes with each other, not an absolute figure.                                                                                                                                 |

The benchmark is not the fsync latency

What matters for etcd is the **fsync latency** (how long each write takes to be confirmed), measured with `fio --fsync=1`. The 4k `dd` gives an idea of the order of magnitude: 59.5 MB/s on `deborah`'s eMMC versus 78.4 MB/s on `invincible`'s SSD. A figure very different from another run is almost always a badly done test (a cache in between), not a faster disk.

If any result contradicts the role assigned in the [Hardware inventory](https://homelab.maceo.cl/en/hardware/inventory/index.md), that document is the one that gets updated — the reports in `ansible/reports/` are not versioned.

# Hardware Inventory

> The hostnames follow the *Invincible* series — see [character category](https://invincible.fandom.com/es/wiki/Categor%C3%ADa:Personajes). `deborah` was previously `orangepi5plus`.

Data **measured** with [`playbook-discovery.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-discovery.yml). Where the nominal value differs from the measured, the *Measured* column is the source of truth for planning.

`invincible` x86_64

Lenovo M920q · i7-8700T 6c/12t

RAM: 15.5 GB

Static IP: 192.168.20.6

Incus: leader · K3s: Worker (agent)

`oliver` x86_64

Lenovo M700 · i3-6100T 2c/4t

RAM: 7.7 GB

Static IP: 192.168.20.7

Incus: Member (manual scheduler) · K3s: Worker (agent)

`deborah` ARM64

Orange Pi 5 Plus · RK3588 8c

RAM: 31.0 GB

Static IP: 192.168.20.5

Incus: Member (arm64 group) · K3s: Control-plane (server)

Nominal specs vs measured

Measured on the nodes themselves with `playbook-discovery.yml` (and verified with `lscpu`, `free`, `dmidecode`, and `lsblk`); RAM is given in GiB, as reported by discovery: **invincible** is the M920q with i7-8700T and 15.5 GB, and **oliver** is the M700 with i3-6100T and 7.7 GB.

## IP Scheme

| Concept                       | invincible     | oliver         | deborah        |
| ----------------------------- | -------------- | -------------- | -------------- |
| SSH IP today (`ansible_host`) | `192.168.20.6` | `192.168.20.7` | `192.168.20.5` |
| Static IP in `br0`            | `192.168.20.6` | `192.168.20.7` | `192.168.20.5` |
| Physical interface → bridge   | `eno2`         | `eno1`         | `enP4p65s0`    |

After `playbook-set-static-ip.yml`, update `ansible_host` in [`inventory.ini`](https://github.com/maceosoft/homelab/blob/main/ansible/inventory.ini) if the connection IP changed.

## Storage and I/O

| Node       | Main Disk                   | 4k Write (dd/fio) |
| ---------- | --------------------------- | ----------------- |
| invincible | Kingston SA400 224GB        | **78.4 MB/s**     |
| oliver     | TECLAST 120GB SSD           | 73.9 MB/s         |
| deborah    | eMMC `mmcblk1` 233GB (root) | 59.5 MB/s         |

Test: `dd` of 40 MB in 4k blocks with direct write (`oflag=direct`). The NVMe WDC 250GB (`nvme0n1`) of `deborah` **is not detected**: `lsblk` and `lspci` do not see it and `dmesg` says `rk-pcie … PCIe Link Fail` (the slot link does not come up).

## RAM Budget (recalculated)

- **invincible (15.5 GB)**: K3s (lightweight Kubernetes distribution) agent + Argo CD controller + Incus leader. It is the x86 node with the most margin and the fastest disk: it supports CAPN (Cluster API Provider for Incus) management and Longhorn simultaneously.
- **oliver (7.7 GB)**: K3s agent (CoreDNS, metrics-server) + Incus in `scheduler.instance manual`. RAM adjusted: that's why it remains a light node and, in the design, outside of Longhorn (see [Longhorn v1](https://homelab.maceo.cl/en/storage/#longhorn-v1-selective-ha-cross-arch)).
- **deborah (31 GB)**: K3s control plane + BIND (role `bind_dns`) + Ollama on demand (role `ollama_ondemand`, group `llm_nodes`, alongside `invincible` and `oliver`): it only uses RAM while responding and releases it about 5 minutes later (see [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md)). Root on eMMC; the NVMe (SSD via PCIe) is not detected (see [Storage and I/O](#storage-and-io)).

## Decision: control plane on deborah

Moving the control plane is only an option

The control plane stays on `deborah`. If it ever makes sense to move it to another node (for example, to `oliver`), the procedure is in [Moving the control plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md); it is not planned today.

**Status:** the API server responds on `192.168.20.5:6443` (deborah). Workers on invincible (`192.168.20.6`) and oliver (`192.168.20.7`). Installed with **K3s v1.36.5+k3s1** (`k3s_install.core.version`); afterwards the SUC (System Upgrade Controller) updates it automatically ([Updating K3s](https://homelab.maceo.cl/en/operacion/actualizar-k3s/index.md)). Actual version: `kubectl get nodes`.

Trade-off Analysis

A: deborah chosen

31 GB RAM, ARM64 (root on eMMC; NVMe not detected)

SBC; thermal throttling; extra services on the same node

B: invincible

I/O 78.4 MB/s (the best measured); 15.5 GB RAM; i7-8700T

It is the Incus leader; costly control plane migration

C: oliver

Node already reserved for quorum

Only 7.7 GB RAM; I/O 73.9 MB/s

**Discovery note:** without NVMe, `deborah` measures the lowest I/O (59.5 MB/s on eMMC) and `invincible` the highest (78.4 MB/s); the report itself suggests considering `invincible` as the control plane. The decision to keep `deborah` is maintained until fsync latency (`fio`) is measured, which is what matters for etcd.

deborah after reboot

If after a reboot `deborah` has the `k3s` binary but no active service or port `:6443`, check `sudo systemctl enable --now k3s` and the logs (`sudo journalctl -u k3s -n 50`).

## See also

- [Hardware discovery (Ansible)](https://homelab.maceo.cl/en/hardware/discovery/index.md)
- [K3s — installation and deployment](https://homelab.maceo.cl/en/k3s/index.md)

# Network

Part of the implementation guide

Run this phase with **[Phase 1 — Network](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md)** and `playbook-set-static-ip.yml`.

## L2 Bridge to the Physical LAN

The **3 nodes** use a Linux bridge (`br0`) over the physical interface, so that Incus instances receive IP directly from the LAN (local network) and are reachable between hosts without needing OVN (Open Virtual Network).

| Node       | Physical Interface | IP on br0 (static) |
| ---------- | ------------------ | ------------------ |
| invincible | `eno2`             | 192.168.20.6       |
| oliver     | `eno1`             | 192.168.20.7       |
| deborah    | `enP4p65s0`        | 192.168.20.5       |

> Confirm with `ip -br link` on the node before applying — the name of the physical interface may differ from this table after an OS update.

Example netplan — invincible:

```
# /etc/netplan/br0.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    eno2:
      dhcp4: no
  bridges:
    br0:
      interfaces: [eno2]
      addresses: [192.168.20.6/22]
      routes:
        - to: default
          via: 192.168.20.1
      nameservers:
        addresses: [192.168.20.5, 192.168.20.1]
```

Example netplan — deborah (same subnet, different interface and last octet):

```
# /etc/netplan/br0.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    enP4p65s0:
      dhcp4: no
  bridges:
    br0:
      interfaces: [enP4p65s0]
      addresses: [192.168.20.5/22]
      routes:
        - to: default
          via: 192.168.20.1
      nameservers:
        addresses: [192.168.20.5, 192.168.20.1]
```

```
ip -br link
# Debian no lo trae instalado por defecto
sudo apt install -y netplan.io
# corta SSH si cambias de IP
sudo netplan apply
```

The Ansible playbook installs `netplan.io`, enables `systemd-networkd` and migrates the NIC (network interface card) out of ifupdown before writing `/etc/netplan/br0.yaml`.

```
ansible-playbook -i inventory.ini playbook-set-static-ip.yml --limit deborah
```

Incus profile to use the bridge in the instances:

```
incus profile device add default eth0 nic nictype=bridged parent=br0
```

## Fixed IP — why it is mandatory here

1. **Incus cluster**: `cluster.https_address` is recorded by IP when joining.
1. **K3s (lightweight Kubernetes distribution)**: kubelet uses the node IP as identity.
1. **CAPN (Cluster API Provider for Incus)**: the `Secret` `lxc-secret` points to `https://<IP>:8443` as a fixed string.

## IP Scheme

| Node       | Current IP (DHCP/pre-bridge) | IP on br0    |
| ---------- | ---------------------------- | ------------ |
| invincible | 192.168.23.162               | 192.168.20.6 |
| oliver     | 192.168.20.87                | 192.168.20.7 |
| deborah    | 192.168.21.211               | 192.168.20.5 |

> The LAN is a `/22` (`192.168.20.0`–`192.168.23.255`), not a `/24` — the "current" IPs in this table are DHCP leases within that same range (not separate subnets). The source of truth at all times is `ansible_host` in `inventory.ini`.

### LAN Addressing (`192.168.20.0/22`, gateway `192.168.20.1`)

The router's DHCP assigns **`192.168.20.31` to `192.168.23.191`**. Everything static lives **outside** that range:

| Range                             | Use                                                                    | Where Configured                                                                                                |
| --------------------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| `192.168.20.1`                    | Gateway (router)                                                       | Router                                                                                                          |
| `192.168.20.4`                    | K3s API VIP (kube-vip, optional)                                       | `kube_vip.address` in `k3s_install.yml`                                                                         |
| `192.168.20.5`, `.6`, `.7`        | deborah, invincible, oliver                                            | `static_ip` in `inventory.ini`                                                                                  |
| `192.168.20.31 – 192.168.23.191`  | **Router DHCP**                                                        | Router                                                                                                          |
| `192.168.23.192 – .199`           | Free                                                                   | —                                                                                                               |
| `192.168.23.200 – .220`           | **MetalLB** (`LoadBalancer`, Gateway; the Kong Gateway uses `.200`)    | [`gitops/metallb/ipaddresspool.yaml`](https://github.com/maceosoft/gitops/blob/main/metallb/ipaddresspool.yaml) |
| `192.168.23.221 – .223`           | Free                                                                   | —                                                                                                               |
| `192.168.23.224/28` (`.224–.239`) | **Incus LB** (OVN), uplink `ipv4.routes`                               | `incus_ovn_lb_routes` in `group_vars/incus_cluster/ovn.yml`                                                     |
| `192.168.23.240 – .249`           | **External OVN**, `ipv4.ovn.ranges` (a virtual router per OVN network) | `incus_ovn_external_ranges` in the same file                                                                    |
| `192.168.23.250 – .255`           | Free (`.255` is the broadcast of the `/22`)                            | —                                                                                                               |

Outside the LAN, private networks that do not overlap with it or each other:

| Network                           | Use                                        |
| --------------------------------- | ------------------------------------------ |
| `10.42.0.0/16` and `10.43.0.0/16` | K3s Pods and Services                      |
| `192.168.19.0/24`                 | Internal OVN network (`ovn-lan`, with NAT) |

Outside the DHCP Range

Fixed IPs, the MetalLB pool, and OVN segments must remain **outside the DHCP range** (`192.168.20.31–192.168.23.191`). If not, the router may assign them to another device, causing conflicts. The preflight check for the role `incus_ovn` fails if its segments overlap with DHCP or MetalLB.

After `playbook-set-static-ip.yml`, update `ansible_host` in [`inventory.ini`](https://github.com/maceosoft/homelab/blob/main/ansible/inventory.ini) to the value of `static_ip` if the connection IP changed.

**This decision in other environments**

[MetalLB](https://metallb.io/) takes free IPs from the LAN (`.200–.220`) for K3s Services `LoadBalancer`, and all web traffic enters through a single Kong Gateway ([Gateway API](https://gateway-api.sigs.k8s.io/)), with [Calico](https://docs.tigera.io/calico/latest/about/) as CNI. No need to buy a load balancer: just reserve a range outside the DHCP.

- **Benefits:** MetalLB provides IPs of `LoadBalancer` on your own servers without extra hardware, and Gateway API is a standard: routes (`HTTPRoute`) are written the same even if you switch Kong for another controller.
- **Dedicated load balancers:** many data centers already have hardware or virtual load balancers, such as [F5 BIG-IP](https://www.f5.com/products/big-ip) or [NetScaler](https://www.netscaler.com/), and the network team prefers the cluster to use them instead of announcing IPs on its own.
- **BGP instead of L2:** MetalLB in [BGP](https://metallb.io/concepts/bgp/) mode announces IPs to data center routers and distributes traffic among nodes; the L2 mode of the HomeLab sends all traffic through one node at a time.
- **Commercial support:** there are versions with support for the same components, such as [Kong Konnect](https://konghq.com/products/kong-konnect) for the gateway and [Calico Enterprise](https://www.tigera.io/tigera-products/calico-commercial-editions/) for networking and its policies.
- **When it still makes sense:** on bare metal servers, branches, or disconnected sites, where there is no cloud load balancer and MetalLB plus a gateway inside the cluster is the simplest to operate.

| Cloud              | Equivalent Service                                                                                                                                                                                                                                                |
| ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCP                | [Cloud Load Balancing](https://cloud.google.com/load-balancing) with the [GKE Gateway controller](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/gateway-api)                                                                                      |
| AWS                | [Elastic Load Balancing](https://aws.amazon.com/elasticloadbalancing/) with the [AWS Load Balancer Controller](https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/guide/gateway/gateway/) (supports Gateway API)                               |
| Azure              | [Application Gateway for Containers](https://learn.microsoft.com/en-us/azure/application-gateway/for-containers/overview) (Gateway API) and [Azure Load Balancer](https://learn.microsoft.com/en-us/azure/aks/load-balancer-standard) for `LoadBalancer`          |
| Oracle Cloud (OCI) | [OCI Load Balancer](https://docs.oracle.com/en-us/iaas/Content/Balance/home.htm) with the [OCI Native Ingress Controller](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengsettingupnativeingresscontroller.htm) (uses `Ingress`, not Gateway API) |

In the cloud, you do not need MetalLB: each Service `LoadBalancer` or `Gateway` creates a managed load balancer, with high availability and integrated certificates. In return, each load balancer is billed separately and its configuration uses annotations or CRDs specific to each cloud; in OCI, if you want Gateway API, you install your own controller (like Kong) behind the load balancer.

### Ollama: router and nodes

The AI assistant has **one single entry point**: the router `ollama-router` (namespace `ai`). Cluster clients use the Service `http://ollama-router.ai.svc`; external clients, `https://ollama.homelab.local` (Kong, only HTTPS; the name is in the `homelab.local` zone of BIND). The router requires a Bearer token.

Each Ollama node (`deborah` `192.168.20.5`, `invincible` `192.168.20.6`, `oliver` `192.168.20.7`, and when powered on, the Mac) has a **socket on port `11434`** that wakes up Ollama; internally, Ollama listens only on `127.0.0.1:11435`. The socket listens **only on the node's IP** (variable `ollama_ondemand_listen`) and the node's firewall allows the port from K3s nodes and from pods (the router can be on the same node). The Ollama node does not authenticate: inference protection is the router's token plus NetworkPolicy in namespace `ai` (which are activated in stages). Details in [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md) and [Add a Node to the Ollama Cluster](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md).

### K3s API Names

Agents, kubeconfig, and certificates use the name `k3s.homelab.local` for the API (port `6443`) instead of a specific node's IP. This way, the control plane can move without affecting clients ([procedure](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md)).

## Exposed Ports and Host Firewall

Each node has a host firewall with `nftables` (role `host_firewall`): its own table, evaluated before kube-proxy and not modifying Calico chains. The policy is `accept` with explicit denials (to avoid breaking K3s, Calico, Longhorn, or MetalLB if they change ports), and traffic between nodes is trusted. The following is restricted:

- SSH (`22`) from the LAN, with authentication via Dex.
- The K3s API (`6443`), kubelet (`10250`), BGP traffic (`179`), FRR (`2623`), and MetalLB (`7946`) traffic, and Typha (`5473`): only between nodes and the pod range.
- Metrics (`9100`, `9120`, `9140`, `9141`): only to the collector and nodes.
- Incus (`8443`): nodes, pods, and operator. Ollama (`11434`): K3s nodes, pods, and machines added to `host_firewall_ollama_extra_cidrs` (the Mac).
- DNS (`53`) from the LAN.

The list of what remains without its own authentication, with the reason and the network control that replaces it, is in [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/#network-controlled-exceptions). The rules are applied with a safety auto-reversal (see [Post-Incident Hardening](https://homelab.maceo.cl/en/security/hardening-post-incidente/index.md)).

# Incus Cluster

Part of the implementation guide

Follow **[Phase 2 — Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md)** for the full order. **Ansible** runs bootstrap, join, cluster groups, scheduler, and UI (user interface) via [`playbook-incus-cluster.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-incus-cluster.yml). The manual blocks below are reference.

## Automation (Ansible)

```
cd ansible
ansible-playbook -i inventory.ini playbook-bootstrap.yml
ansible-playbook -i inventory.ini playbook-incus-cluster.yml
```

Roles: [`incus_cluster`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_cluster/README.md), [`incus_ui`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_ui/README.md).

## Storage

Default HomeLab: driver **`dir`**, pool **`local`**, path `/var/lib/incus/storage-pools/local`. Ansible validates ≥ **20 GB** free before init (`incus_storage_min_free_gb` in [`group_vars/incus_cluster/vars.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/vars.yml)).

| Aspect            | Default                               |
| ----------------- | ------------------------------------- |
| Driver            | `dir`                                 |
| Pool              | `local` (one per member)              |
| Path              | `/var/lib/incus/storage-pools/local`  |
| Profile `default` | root on `local`, NIC bridged to `br0` |

### Per node

| Node       | Note                                                                                       |
| ---------- | ------------------------------------------------------------------------------------------ |
| invincible | 224 GB SSD; leader + UI                                                                    |
| oliver     | 120 GB SSD; local pool required; manual scheduler                                          |
| deborah    | eMMC boot; optional override to NVMe: `incus_storage_path: /srv/incus/storage-pools/local` |

### `dir` limitations

- No pool-level CoW; instance snapshots and export tarball work.
- VMs (virtual machines) slower than with `zfs`/`btrfs`.
- Each member has a local pool; no Ceph by default.

### Alternatives

| Driver  | When                     | Variable                                       |
| ------- | ------------------------ | ---------------------------------------------- |
| `dir`   | **Default** HomeLab      | `incus_storage_driver: dir`                    |
| `btrfs` | Efficient snapshots      | `incus_storage_driver: btrfs` + disk/subvolume |
| `zfs`   | VMs + advanced snapshots | `incus_storage_driver: zfs`                    |
| `lvm`   | Thin LVM for VMs         | `incus_storage_driver: lvm`                    |

Trade-off analysis

Ansible (`playbook-incus-cluster.yml`)

Idempotent bootstrap/join/UI

Requires prior playbook (`incus_install`)

Manual

Per-node debugging

Single-use tokens

`dir`

No dedicated disk

Slow VMs; no pool CoW

`btrfs` / `zfs`

Efficient snapshots

Disk/loop; migration if already clustered

UI via Ansible

Same playbook as cluster

Only on invincible (native daemon)

Cluster already initialized

Changing driver requires manual migration; choose before the first playbook.

Docs: [Storage pools](https://linuxcontainers.org/incus/docs/main/explanation/storage/)

## Bootstrap (invincible) — manual

```
curl https://pkgs.zabbly.com/get/incus-stable | sudo bash -x
sudo incus admin init
# clustering=yes, no te unes (bootstrap), storage backend dir
```

## Join (oliver and deborah) — manual

```
# ejecutar en invincible; genera token de un solo uso
incus cluster add oliver
# otro token distinto
incus cluster add deborah
```

On each node: install Incus and `incus admin init`, answering yes to join, with the corresponding token.

## Cluster groups (architecture)

```
incus cluster group create x86-nodes
incus cluster group create arm64-nodes

incus cluster group assign invincible x86-nodes,default
incus cluster group assign oliver x86-nodes,default
incus cluster group assign deborah arm64-nodes,default
```

## Protect oliver RAM (manual scheduler)

oliver has 7.7 GB measured, so it stays as a light node for Incus quorum without automatic instances:

```
incus cluster set oliver scheduler.instance manual
```

Excludes `oliver` from automatic scheduling; it only receives instances if explicitly targeted with `--target=oliver`. Still counts as a quorum vote.

## Virtual machines (VM)

### Which nodes can run VMs

A VM needs KVM (Kernel-based Virtual Machine, hardware virtualization by the kernel), the kernel module `vhost_vsock` and the Incus driver `qemu`. Per node status:

| Node       | Architecture | Runs VMs? | Reason                                                                                                 |
| ---------- | ------------ | --------- | ------------------------------------------------------------------------------------------------------ |
| invincible | x86_64       | Yes       | Driver `lxc` + `qemu`, `vhost_vsock` available                                                         |
| oliver     | x86_64       | Yes       | Same as invincible                                                                                     |
| deborah    | ARM64        | **No**    | Its vendor kernel (5.10, Rockchip) was built without `vhost_vsock`, and its Incus driver is only `lxc` |

If a VM is created targeting `Auto` and the scheduler picks deborah, it fails with `Instance type "virtual-machine" is not supported on this server: vhost_vsock kernel module not loaded`. Always specify the node:

```
incus launch images:debian/12 mi-vm --vm --target invincible
```

In the UI, on the create form, pick the image first (until then **Instance type**, **Target**, and **Profiles** are disabled), then **Virtual machine**, and in **Target** an x86 node (invincible or oliver).

### `cloud` images and the agent disk

Incus VMs are controlled from outside via `incus-agent`, a program that runs inside the VM and enables `incus exec`, console, file copy, IP visibility, and clean VM shutdown. Incus passes it to the VM as a read-only virtual disk named `agent:config`, which carries the binary, its certificates, and the service that starts it.

With a regular image (e.g., `almalinux/8`) Incus adds it automatically. The `cloud` variants (e.g., `almalinux/8/cloud`, designed for cloud-init, which configures user, SSH keys, and network on boot) require adding it manually. If missing, the VM won't boot:

```
Error: This virtual machine image requires an agent:config disk be added
```

The `default` profile doesn't include it (nor should it: it's shared by containers and `agent:config` only applies to VMs). Add it to the VM:

**Instances → (the VM) → Configuration → YAML configuration**, and inside `devices:` add

```
devices:
  agent:
    type: disk
    source: agent:config
```

Save with **Save changes** and click **Start**.

```
incus config device add mi-vm agent disk source=agent:config
incus start mi-vm
```

Or, if you use `cloud` images often, create a separate profile and apply it at launch:

```
incus profile create vm-agent
incus profile device add vm-agent agent disk source=agent:config
incus launch images:almalinux/8/cloud mi-vm --vm -p default -p vm-agent --target invincible
```

## Admin UI

Incus exposes a **native web UI** (`incus-ui-canonical`) to manage instances, profiles, networks, and the cluster. It does not use kubeconfig; initial authentication is by **client certificate** and, after Phase 4, by **OIDC (OpenID Connect)** (Dex → GitHub).

### Installation (Ansible or manual)

Ansible installs the UI on **invincible** (`incus_ui`, tag `ui`). The UI is **not** a container: it is served by the Incus daemon on `core.https_address`.

Included in `playbook-incus-cluster.yml` (tag `ui`).

On **invincible**:

```
sudo apt install incus-ui-canonical
sudo incus config set core.https_address=:8443
```

`/etc/hosts` on your workstation — full block in [HomeLab Overview](https://homelab.maceo.cl/en/implementacion/resumen-homelab/#dns-on-your-workstation):

Initial access: [`https://incus.homelab.local:8443`](https://incus.homelab.local:8443) → client certificate.

UI

Ansible (tag `ui`)

Included in `playbook-incus-cluster.yml`

Only on invincible

Manual (`apt` + `config set`)

Without Ansible

Easy to drift from the rest of the cluster

### OIDC via Dex (Phase 4)

With OIDC configured, the UI login screen offers **Login with SSO** (single sign-on) (GitHub via Dex) and **Login with TLS** (Transport Layer Security) (client certificate):

View screenshot: Incus UI access

After [Phase 4 — Incus UI OIDC](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#incus-ui-oidc), with the HomeLab CA (certificate authority) already installed on the node (see the warning below):

```
incus config set oidc.issuer=https://argocd.homelab.local/api/dex
incus config set oidc.client.id=incus-ui
```

Incus (7.x) only knows five OIDC keys: `oidc.issuer`, `oidc.client.id`, `oidc.audience`, `oidc.claim`, and `oidc.scopes`. There are no `oidc.client.secret` or `oidc.groups.claim`: Incus does not send a secret, so in Dex the client `incus-ui` is **public** (PKCE —Proof Key for Code Exchange—, `public: true`).

Incus must trust the HomeLab CA

The Dex certificate is signed by the HomeLab's own CA (cert-manager). If the node does not know it, Incus cannot validate the identity provider and login fails. The playbook (`--tags incus`, or just `--tags incus_ca`) installs it in `invincible` and restarts `incus`. Manually, on the node:

```
kubectl -n cert-manager get secret homelab-ca -o jsonpath='{.data.ca\.crt}' | base64 -d \
  | sudo tee /usr/local/share/ca-certificates/homelab-ca.crt > /dev/null
sudo update-ca-certificates
sudo systemctl restart incus
```

Verify with `curl -s -o /dev/null -w "%{http_code} ssl_verify=%{ssl_verify_result}\n" https://argocd.homelab.local/api/dex/.well-known/openid-configuration` (must return `200 ssl_verify=0`). Details in [Phase 4](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#incus-ui-oidc).

### OIDC user permissions

Incus 7.x does **not** ship with `incus auth` (groups and `identity-provider-group` are from LXD —the project Incus forked from—). With OIDC:

| Scenario                                   | Result                                                                            |
| ------------------------------------------ | --------------------------------------------------------------------------------- |
| Member of team `devops` in org `maceosoft` | Dex allows them through and, with Incus default config, they have **full access** |
| Account not in that team                   | Dex rejects the login: never reaches Incus                                        |

Access control is therefore membership in team `devops` (filter `orgs`/`teams` of the GitHub connector in `dex.config`). For granular permissions (per project, read-only) Incus uses [OpenFGA](https://linuxcontainers.org/incus/docs/main/authentication/), which is not configured in the HomeLab.

References:

- [Incus OIDC](https://linuxcontainers.org/incus/docs/main/authentication/#openid-connect-oidc-authentication)

## Load balancing and OVN

Incus has network load balancers (`incus network load-balancer`) and port forwards (`incus network forward`), but **network load balancers only exist on OVN networks** (Open Virtual Network, software-defined virtual network). In the HomeLab **OVN** is deployed (see [Deploy OVN and segments](#deploy-ovn-and-segments)): `br0` remains an unmanaged bridge for existing instances, and the uplink network `UPLINK` and the OVN network `ovn-lan` are created, making network load balancers available.

`lb01` is not the Incus load balancer

The instance `lb01` appearing in `incus list` is an `nginx` OCI container (for testing). The real load balancing in the HomeLab lives in other layers:

| What is balanced                                  | Who does it                                                                                | Where                                                                             |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| Kubernetes applications (Services `LoadBalancer`) | **MetalLB**, with the `192.168.23.200–.220` IPs (the Kong Gateway uses `.200`)             | [GitOps Catalog](https://homelab.maceo.cl/en/gitops/index.md)                     |
| Control plane of Cluster API clusters             | **CAPN**, with its own load balancer (`LOAD_BALANCER='lxc: {}'`, a container inside Incus) | [Phase 6 — CAPN](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md) |
| K3s API                                           | The control plane IP (`kube-vip` is optional and disabled)                                 | [Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)   |

### Should you enable OVN?

What OVN requires and what it implies for the HomeLab. The segments and deployment are in [Deploy OVN and segments](#deploy-ovn-and-segments).

| OVN Requirement                                                                                  | Status in HomeLab                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Packages `ovn-central` (3 nodes, highly available database), `ovn-host` and `openvswitch-switch` | They are on Debian on all three nodes, with OVN 25.03 and Open vSwitch 3.5 on Debian 13 (`invincible`, `oliver`); `deborah` remains on Debian 12 until [updating it](https://homelab.maceo.cl/en/operacion/actualizar-debian-deborah/index.md), and until then its versions do not match |
| Shared Layer 2 network between nodes                                                             | Yes: all three are on the same LAN via `br0`                                                                                                                                                                                                                                             |
| Uplink: an unmanaged bridge or an unused NIC                                                     | `br0` works; no need to redo it                                                                                                                                                                                                                                                          |
| Reserved IP range outside DHCP for OVN router and virtual IPs                                    | Defined: `192.168.23.224/28` (LB) and `192.168.23.240–.249` (external OVN), outside DHCP and MetalLB. See [Deploy OVN and segments](#deploy-ovn-and-segments)                                                                                                                            |
| Instances inside an OVN network                                                                  | Instances hanging from `br0` (such as `lb01`, `test` and `vm01`) have LAN IPs: to use OVN they must be connected to `ovn-lan`                                                                                                                                                            |
| Memory per node (OVS, `ovn-controller` and central database)                                     | `oliver` has 7.7 GB: advisable to measure OVN consumption there                                                                                                                                                                                                                          |

| Pros                                                        | Cons                                                                                        |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Native load balancers and forwards in Incus                 | Three new components per node and a distributed database to operate                         |
| Isolated virtual networks per project or cluster, with ACLs | Must reserve a new IP range and move instances that want to use them                        |
| `LOAD_BALANCER: ovn` in CAPN for Phase 6 clusters           | Only benefits instances that move to an OVN network; K3s runs on hosts and does not benefit |

### Deploy OVN and segments

The role [`incus_ovn`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_ovn/README.md), within [`playbook-incus-cluster.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-incus-cluster.yml), installs OVN on the three nodes (the highly available database, with `invincible` creating it and the others joining), configures Incus and creates the *uplink* network over `br0` and an OVN network. It is **safe by default**: without `-e incus_ovn_confirm=true` it only runs the preflight (Incus version, packages, free memory, that `br0` exists and that reserved IPs do not respond) and installs nothing. With `incus_ovn_enabled: false` (the default value) the cluster playbook does not even touch it.

| Segment                                      | Value                           | Where configured                        | Variable                    |
| -------------------------------------------- | ------------------------------- | --------------------------------------- | --------------------------- |
| **LB** (IP of load balancers and forwards)   | `192.168.23.224/28`             | `ipv4.routes` of the uplink             | `incus_ovn_lb_routes`       |
| **External OVN** (one IP per virtual router) | `192.168.23.240–192.168.23.249` | `ipv4.ovn.ranges` of the uplink         | `incus_ovn_external_ranges` |
| **Internal OVN** (private subnet, with NAT)  | `192.168.19.1/24`               | `ipv4.address` of the `ovn-lan` network | `incus_ovn_internal_cidr`   |

The first two are from the LAN (`192.168.20.0/22`) and must be **outside the router's DHCP** (`192.168.20.31–192.168.23.191`) and the MetalLB pool (`192.168.23.200–.220`). The allocation of the high free zone of the LAN:

| Range                             | Use                                                           |
| --------------------------------- | ------------------------------------------------------------- |
| `192.168.23.192–.199`             | Free                                                          |
| `192.168.23.200–.220`             | **MetalLB** (21 IPs; Kong Gateway uses `.200`)                |
| `192.168.23.221–.223`             | Free                                                          |
| `192.168.23.224/28` (`.224–.239`) | **Incus LB** (16 IPs)                                         |
| `192.168.23.240–.249`             | **External OVN** (10 IPs, one virtual router per OVN network) |
| `192.168.23.250–.255`             | Free (`.255` is the broadcast of the `/22`, not used)         |

The preflight **fails if segments overlap** with DHCP, with MetalLB, with each other, or with the broadcast. The internal one is private and does not touch the LAN: it also does not overlap with K3s ranges (`10.42.0.0/16` and `10.43.0.0/16`). Current instances (`lb01`, `test`, `vm01`) remain on `br0`: only those you connect to `ovn-lan` use OVN.

```
cd ansible && source ../.venv/bin/activate
# preflight
ansible-playbook -i inventory.ini playbook-incus-cluster.yml --tags ovn --skip-tags always -e incus_ovn_enabled=true
# instala
ansible-playbook -i inventory.ini playbook-incus-cluster.yml --tags ovn --skip-tags always -e incus_ovn_enabled=true -e incus_ovn_confirm=true
```

Then, a network load balancer is created with `incus network load-balancer create ovn-lan 192.168.23.225` and its configuration. The listen address must be in the `ipv4.routes` of the uplink and not overlap with another network in use.

**Verify deployment:** `ovn-central`, `ovn-controller` and Open vSwitch active on all three nodes; the Northbound database clustered with leader (`ovn-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound`); one Geneve *chassis* per node (`ovn-sbctl show`); and `UPLINK` and `ovn-lan` in `Created` state (`incus network list`).

OVN database is not encrypted

Following the Incus guide, the playbook exposes OVN databases via `tcp` without SSL (Secure Sockets Layer) on the nodes' LAN. Acceptable in a lab; for production you would need to enable SSL in OVN.

**Alternatives without OVN:** MetalLB for applications, the CAPN (Cluster API Provider for Incus) load balancer for new clusters, and a container with HAProxy or nginx plus `keepalived` if a virtual IP is needed for instances.

## Update or uninstall Incus

Both playbooks are **safe by default**: without the confirmation variable they only show what they would do.

| What                                      | Playbook                                                                                                              | Confirmation                      | Effect                                                                                                                 |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Update Incus to latest repository version | [`playbook-upgrade-incus.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-upgrade-incus.yml)     | `-e incus_upgrade_confirm=true`   | Updates `incus*` packages node by node and waits for all three members to be `ONLINE`. Does not delete instances       |
| Uninstall cluster (to rebuild it)         | [`playbook-uninstall-incus.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-uninstall-incus.yml) | `-e incus_uninstall_confirm=true` | **Destructive:** deletes instances, images, managed networks, the pool and packages. Does not touch `br0`, K3s or BIND |

The cluster requires **the same Incus version on all members**: while they differ, the API is blocked on those already updated until the others catch up. That is why the update playbook runs through the three nodes sequentially and verifies at the end. To rebuild the cluster after uninstalling: `playbook-bootstrap.yml` and `playbook-incus-cluster.yml`.

## Backup and snapshots (optional)

They are not part of the default bootstrap. Enable them when you want to protect instances or cluster state. The step-by-step is in [Incus Backups](https://homelab.maceo.cl/en/operacion/backups-incus/index.md).

| Method                      | Scope                               | When to use                                     |
| --------------------------- | ----------------------------------- | ----------------------------------------------- |
| **Instance snapshots**      | One instance, same storage pool     | Quick rollback; does not replace offsite backup |
| **Export tarball**          | Instance or portable volume         | Restorable copy to another pool or server       |
| **DB dump**                 | Incus metadata (networks, profiles) | Lightweight cron addition                       |
| **Tar of `/var/lib/incus`** | Full server                         | DR of bootstrap node                            |

### Scheduled snapshots (per instance)

```
# Diario a las 06:00; expira a los 7 días
incus config set <instancia> snapshots.schedule="0 6 * * *"
incus config set <instancia> snapshots.expiry=7d
incus config set <instancia> snapshots.pattern="{{ creation_date|date:'2006-01-02' }}"
```

Manual snapshot: `incus snapshot create <instancia> <nombre>`

### Export and metadata dump

```
incus export <instancia> /backup/<instancia>.tar.gz
incus admin sql local .dump > /backup/incus-local.sql
incus admin sql global .dump > /backup/incus-global.sql
```

### Limitations

- Snapshots live in the **same storage pool** as the instance; if you lose the disk, you lose snapshots and data.
- In a cluster, schedule backups on the node hosting the instance or use `incus copy --refresh` to another Incus server for offsite copies.
- For CAPN/critical workloads, combine snapshots with periodic export or replication.

Official docs: [Instance backup](https://linuxcontainers.org/incus/docs/main/howto/instances_backup/), [Server backup](https://linuxcontainers.org/incus/docs/main/backup/).

# K3s — Manual Installation (Reference)

Recommended Path

Use **[Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)** with `playbook-k3s.yml`. This page documents manual steps (curl) as an alternative legacy approach.

K3s (lightweight Kubernetes distribution) hybrid cluster across the 3 nodes of the Homelab.

> The Homelab hostnames follow the *Invincible* series — see [characters](https://invincible.fandom.com/es/wiki/Categor%C3%ADa:Personajes).

## Cluster Nodes

| Node       | IP           | K3s Role               |
| ---------- | ------------ | ---------------------- |
| deborah    | 192.168.20.5 | server (control plane) |
| invincible | 192.168.20.6 | agent (worker)         |
| oliver     | 192.168.20.7 | agent (worker)         |

Verification from a node with API (Application Programming Interface) server access:

```
kubectl get nodes -o wide
kubectl get pods -A
```

Residual Traefik

The cluster was installed with `--disable traefik`, but residual pods `svclb-traefik-*` (K3s ServiceLB) persist. Remove them with the role `k3s_cleanup_traefik` or `ansible-playbook playbook-k3s.yml --tags traefik`.

Control-plane after reboot

If `deborah` does not expose `:6443` after reboot, verify: `sudo systemctl enable --now k3s && sudo journalctl -u k3s -n 50`

## 1. Prerequisites (all 3 nodes)

```
# Deshabilitar swap (requerido por K3s)
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# Módulos de red y sysctl para el CNI (Flannel/VXLAN por defecto)
cat <<EOF | sudo tee /etc/modules-load.d/k3s.conf
br_netfilter
overlay
EOF
sudo modprobe br_netfilter overlay

cat <<EOF | sudo tee /etc/sysctl.d/k3s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
```

Disable swap (K3s requires this for predictable scheduling), load the kernel modules needed by the CNI (Container Network Interface) to route traffic between pods, and enable IP forwarding so each node can act as a pod router.

Network

If these nodes run Incus cluster instances (see [Incus Cluster](https://homelab.maceo.cl/en/incus/cluster-setup/index.md)), use the same bridge/cluster group already defined — no need for an additional bridge just for K3s.

Automatable with: `ansible-playbook -i inventory.ini playbook-k3s.yml --tags prereqs`

Bootstrap options (kube-vip, Argo CD controller, labels, …): [`playbook-options.md`](https://homelab.maceo.cl/en/k3s/playbook-options/index.md). GitOps: [`Fase 4 — GitOps`](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md).

## 2. Control Plane Installation (deborah)

```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.5+k3s1 \
  INSTALL_K3S_EXEC="server \
  --disable traefik \
  --disable servicelb \
  --write-kubeconfig-mode 600 \
  --node-ip 192.168.20.5 \
  --tls-san 192.168.20.5 \
  --tls-san deborah" sh -
```

- `--disable traefik/servicelb`: installed later (or replaced by a Gateway API controller + MetalLB) to avoid losing control over versions.
- `--write-kubeconfig-mode 600`: kubeconfig readable only by root (hardened vs 644).
- `--tls-san`: adds the fixed IP to the API server certificate, avoiding TLS (Transport Layer Security) errors if accessed by IP instead of hostname.

Token to join workers:

```
sudo cat /var/lib/rancher/k3s/server/node-token
```

Automatable with: `ansible-playbook -i inventory.ini playbook-k3s.yml --limit k3s_server`

## 3. Joining Workers

**invincible (15.5 GB — also runs CAPN as management cluster):**

```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.5+k3s1 \
  K3S_URL=https://192.168.20.5:6443 \
  K3S_TOKEN=<TOKEN> \
  INSTALL_K3S_EXEC="agent --node-ip 192.168.20.6" sh -
```

**oliver (7.7 GB — lightweight profile, same criteria as its `scheduler.instance manual` in Incus):**

```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.5+k3s1 \
  K3S_URL=https://192.168.20.5:6443 \
  K3S_TOKEN=<TOKEN> \
  INSTALL_K3S_EXEC="agent --node-ip 192.168.20.7 --kubelet-arg=max-pods=110" sh -
```

Automatable with: `ansible-playbook -i inventory.ini playbook-k3s.yml --limit k3s_agent`

## 4. Verification

```
kubectl get nodes -o wide
kubectl get pods -A
```

Label the nodes by architecture and capacity to use `nodeSelector`/`nodeAffinity` in manifests:

```
kubectl label node oliver workload-tier=light
kubectl label node deborah workload-tier=heavy kubernetes.io/arch=arm64
kubectl label node invincible node.longhorn.io/create-default-disk=true
kubectl label node deborah node.longhorn.io/create-default-disk=true
kubectl label node oliver node.longhorn.io/create-default-disk=false
```

Multi-arch

Any container image must support `arm64/v8` (`docker manifest inspect <imagen>`) before scheduling it without `nodeSelector`, or it will fail on `deborah`.

## 5. Storage

See [Storage](https://homelab.maceo.cl/en/storage/index.md) for the details of replicated OpenEBS/Longhorn x86↔arm64. GitOps manifests in [`gitops/storage/`](https://github.com/maceosoft/gitops/tree/main/storage).

## 6. Automated Update Plan

Versioned manifests in [`gitops/k3s-upgrade/`](https://github.com/maceosoft/gitops/tree/main/k3s-upgrade).

### 6.1 K3s (server + agents) — `system-upgrade-controller`

```
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml
kubectl apply -f gitops/k3s-upgrade/
```

`concurrency: 1` is intentional in a 3-node cluster: it never leaves the cluster without full failover capacity. The controller checks the `stable` channel periodically and, upon detecting a new version, cordons and drains one node at a time — `deborah` (control plane) first, then `invincible`/`oliver`.

### 6.2 Base Operating System (all 3 nodes)

In this Homelab, Debian updates **do not install automatically**: each node publishes how many are pending and monitoring alerts via Discord. The procedure is in [Patching the Nodes](https://homelab.maceo.cl/en/operacion/parchear-nodos/index.md).

### 6.3 Incus

Follows the package cycle from the Zabbly repo:

```
sudo apt update && sudo apt upgrade -y incus incus-client
```

If Incus clustering is active, verify that the 3 major versions match before updating — a version mismatch between Incus cluster members can break quorum.

### 6.4 Summary of Cadence

| Component           | Mechanism                                                                                          | Frequency                        | Window                                              |
| ------------------- | -------------------------------------------------------------------------------------------------- | -------------------------------- | --------------------------------------------------- |
| K3s (server/agents) | system-upgrade-controller, stable channel                                                          | Automatic upon detecting release | Continuous, one node at a time, control plane first |
| Base OS             | Metric alert + [Patching the Nodes](https://homelab.maceo.cl/en/operacion/parchear-nodos/index.md) | Upon receiving the alert         | One node at a time                                  |
| Incus               | Manual apt or monthly cron                                                                         | Monthly                          | Scheduled maintenance                               |

## 7. Post-Installation Checklist

- K3s v1.36.5+k3s1 installed (stable channel)
- Workers invincible/oliver joined to API at `192.168.20.5:6443`
- overlay/br_netfilter modules loaded on workers
- Argo CD deployed (application-controller on invincible)
- `kubectl get nodes` shows 3 nodes in `Ready` (verify CP after reboot)
- Architecture, tier, and Longhorn labels applied
- Traefik svclb removed
- `system-upgrade-controller` deployed with both `Plan` applied
- Patch metrics active ([`playbook-apt-metrics.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-apt-metrics.yml))
- kubeconfig backed up outside the cluster (never committed — see `.gitignore`)

# K3s Playbook Options (bootstrap)

Appendix to **[Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)**.

The playbook [`playbook-k3s.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-k3s.yml) installs **only what Argo CD cannot do**: K3s binary (lightweight Kubernetes distribution), kube-vip CP (control plane), Argo CD controller, node labels, etc. Addons (MetalLB, Gateway, Longhorn, vCluster, …) live in the [`gitops`](https://github.com/maceosoft/gitops) repo.

Configuration: [`ansible/group_vars/incus_cluster/k3s_install.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/k3s_install.yml)

Role catalog: [Ansible — roles](https://homelab.maceo.cl/en/ansible/index.md)

```
cd ansible
ansible-playbook -i inventory.ini playbook-k3s.yml
```

## Table A — Playbook Options

| Option             | Variable                               | Tag          | Default         | When to enable                                 |
| ------------------ | -------------------------------------- | ------------ | --------------- | ---------------------------------------------- |
| Core K3s           | `k3s_install.core.*`                   | `core`       | Current HomeLab | Always (server/agent)                          |
| kube-vip CP        | `k3s_install.kube_vip.enabled`         | `kube_vip`   | `false`         | Stable VIP on `:6443`; **no** Service LB       |
| Argo CD controller | `k3s_install.gitops_argocd.enabled`    | `argocd`     | `false`         | GitOps bootstrap; apps in gitops               |
| Node labels        | `k3s_install.node_labels.enabled`      | `labels`     | `false`         | Longhorn/arch/tier labels via `host_vars`      |
| Traefik cleanup    | `k3s_install.traefik_cleanup.enabled`  | `traefik`    | `true`          | After install with `--disable traefik`         |
| Fetch kubeconfig   | `k3s_install.fetch_kubeconfig.enabled` | `kubeconfig` | `false`         | Copy kubeconfig to operator                    |
| OS hardening       | `k3s_install.os_hardening.enabled`     | `hardening`  | `false`         | Staggered `unattended-upgrades`                |
| External CNI       | `k3s_install.core.cni`                 | `cni`        | `flannel`       | Canal, Calico or Cilium (not embedded Flannel) |

Table A

kube-vip

API VIP without depending on CP node IP

Does not replace MetalLB

Argo CD in Ansible

Controller ready before Phase 4

More time in Phase 3

Traefik cleanup

Avoids conflict with Gateway controller

Extra play after agents

fetch_kubeconfig

Kubeconfig on workstation

Local file with credentials

OS hardening

Staggered automatic patches

Scheduled reboots

External CNI

NetworkPolicy (Calico/Cilium) or Canal

Requires `--flannel-backend=none`; `cni` play before agents

### Core (`k3s_install.core`)

| Field                | Default            | Description                                                    |
| -------------------- | ------------------ | -------------------------------------------------------------- |
| `version`            | `v1.36.5+k3s1`     | K3s version (stable channel)                                   |
| `disable_components` | traefik, servicelb | `--disable` on server                                          |
| `cluster_cidr`       | `10.42.0.0/16`     | Pod CIDR                                                       |
| `service_cidr`       | `10.43.0.0/16`     | Service CIDR                                                   |
| `cni`                | `flannel`          | `flannel`, `canal`, `calico`, `cilium`                         |
| `flannel_backend`    | `vxlan`            | Only if `cni: flannel`: `vxlan`, `host-gw`, `wireguard-native` |
| `embedded_registry`  | `false`            | Local image cache (`--embedded-registry`)                      |
| `tls_sans_extra`     | `[]`               | Additional SANs in API certificate                             |

## CNI

K3s ships with **embedded Flannel**. Canal, Calico and Cilium require `--flannel-backend=none` and `--disable-network-policy` (role [`k3s_cni`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_cni/README.md), tag `cni`, **after** server and **before** agents).

| CNI                                                                                      | `k3s_install.core.cni` | Installation                                    |
| ---------------------------------------------------------------------------------------- | ---------------------- | ----------------------------------------------- |
| [Flannel](https://github.com/flannel-io/flannel)                                         | `flannel` (default)    | Embedded; adjust `flannel_backend`              |
| [Canal](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/flannel) | `canal`                | Canal manifest (Ansible or Application `canal`) |
| [Calico](https://docs.tigera.io/calico/latest/about/)                                    | `calico`               | Tigera operator + custom resources              |
| [Cilium](https://docs.cilium.io/)                                                        | `cilium`               | Official Helm chart                             |

Enable it by editing [`group_vars/incus_cluster/k3s_install.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/k3s_install.yml) (`core.cni: calico`, `gitops_argocd.enabled: true`) and then:

```
ansible-playbook -i inventory.ini playbook-k3s.yml
```

`-e k3s_install.x.y=valor` does not work

Ansible does not merge dotted keys into an existing dict — it creates a separate new variable (literally named `"k3s_install.x.y"`) and `k3s_install` remains untouched: **the default CNI was installed silently**. A `-e` with JSON doesn't help either: it **replaces** the entire dict, so `core.version`, `cluster_cidr`, etc. are lost. The playbook now detects both cases and **fails** before touching the nodes.

To change a single option from the command line there are top-level variables:

| Variable                    | Values                                 | Equivalent to                       |
| --------------------------- | -------------------------------------- | ----------------------------------- |
| `k3s_cni`                   | `flannel`, `canal`, `calico`, `cilium` | `k3s_install.core.cni`              |
| `k3s_gitops_argocd_enabled` | `true` / `false`                       | `k3s_install.gitops_argocd.enabled` |
| `k3s_node_labels_enabled`   | `true` / `false`                       | `k3s_install.node_labels.enabled`   |

```
ansible-playbook -i inventory.ini playbook-k3s.yml \
  -e k3s_cni=canal \
  -e k3s_gitops_argocd_enabled=true \
  -e k3s_node_labels_enabled=true
```

The CNI is chosen **before creating the cluster**: changing it later on a running cluster does not reinstall the network and can leave two CNIs fighting. If you already installed with another, the change is a migration, not a flag.

Calico is managed by Argo CD after bootstrap (Applications `calico-operator` + `calico-config`, manual sync; see [Calico with Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-calico/index.md)). GitOps alternatives (manual sync): `canal` or `cilium` in [`gitops/argocd/apps/`](https://github.com/maceosoft/gitops/tree/main/argocd/apps) — only if you installed K3s with `cni` different from `flannel` via Ansible **or** you apply the manifest after `--flannel-backend=none`. **Do not** enable more than one CNI (Container Network Interface) app at a time.

### Trade-off Analysis — CNI

| CNI                             | Advantage                           | Cost / Risk                                        |
| ------------------------------- | ----------------------------------- | -------------------------------------------------- |
| Flannel                         | Zero extra steps; low overhead      | No native NetworkPolicy                            |
| Canal                           | Calico policies on Flannel          | Legacy; two components                             |
| Calico                          | Mature NetworkPolicy; optional BGP  | More pods/RAM                                      |
| Cilium                          | eBPF; Hubble; L3–L7 policies        | CPU/kernel; validate ARM64                         |
| `vxlan` vs `host-gw`            | `host-gw`: less encapsulation at L2 | `host-gw` requires direct L2 between nodes         |
| Ansible `k3s_cni` vs GitOps app | Same play as K3s bootstrap          | GitOps useful if you reinstall CNI without Ansible |

### Execution Order (playbook)

```
flowchart LR
  prereqs[k3s_prereqs] --> server[k3s_server]
  server --> cni[k3s_cni_opcional]
  cni --> kube_vip[k3s_kube_vip_opcional]
  server --> kube_vip
  kube_vip --> agent[k3s_agent]
  agent --> traefik[k3s_cleanup_traefik]
```

The `cni` play is skipped if `k3s_install.core.cni` is `flannel` (default).

### kube-vip

Control plane **VIP** (virtual IP) only (`cp_enable=true`, `svc_enable=false`). Does not replace MetalLB or K3s ServiceLB.

| Field       | Default        | Description    |
| ----------- | -------------- | -------------- |
| `address`   | `192.168.20.4` | Free IP on LAN |
| `interface` | `br0`          | L2 interface   |
| `mode`      | `arp`          | `arp` or `bgp` |

### Trade-off Analysis — kube-vip

| Alternative | Advantage                   | Cost / Risk                        |
| ----------- | --------------------------- | ---------------------------------- |
| `arp`       | Simple on HomeLab L2 LAN    | Depends on router ARP response     |
| `bgp`       | Integration with router BGP | Requires configured router/peering |

With `kube_vip.enabled=true`, `k3s_api_url` points to the VIP and agents join via it.

### Per-host (`host_vars/`)

```
k3s_agent_kubelet_args:
  - max-pods=110
k3s_node_labels:
  workload-tier: light
  node.longhorn.io/create-default-disk: "false"
```

## Table B — gitops (not playbook)

Full catalog and step-by-step guide in [Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md).

| Component                   | Application                                          | Sync                    |
| --------------------------- | ---------------------------------------------------- | ----------------------- |
| Sealed Secrets              | `sealed-secrets`                                     | auto (wave 0)           |
| OpenEBS                     | `openebs`                                            | auto (wave 0)           |
| StorageClasses              | `homelab-storage`                                    | auto (wave 0)           |
| MetalLB                     | `metallb` + `metallb-config`                         | auto                    |
| Gateway API + cert-manager  | `gateway-api`, `cert-manager`, `cert-manager-config` | auto (waves 0–1)        |
| Gateway Controller          | `kong` (or `traefik`, `nginx-gateway`)               | auto (wave 1; only one) |
| Longhorn                    | `longhorn`                                           | auto (wave 1)           |
| Argo CD Dex/route           | `argocd-route`, `argocd`                             | auto (wave 2)           |
| K3s upgrades                | `k3s-upgrade`                                        | auto (wave 3)           |
| vCluster Platform           | `vcluster-platform`                                  | manual                  |
| CAPN demo                   | `capn-demo`                                          | manual                  |
| CNI Calico                  | `calico-operator`, `calico-config`                   | manual (enabled)        |
| Alternative CNIs (optional) | `canal`, `cilium`                                    | manual                  |

### Trade-off Analysis — kube-vip vs MetalLB

| Alternative                  | Advantage                         | Cost / Risk                                            |
| ---------------------------- | --------------------------------- | ------------------------------------------------------ |
| kube-vip                     | Stable VIP for API only `:6443`   | Does not expose Gateways or Services                   |
| MetalLB                      | LoadBalancer for Gateway and apps | Pool `192.168.23.200–.220` must be free (outside DHCP) |
| CNI via Ansible (`k3s_cni`)  | Same play as K3s bootstrap        | Changing CNI live is disruptive                        |
| CNI via GitOps (manual apps) | Reinstall without Ansible         | Only if K3s already has `flannel-backend=none`         |

## After the Ansible Playbook

Continue in [Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md).

## Dex OAuth Playbook (Phase 4)

[`playbook-dex-oauth-secrets.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-dex-oauth-secrets.yml) — generates `VCLUSTER_CLIENT_SECRET` in 1Password if missing, applies `argocd-secret` and configures Incus OIDC (OpenID Connect) on invincible. Requires `KUBECONFIG` exported and the 1Password app unlocked.

| Tag        | What it does                                                                                                              |
| ---------- | ------------------------------------------------------------------------------------------------------------------------- |
| `ensure`   | Generates the missing secret in 1Password (`VCLUSTER_CLIENT_SECRET`)                                                      |
| `argocd`   | Applies `dex.*` keys in `argocd-secret` and restarts Dex                                                                  |
| `incus`    | Installs HomeLab CA on invincible, **restarts `incus`** (if CA changed) and configures `oidc.issuer` and `oidc.client.id` |
| `incus_ca` | Only installs HomeLab CA on invincible (and restarts `incus` if changed)                                                  |
| `vcluster` | Applies `VCLUSTER_CLIENT_SECRET` on vCluster Platform (requires `apply_vcluster: true`)                                   |

```
cd ansible
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml --tags ensure,argocd,incus
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml --tags incus_ca
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml --tags vcluster \
  -e dex_oauth.apply_vcluster=true
```

`apply_incus` is in `true` by default on `group_vars`.

Variables: [`group_vars/dex_oauth_secrets.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/dex_oauth_secrets.yml).

## Examples

```
# Mínimo (con los defaults de group_vars/incus_cluster/k3s_install.yml)
ansible-playbook -i inventory.ini playbook-k3s.yml

# Solo un play puntual (ej. reintentar kube-vip), con las opciones ya
# habilitadas en group_vars/incus_cluster/k3s_install.yml:
ansible-playbook -i inventory.ini playbook-k3s.yml --tags kube_vip

# Post-install GitOps
kubectl apply -f gitops/bootstrap/root-appset.yaml
```

For VIP + Argo CD controller: enable `kube_vip.enabled: true` and `gitops_argocd.enabled: true` in `group_vars/incus_cluster/k3s_install.yml` before running the playbook (see the warning above about why `-e` does not work for this).

## kube-vip vs MetalLB

```
flowchart TB
  subgraph cp_vip [ControlPlane VIP]
    kubeVip[kube_vip svc_enable_false]
    apiServer[API_6443]
    kubeVip --> apiServer
  end
  subgraph svc_lb [Service LoadBalancer]
    metalLB[MetalLB_L2]
    svcLB[Service_type_LoadBalancer]
    metalLB --> svcLB
  end
  gateway[Gateway_Kong] --> svcLB
```

kube-vip **must never** manage `LoadBalancer` Services (`svc_enable=false`).

## Incus Cluster Playbook (Phase 2)

[`playbook-incus-cluster.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-incus-cluster.yml) — bootstrap, join, cluster groups, manual scheduler and UI (user interface). Run after `playbook-bootstrap.yml`.

```
cd ansible
ansible-playbook -i inventory.ini playbook-incus-cluster.yml
```

| Tag         | What it does                                |
| ----------- | ------------------------------------------- |
| `bootstrap` | Preseed init on invincible                  |
| `join`      | Tokens + join oliver/deborah (`serial: 1`)  |
| `groups`    | `x86-nodes`, `arm64-nodes`, assignments     |
| `scheduler` | `oliver` → `scheduler.instance manual`      |
| `ui`        | `incus-ui-canonical` + `core.https_address` |

Variables: [`group_vars/incus_cluster/vars.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/vars.yml). Guide: [Phase 2 — Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md).

# GitOps — catalog and troubleshooting

Part of the implementation guide

The step-by-step walkthrough is in **[Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)**. Ansible bootstrap (K3s, network): [Phases 0–3](https://homelab.maceo.cl/en/implementacion/index.md).

Reference for Argo CD **Applications** and troubleshooting. Manifests in [`gitops/`](https://github.com/maceosoft/gitops).

Related documentation:

- [K3s playbook options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md) — Ansible flags (kube-vip, Argo CD controller, …)
- [vCluster Platform](https://homelab.maceo.cl/en/vcluster/index.md) — SSO (single sign-on) Dex + web kubeconfig (K3s host + vClusters)
- [Incus Cluster](https://homelab.maceo.cl/en/incus/cluster-setup/#admin-ui) — web UI + OIDC (OpenID Connect) Dex
- [Storage](https://homelab.maceo.cl/en/storage/index.md) / [Secrets](https://homelab.maceo.cl/en/secrets/index.md)
- [Tech stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md)

## Two repos, two responsibilities

| Repo                                              | What it installs                                        | Tool                                       |
| ------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------ |
| [`homelab`](https://github.com/maceosoft/homelab) | K3s binary, kube-vip CP, Argo CD controller, OS network | [Ansible](https://docs.ansible.com/)       |
| [`gitops`](https://github.com/maceosoft/gitops)   | MetalLB, Gateway API, storage, vCluster, CAPN, …        | [Argo CD](https://argo-cd.readthedocs.io/) |

```
flowchart LR
  A[Ansible bootstrap] --> B[ArgoCD controller]
  B --> C[ApplicationSet homelab-root]
  C --> D[Apps descomentadas, por olas -1 a 3]
  D --> E[Apps manual]
```

______________________________________________________________________

## GitOps component catalog

Each row is an Argo CD **Application** (except auxiliary manifests). Sync **auto** = synced by `homelab-root` (if uncommented in `root-appset.yaml`); **manual** = stays commented until you enable it.

### Bootstrap

| Manifest                                                                                                 | Sync           | Purpose                                                                                                                                                                                                                                                                                            |
| -------------------------------------------------------------------------------------------------------- | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`bootstrap/root-appset.yaml`](https://github.com/maceosoft/gitops/blob/main/bootstrap/root-appset.yaml) | Once, manually | **ApplicationSet**: generates the root Application `homelab-root` with the uncommented apps from its list, deployed in waves. It is the only `kubectl apply` GitOps after having the controller; Argo CD does not manage it, so it is reapplied with `kubectl apply -f` whenever the list changes. |

### Wave 0 — Foundations (storage + LB controller)

| Application           | Product                                                          | Purpose                                                                                                                                       |
| --------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **`sealed-secrets`**  | [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) | Controller that decrypts `SealedSecret` in the cluster.                                                                                       |
| **`openebs`**         | [OpenEBS](https://openebs.io/docs)                               | Per-node local storage (LocalPV).                                                                                                             |
| **`homelab-storage`** | —                                                                | StorageClasses `openebs-hostpath` (default) and `longhorn-mixto` ([trade-offs](https://homelab.maceo.cl/en/storage/#cluster-storageclasses)). |
| **`metallb`**         | [MetalLB](https://metallb.universe.tf/)                          | LoadBalancer controller on LAN. **Not** kube-vip.                                                                                             |
| **`gateway-api`**     | [Gateway API](https://gateway-api.sigs.k8s.io/)                  | CRDs `Gateway`, `HTTPRoute`, etc. (standard channel).                                                                                         |
| **`cert-manager`**    | [cert-manager](https://cert-manager.io/docs/)                    | TLS certificate issuance.                                                                                                                     |

### Wave 1 — Service mesh + HA storage

| Application                                | Product                                                                                | Purpose                                                                                                                                                                                                                                 |
| ------------------------------------------ | -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`metallb-config`**                       | [MetalLB](https://metallb.universe.tf/)                                                | L2 pool `192.168.23.200–192.168.23.220` (reserved outside router DHCP).                                                                                                                                                                 |
| **`cert-manager-config`**                  | [cert-manager](https://cert-manager.io/docs/)                                          | HomeLab private CA (`ClusterIssuer homelab-ca`).                                                                                                                                                                                        |
| **`kong`** (or `traefik`, `nginx-gateway`) | [Kong Ingress Controller](https://developer.konghq.com/kubernetes-ingress-controller/) | [Gateway API](https://gateway-api.sigs.k8s.io/) controller and `Gateway homelab` (HTTP/HTTPS, `*.homelab.local`). Only one is enabled; see [Switch Gateway Controller](https://homelab.maceo.cl/en/operacion/cambiar-gateway/index.md). |
| **`longhorn`**                             | [Longhorn](https://longhorn.io/docs/)                                                  | Cross-arch replicated storage.                                                                                                                                                                                                          |

### Wave 2 — Accessible Argo CD + Dex

| Application / folder                                                                                        | Product                                                                              | Purpose                                                                                                                                                                                                                                                                                                         |
| ----------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`argocd-route`**                                                                                          | [Gateway API](https://gateway-api.sigs.k8s.io/)                                      | `HTTPRoute`: UI at [`https://argocd.homelab.local`](https://argocd.homelab.local).                                                                                                                                                                                                                              |
| **`argocd/config/`** (`argocd`)                                                                             | [Dex](https://dexidp.io/docs/)                                                       | GitHub connector + staticClients for vCluster and Incus UI.                                                                                                                                                                                                                                                     |
| **`victoria-metrics`** *(optional)*                                                                         | [VictoriaMetrics](https://docs.victoriametrics.com/helm/victoria-metrics-k8s-stack/) | Monitoring: metrics, Alertmanager (alerts to Discord) and Grafana at [`https://grafana.homelab.local`](https://grafana.homelab.local). Requires `playbook-platform-secrets.yml` Secrets before enabling ([4.13](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional)). |
| [`argocd/config/notifications/`](https://github.com/maceosoft/gitops/tree/main/argocd/config/notifications) | Argo CD Notifications                                                                | Services, triggers and templates for Discord notifications ([4.14](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#414-argocd-notifications-to-discord-optional)); `test.sh` tests them without sending.                                                                                              |
| [`argocd/secrets/`](https://github.com/maceosoft/gitops/tree/main/argocd/secrets)                           | [1Password SDK](https://github.com/1Password/onepassword-sdk-python)                 | OAuth from local app                                                                                                                                                                                                                                                                                            |

**What it looks like.** The Argo CD UI at [`https://argocd.homelab.local`](https://argocd.homelab.local) prompts for GitHub login (Dex):

View screenshot: Argo CD login

The root application `homelab-root` groups all others: in its tree view, each "application" box is one of the Applications from the catalog.

View screenshot: homelab-root tree

### Wave 3 — Operations

| Application                                | Product                                                                           | Purpose                                                                                                                                                                                    |
| ------------------------------------------ | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **`k3s-upgrade`**                          | [system-upgrade-controller](https://github.com/rancher/system-upgrade-controller) | Staggered K3s upgrades.                                                                                                                                                                    |
| **`victoria-metrics-config`** *(optional)* | [VictoriaMetrics](https://docs.victoriametrics.com/operator/)                     | Alert rules (`VMRule`), scrapes for cert-manager, Longhorn and Argo CD, and the `HTTPRoute` Grafana dashboard. Goes after `victoria-metrics` because its CRDs are installed by that chart. |

### AI Assistant (folder `ai/`)

| Application         | Purpose                                                                                                                                                                                                                                                                                                                                      |
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`ollama-router`** | Namespace `ai`, Service, Kong route (HTTPS only, `ollama.homelab.local`) and single router for Ollama nodes: automatic registration, per-client Bearer token, node selection by model, and metrics by model and node. Replaces per-node Services and the `/api/tags` cache.                                                                  |
| **`open-webui`**    | Assistant web chat (`open-webui.homelab.local`), with SSO via Dex (client `open-webui`).                                                                                                                                                                                                                                                     |
| **`open-terminal`** | Terminal with code execution for the assistant, isolated in the `ai-sandbox` namespace and accessible only from Open WebUI.                                                                                                                                                                                                                  |
| **`ai-tools`**      | Read-only tools (Kubernetes and metrics) used by the assistant.                                                                                                                                                                                                                                                                              |
| **`k8sgpt`**        | K8sGPT: cluster analysis explained by Ollama. The alert summary lives in `ai/alert-triage`. **Staged** (commented in `root-appset.yaml`): its chart includes a ClusterRole `*/*/*`; the decision to enable it is up to the user (see [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/#optional-checks)). |

### Logs and Reports (folders `monitoring/` and `ai/log-report/`)

| Application         | Purpose                                                                                                                   |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **`victoria-logs`** | VictoriaLogs: log database (30-day retention), ClusterIP only.                                                            |
| **`vector`**        | Vector: DaemonSet that reads pod logs and node journals and sends them to VictoriaLogs.                                   |
| **`log-report`**    | CronJobs `log-digest-daily` (daily summary saved as log) and `log-report-weekly` (weekly AI report to the `#ia` channel). |

See [Centralized Logs and AI Reports](https://homelab.maceo.cl/en/operacion/logs/index.md).

### Security (folder `security/` and NetworkPolicies per namespace)

Security manifests, almost all **to be enabled** one by one (commented in `root-appset.yaml`, with README for checks and rollback):

- NetworkPolicies `default-deny` with explicit permissions (Applications `netpol-ai`, `netpol-monitoring`, `netpol-longhorn-system`, `netpol-kong`, `netpol-gateway` and `netpol-arc-runners`, folder [`security/networkpolicies/`](https://github.com/maceosoft/gitops/tree/main/security/networkpolicies)). The one for `ai-sandbox` travels with the app `open-terminal`.
- Pod Security Admission labels in `warn` and `audit` (`namespace-labels`) and `allowedRoutes` by selector.
- `oauth2-proxy` (optional) to publish internal interfaces behind Dex.
- [`security/k8sgpt-rbac/`](https://github.com/maceosoft/gitops/tree/main/security/k8sgpt-rbac) and [`security/calico-policy-edit/`](https://github.com/maceosoft/gitops/tree/main/security/calico-policy-edit): documentation and proposals for K8sGPT (staged) and for the Calico risk.

**Enabled:** `security-rbac`, which grants read-only (`view`) to the `oidc:maceosoft:devops` group when logging in via Dex. The NetworkPolicies for logs, `ai/tools`, `alert-triage` and Open Terminal already travel with their application. Kong Manager disabled and Argo CD without the local `admin` are part of the normal configuration. The report CronJob scripts have their source versioned in [`scripts/log-report/`](https://github.com/maceosoft/gitops/tree/main/scripts/log-report). See [Authentication Model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md).

See [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md).

### Manual Sync (you decide when)

| Application                                 | Product                                                                                  | Purpose                                                                                                                                                                                                                                                                |
| ------------------------------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **`vcluster-platform`**                     | [vCluster Platform](https://www.vcluster.com/docs/platform/)                             | UI: K3s host kubeconfig (connected) + vClusters.                                                                                                                                                                                                                       |
| **`capn-demo`**                             | [CAPN](https://capn.linuxcontainers.org/)                                                | Cluster API over Incus.                                                                                                                                                                                                                                                |
| **`calico-operator`** + **`calico-config`** | [Calico](https://docs.tigera.io/calico/latest/about/)                                    | Calico CNI (CRDs + operator + `Installation`). **Active**: Argo CD manages and updates Calico (v3.33.0); Ansible installed it when creating the cluster. **Manual** sync. See [Calico with Argo CD](https://homelab.maceo.cl/en/operacion/actualizar-calico/index.md). |
| **`canal`**                                 | [Canal](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/flannel) | Canal CNI (Flannel + Calico policies).                                                                                                                                                                                                                                 |
| **`cilium`**                                | [Cilium](https://docs.cilium.io/)                                                        | Cilium CNI via Helm.                                                                                                                                                                                                                                                   |

CNI: Phase 3, not Phase 4

The CNI (Container Network Interface) is chosen in [Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md) (`k3s_install.core.cni`). By default **embedded Flannel** (Ansible, no Application). The apps above are GitOps alternatives if K3s (lightweight Kubernetes distribution) already has `--flannel-backend=none`. See [playbook-options — CNI](https://homelab.maceo.cl/en/k3s/playbook-options/#cni).

### Support Manifests (not Applications)

| Path                                                                                                           | Purpose                                                                                                                         |
| -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| [`storage/`](https://github.com/maceosoft/gitops/tree/main/storage)                                            | StorageClasses via `homelab-storage`.                                                                                           |
| [`metallb/`](https://github.com/maceosoft/gitops/tree/main/metallb)                                            | Pool + L2Advertisement.                                                                                                         |
| [`k3s-upgrade/`](https://github.com/maceosoft/gitops/tree/main/k3s-upgrade)                                    | `Plan` SUC CRs.                                                                                                                 |
| [`monitoring/`](https://github.com/maceosoft/gitops/tree/main/monitoring)                                      | VictoriaMetrics chart values, rules, scrapes, Grafana route, and rule tests (`victoria-metrics` and `victoria-metrics-config`). |
| [`vcluster/values/platform.yaml`](https://github.com/maceosoft/gitops/blob/main/vcluster/values/platform.yaml) | Platform Helm values + Dex OIDC.                                                                                                |
| [`capn/clusters/`](https://github.com/maceosoft/gitops/tree/main/capn/clusters)                                | Clusters generated with `clusterctl`.                                                                                           |
| [`ai/`](https://github.com/maceosoft/gitops/tree/main/ai)                                                      | AI assistant manifests: `ollama-router`, `open-webui`, `open-terminal`, `tools`, `alert-triage`, `k8sgpt` and `log-report`.     |
| [`cni/calico/`](https://github.com/maceosoft/gitops/tree/main/cni/calico)                                      | Calico Installation CR (`calico-config` Application).                                                                           |

______________________________________________________________________

## kube-vip (Ansible) vs MetalLB (GitOps)

|                       | [kube-vip](https://kube-vip.io/) | [MetalLB](https://metallb.universe.tf/) |
| --------------------- | -------------------------------- | --------------------------------------- |
| **Who installs it**   | Ansible (`k3s_kube_vip`)         | Argo CD                                 |
| **What gets the VIP** | Only API server `:6443`          | Services `LoadBalancer`                 |
| **Key config**        | `svc_enable=false`               | Pool `192.168.23.200–.220`              |

______________________________________________________________________

## Quick troubleshooting

| Symptom                                                                                                                                    | Check                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Application `Degraded`                                                                                                                     | `kubectl describe application -n argocd <nombre>`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| `arc-runners` `OutOfSync` and the listener pod restarts every few minutes                                                                  | Argo CD prunes in a loop the resources created by the ARC controller (`AutoscalingListener`, `Role`, `RoleBinding`: they copy the `app.kubernetes.io/instance` label). Fixed with `application.resourceTrackingMethod: annotation` in `argocd-cm` (already in `gitops/argocd/config`). Check: `kubectl -n argocd get cm argocd-cm -o jsonpath='{.data.application\.resourceTrackingMethod}'` → `annotation`; if not, sync `argocd` and restart the controller: `kubectl -n argocd rollout restart statefulset argocd-application-controller` |
| `openebs` `OutOfSync` with sync stuck at `Running`                                                                                         | The chart's `pre-upgrade` hook Job doesn't finish and Argo CD waits for it. In OpenEBS it uses an image (`bitnami/kubectl:1.25.15`) that no longer exists on Docker Hub; the hook only migrates from v3 to v4 and is disabled in `argocd/apps/openebs.yaml` (`preUpgradeHook.enabled: false`). If the old Job is still there with the `argocd.argoproj.io/hook-finalizer` finalizer, cancel the operation (`status.operationState.phase: Terminating`) and remove the finalizer                                                              |
| `longhorn` with sync stuck at `Running`: the `longhorn-pre-upgrade` Job has no pod (`serviceaccount "longhorn-service-account" not found`) | In a **clean install** with Argo CD that Job runs before its ServiceAccount and the sync waits forever: disable it with `preUpgradeChecker.jobEnabled: false` (already set in `argocd/apps/longhorn.yaml`). If you cancel the operation (`status.operationState.phase: Terminating`), remove the Job's finalizer and refresh. In an **upgrade** the Job does run and fails for a reason if the upgrade path isn't supported: read `kubectl logs -n longhorn-system job/longhorn-pre-upgrade` before disabling it                             |
| Gateway without address                                                                                                                    | `metallb-config` sync, free pool in LAN                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| vCluster SSO fails                                                                                                                         | `playbook-dex-oauth-secrets.yml --tags ensure,argocd,vcluster`; restart Dex                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| Incus UI SSO fails                                                                                                                         | `incus config get oidc.issuer` / `oidc.client.id` in invincible; `invincible` must trust the HomeLab CA (`--tags incus_ca`); public `incus-ui` client in `dex.config`; restart Dex                                                                                                                                                                                                                                                                                                                                                           |
| Dex login without GitHub                                                                                                                   | GitHub secrets in `argocd-secret`; OAuth App Redirect URIs correct                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| GitHub login rejected                                                                                                                      | User must belong to the `devops` team in the `maceosoft` org                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| Longhorn pod pending                                                                                                                       | Labels `node.longhorn.io/create-default-disk` ([playbook labels](https://homelab.maceo.cl/en/k3s/playbook-options/index.md))                                                                                                                                                                                                                                                                                                                                                                                                                 |
| CNI pods not Ready                                                                                                                         | K3s with `--flannel-backend=none` before sync; only **one** CNI app active ([CNI](https://homelab.maceo.cl/en/k3s/playbook-options/#cni))                                                                                                                                                                                                                                                                                                                                                                                                    |

Source manifests: [`gitops/argocd/apps/`](https://github.com/maceosoft/gitops/tree/main/argocd/apps).

Runnable guide: **[Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)**.

# Storage

Part of the implementation guide

Deployed in **[Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)** (waves 0–1). Longhorn labels: **[Phase 3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)**.

Hybrid x86_64 + ARM64 (64-bit ARM architecture) — GitOps manifests at [`gitops/storage/`](https://github.com/maceosoft/gitops/tree/main/storage).

## Diagram — storage tiers

```
flowchart TB
  PVC["PVC workload"]
  subgraph tier_local ["Default (3 nodos)"]
    OB["OpenEBS LocalPV"]
    INV["invincible"]
    NOL["oliver"]
    DEB["deborah"]
    OB --> INV
    OB --> NOL
    OB --> DEB
  end
  subgraph tier_ha ["HA cross-arch"]
    LH["Longhorn v1"]
    INV2["invincible (labeled)"]
    DEB2["deborah (labeled)"]
    LH --> INV2
    LH --> DEB2
  end
  PVC --> OB
  PVC --> LH
```

- **[OpenEBS](https://openebs.io/docs)** — Default LocalPV; 3 nodes; no replication.
- **[Longhorn](https://longhorn.io/docs/)** — Replicated x86↔arm64; designed for invincible + deborah (see [Longhorn v1](#longhorn-v1-selective-ha-cross-arch)).

## OpenEBS LocalPV (default)

DaemonSet on 3 nodes. Default StorageClass for almost everything.

Deployed via Argo CD Application `openebs` (Helm).

## Longhorn v1 (Selective HA, cross-arch)

**Always use the v1 engine (default), never v2/SPDK** (Storage Performance Development Kit): v2 has a documented bug of blocked I/O on ARM64 with NVMe (PCIe SSD) + 2 cores.

Its UI has no login screen and is not published by the Gateway: it opens with `kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80` and [`http://127.0.0.1:8080`](http://127.0.0.1:8080). The dashboard summarizes volumes, space, and nodes:

View screenshot: Longhorn dashboard

```
kubectl label node invincible node.longhorn.io/create-default-disk=true
kubectl label node deborah node.longhorn.io/create-default-disk=true
kubectl label node oliver node.longhorn.io/create-default-disk=false
```

With `numberOfReplicas: "2"` and 2 eligible nodes (x86 + arm64), each volume is replicated across architectures.

Trade-off analysis

OpenEBS LocalPV chosen

Simple on 3 nodes; default for almost everything

No HA if pod node fails

Longhorn v1

Replica x86↔arm64; volume UI

Only 2 eligible nodes; more RAM/pods

OpenEBS only

Smaller operational surface

Critical PVCs without cross-node replica

Mayastor discarded

High NVMe performance

≥2 GiB hugepages — unfeasible on M700

**This decision in other environments**

Longhorn (`longhorn-mixto`, 2 replicas) replicates important data across the nodes' own disks, and OpenEBS LocalPV (`openebs-hostpath`, default) provides fast local disk without replicas for rebuildable workloads. No storage array or licenses, and if a node fails critical data remains on the other.

- **Benefits:** distributed storage on the nodes themselves survives a server failure without data loss, provides snapshots and volumes that move with the pod, and avoids depending on a storage array; separating replicated disk (critical data) from local disk (rebuildable) lowers cost without risking what matters.
- The norm is a **corporate SAN/NAS** (with its CSI driver) or **supported** software-defined storage: [SUSE Storage](https://www.suse.com/products/rancher/storage/) (Longhorn's commercial version), [Portworx](https://portworx.com/), or Ceph/Rook with support from [Red Hat OpenShift Data Foundation](https://www.redhat.com/en/technologies/cloud-computing/openshift-data-foundation) or [IBM Storage Ceph](https://www.ibm.com/products/storage-ceph).
- Replication is not a backup: **off-cluster copies** (another site or object storage) with defined and tested RPO/RTO (how much data you can lose and how long to recover) are required.
- **Dedicated disks** and a separate storage network are used, with 3+ replicas and measured performance; here Longhorn shares disk and network with everything else.
- Longhorn **does** fit small sites, edge, or stores with few nodes and no array, where simplicity and predictable cost matter more than peak performance.

| Cloud              | Equivalent service                                                                                                                                                                                                                                                                                                                                               |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| GCP                | [Persistent Disk](https://docs.cloud.google.com/compute/docs/disks/persistent-disks) / [Hyperdisk](https://docs.cloud.google.com/compute/docs/disks/hyperdisks) and [Filestore](https://docs.cloud.google.com/filestore/docs) (RWX), with the [GKE CSI driver](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver) |
| AWS                | [Amazon EBS](https://aws.amazon.com/ebs/) and [Amazon EFS](https://aws.amazon.com/efs/) (RWX), with their CSI drivers ([EBS](https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html), [EFS](https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html))                                                                                               |
| Azure              | [Azure Disk Storage](https://azure.microsoft.com/en-us/products/storage/disks) and [Azure Files](https://azure.microsoft.com/en-us/products/storage/files) (RWX), with the [AKS CSI drivers](https://learn.microsoft.com/en-us/azure/aks/csi-storage-drivers)                                                                                                    |
| Oracle Cloud (OCI) | [Block Volume](https://docs.oracle.com/en-us/iaas/Content/Block/home.htm) and [File Storage](https://docs.oracle.com/en-us/iaas/Content/File/home.htm) (RWX), with the [OKE CSI driver](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingpersistentvolumeclaim.htm)                                                                       |

You get managed disks with replication, snapshots, and provider SLA, without managing storage nodes. You lose control and portability: each cloud has its own classes and limits, and a block disk is usually tied to a zone.

## Cluster StorageClasses

A StorageClass tells Kubernetes **how** to create a PersistentVolumeClaim (PVC) volume. Here are the ones that exist today (`kubectl get storageclass`); those from `gitops/storage/` are created by the Application `homelab-storage`, the rest come from charts or K3s.

| StorageClass       | Created by                                                                                                        | Provisioner                          | Replicas                                                                                                                     | Binding                | Expandable | Default?              |
| ------------------ | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | ---------------------- | ---------- | --------------------- |
| `openebs-hostpath` | [`gitops/storage/openebs-local.yaml`](https://github.com/maceosoft/gitops/blob/main/storage/openebs-local.yaml)   | OpenEBS LocalPV (`openebs.io/local`) | 1 (node disk)                                                                                                                | `WaitForFirstConsumer` | No         | **Yes**               |
| `local-path`       | K3s (addon `local-storage`)                                                                                       | `rancher.io/local-path`              | 1 (node disk)                                                                                                                | `WaitForFirstConsumer` | No         | **Yes** (see warning) |
| `longhorn`         | Longhorn chart                                                                                                    | Longhorn (`driver.longhorn.io`)      | `defaultClassReplicaCount` (2 in [`longhorn.yaml`](https://github.com/maceosoft/gitops/blob/main/argocd/apps/longhorn.yaml)) | `Immediate`            | Yes        | No                    |
| `longhorn-mixto`   | [`gitops/storage/storageclasses.yaml`](https://github.com/maceosoft/gitops/blob/main/storage/storageclasses.yaml) | Longhorn                             | 2 fixed                                                                                                                      | `Immediate`            | Yes        | No                    |
| `longhorn-static`  | Longhorn chart                                                                                                    | Longhorn                             | — (no parameters)                                                                                                            | `Immediate`            | Yes        | No                    |

All have `reclaimPolicy: Delete`: **deleting the PVC deletes the volume and its data**.

Two default StorageClasses today

`openebs-hostpath` and `local-path` carry the annotation `is-default-class`. With two defaults, a PVC without `storageClassName` is ambiguous: Kubernetes uses the one created most recently and the result depends on the order each was installed. The recommendation is to keep only one. In Ansible, `local-storage` is already in `k3s_install.core.disable_components` (alongside `traefik` and `servicelb`), but that only affects new installs: this cluster retains `local-path` until K3s is reinstalled. Meanwhile, always request the class explicitly in PVCs.

### `openebs-hostpath` — local, fast, node-bound

Creates a folder on the disk of the node where the pod runs.

| Pros                                                                              | Cons                                                                                     |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Fastest: local disk, no network or replication                                    | **No replication**: if the node goes down, data is unavailable until it returns          |
| Near-zero RAM/CPU overhead; works the same on x86_64 and ARM64                    | Pod is **pinned to the node** where the volume was born: cannot be rescheduled elsewhere |
| Works on all 3 nodes, no disk or package prep needed                              | Cannot be expanded, no snapshots or quotas (pod can fill the node disk)                  |
| With `WaitForFirstConsumer` the volume is born where the scheduler places the pod | Only `ReadWriteOnce` (single node at a time)                                             |

**Use for:** caches, rebuildable data, test databases, anything you can lose painlessly.

### `local-path` — the K3s one

Same model as `openebs-hostpath` (local folder, no replication, node-bound), but ships with K3s. Today it duplicates the previous one: see the warning above. Do not use it in new manifests.

### `longhorn` and `longhorn-mixto` — replicated

Longhorn stores each volume as a replicated block across multiple nodes, with snapshots, backups, and a UI.

| Pros                                                                             | Cons                                                                                                                 |
| -------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **Survives a node failure**: replicas live on different nodes (x86_64 and ARM64) | Writes are slower: every write is acknowledged on all replicas **over the network**                                  |
| Can be expanded live (`allowVolumeExpansion`), with snapshots and backups        | Consumes extra **RAM and CPU** on each node (managers, instance managers) and space: with 2 replicas, 2× the size    |
| `ReadWriteMany` (multiple pods at once) via an NFS server per volume             | Requires `open-iscsi` and `nfs-common` on nodes (role `k3s_prereqs`)                                                 |
| UI to inspect each volume's state                                                | With `Immediate` replicas are scheduled **without knowing** where the pod will run, so reads may go over the network |

**Which one:**

- `longhorn`: its replicas come from the chart value (`defaultClassReplicaCount`). If someone changes the chart, the behavior of **new** PVCs changes. This is what `arc-cache` uses today (the [runners' cache](https://homelab.maceo.cl/en/operacion/cache-arc/index.md)).
- `longhorn-mixto`: **2 fixed replicas**, `dataLocality: disabled` (replica does not stick to the pod's node: more fault-tolerant, more network traffic) and `staleReplicaTimeout: 30` (minutes before discarding a stale replica). Use it when you want a replica count that does not depend on the chart.

**Use for:** data that matters (databases, application state) and shared volumes.

StorageClass parameters cannot be edited

Kubernetes does not allow changing the `parameters` of an existing StorageClass. To change replicas you must **delete and recreate the class** (existing PVCs do not change). Today `kubectl get storageclass longhorn` may show `numberOfReplicas: "3"` even though `longhorn.yaml` asks for 2, if the class was created before the change; with 3 replicas and few eligible nodes volumes end up degraded.

On how many nodes does Longhorn live?

The `node.longhorn.io/create-default-disk` labels above are only honored if the setting `createDefaultDiskLabeledNodes` is set to `true`. Today it is set to `false` (verified with `kubectl -n longhorn-system get settings.longhorn.io`), so **all three nodes** contribute disk, including `oliver`. To limit it to `invincible` and `deborah`, enable that setting in `defaultSettings` of `longhorn.yaml` and disable `oliver`'s disk in the UI.

In the Longhorn UI: *Node* → `oliver` → *Edit node and disks* → on its disk, **Scheduling** to *Disable* → *Save*.

Check the disk name and disable its *scheduling* on the `nodes.longhorn.io` resource:

```
kubectl -n longhorn-system get nodes.longhorn.io oliver -o jsonpath='{.spec.disks}{"\n"}'
kubectl -n longhorn-system patch nodes.longhorn.io oliver --type merge \
  -p '{"spec":{"disks":{"<disco>":{"allowScheduling":false}}}}'
```

### `longhorn-static` — only for pre-existing volumes

Parameterless class created by the chart to **manually bind** a volume that already exists in Longhorn (static provisioning). It does not create new volumes on its own: do not use it in a regular PVC.

### Binding: `WaitForFirstConsumer` or `Immediate`

| Mode                                | What it does                                                              | Trade-off                                                                   |
| ----------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `WaitForFirstConsumer` (both local) | Waits for the pod to be scheduled and creates the volume **on that node** | Volume always lands with the pod; afterward it stays bound to that node     |
| `Immediate` (Longhorn's)            | Creates the volume as soon as the PVC exists                              | Works without a pod, but replicas are chosen without knowing the pod's node |

### Which one do I choose?

| I need…                                                     | Use                                                                 |
| ----------------------------------------------------------- | ------------------------------------------------------------------- |
| Data I **cannot** lose, or the pod to be able to move nodes | `longhorn-mixto`                                                    |
| Fastest possible and data is rebuildable                    | `openebs-hostpath`                                                  |
| A volume used by multiple pods at once (`ReadWriteMany`)    | `longhorn` or `longhorn-mixto`                                      |
| Grow a volume without recreating it                         | `longhorn` or `longhorn-mixto`                                      |
| Deleting the PVC **not** to delete the data                 | None: all are `Delete`; create a class with `reclaimPolicy: Retain` |

To request a class, put it in the PVC:

```
spec:
  storageClassName: longhorn-mixto
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 10Gi
```

## Upgrade and reinstall Longhorn

Longhorn only supports upgrading **one minor version at a time** (1.6 → 1.7 → … → 1.13). If the chart skips versions, the `longhorn-pre-upgrade` Job fails on purpose and, if you disable it, the new `longhorn-manager` crash with `upgrading from vX to vY for minor version is not supported`. **During an upgrade do not disable that Job.** In a clean install with Argo CD you do have to disable it (`preUpgradeChecker.jobEnabled: false`): it runs before its ServiceAccount, cannot create the pod, and the sync stays waiting for it forever.

If what Longhorn stores can be lost (in the HomeLab, the ARC cache —Actions Runner Controller—), it is faster to **reinstall from scratch** on the new version than to step through every intermediate version. Use [`playbook-reset-longhorn.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-reset-longhorn.yml), which is **destructive** (deletes PVCs, volumes, CRDs, the namespace, and `/var/lib/longhorn` on each node) and safe by default: without `-e longhorn_reset_confirm=true` it only shows what would be lost.

1. In `gitops/bootstrap/root-appset.yaml`, comment out `longhorn` and `arc-runners` and push. This way Argo CD does not recreate what gets deleted (the Applications disappear, their resources remain in the cluster).
1. `ansible-playbook -i inventory.ini playbook-reset-longhorn.yml` (preflight) and then with `-e longhorn_reset_confirm=true`.
1. Uncomment `longhorn` and `arc-runners`, push: Argo CD installs the version from `targetRevision` from scratch.

## Why not Mayastor

Mayastor requires ≥2 GiB of fixed hugepages — ruled out for the M700.

Runnable guide: **[Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)**.

# Secret Management

Part of the implementation guide

- **[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets)** — Phase 4 wave 0; CAPN (Cluster API Provider for Incus) Phase 6.
- **OAuth (Open Authorization) Dex** — [Phase 4 step 4.7](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#47-oauth-secrets-github-vcluster-incus-ui).
- **maceosoft platform** (Argo CD Apps, ARC —Actions Runner Controller— and OpenTofu, FTP) — [1Password → cluster](https://homelab.maceo.cl/en/maceosoft/github/#step-3-cluster-and-platform-secrets).

## Diagram — three secret pathways

```
flowchart LR
  subgraph git [En_git]
    SS[Sealed_Secrets]
    SOPS[SOPS_age]
  end
  subgraph station [Estacion_operador]
    OP[1Password_SDK_DesktopAuth]
  end
  subgraph cluster [En_cluster]
    SEC[k8s_Secret]
  end
  SS --> SEC
  SOPS --> git
  OP --> SEC
```

## Sealed Secrets (default in cluster)

[Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) — a controller (~20–30 MB RAM), no external account.

```
kubeseal --format yaml < secret-plano.yaml > secret-sellado.yaml
kubectl apply -f secret-sellado.yaml
rm secret-plano.yaml
```

The encryption key lives in 1Password

By default the controller generates its key pair inside the cluster (and rotates it every 30 days): if you reinstall K3s, it's lost and the `SealedSecret` in Git can no longer be decrypted. Here the pair is generated **once**, stored in 1Password (item `sealed-secrets`) and `playbook-platform-secrets.yml` carries it to the cluster; automatic rotation is disabled (`keyrenewperiod: "0"`). How to generate it: [maceosoft platform — Step 2](https://homelab.maceo.cl/en/maceosoft/github/#step-2-repos-argocd-app-and-ftp-manual).

Since the certificate (the public key) is also in 1Password, you can seal without cluster access (`op read "op://HomeLab/sealed-secrets/certificate" > tls.crt` pulls it down): `kubeseal --cert tls.crt --format yaml < secret-plano.yaml > secret-sellado.yaml`.

## SOPS + age (complement, zero pods)

[SOPS (Secrets OPerationS)](https://github.com/getsops/sops) — pre-bootstrap config (Incus tokens, etc.).

```
age-keygen -o key.txt
sops --encrypt --age <public-key> secrets.yaml > secrets.enc.yaml
```

## OAuth Dex (GitHub + vCluster + Incus UI) — 1Password local SDK

**GitHub OAuth App** credentials and Dex clients **vCluster Platform** and **Incus UI** do not go in Git.

| Source                                         | Fields                                                                                |
| ---------------------------------------------- | ------------------------------------------------------------------------------------- |
| **GitHub** (OAuth App from org `maceosoft`)    | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`                                            |
| **Ansible** (`playbook-dex-oauth-secrets.yml`) | `VCLUSTER_CLIENT_SECRET` (generates in 1Password if missing; applies to Dex and Helm) |

`VCLUSTER_CLIENT_SECRET` is **not** created in GitHub: Ansible generates it in 1Password if missing. Incus doesn't use a secret (it's a public OIDC —OpenID Connect— client in Dex).

They're read with the **[1Password Python SDK](https://github.com/1Password/onepassword-sdk-python)** and **DesktopAuth** on the workstation where the 1Password app is unlocked.

Defaults: **personal account** (`ONEPASSWORD_ACCOUNT_NAME=Personal`), vault **`HomeLab`**, one item with fields `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `VCLUSTER_CLIENT_SECRET`. Test: `ansible/scripts/check_1password.py`.

Homepage `https://argocd.homelab.local` (informational); Redirect URIs `https://argocd.homelab.local/api/dex/callback` (critical).

With the CLI: the OAuth App and its client secret are created **only from the web**; saving the item with `op item create` is in [maceosoft platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/#oauth-app-maceosoft-dex). To read a field without displaying it: `op read "op://HomeLab/maceosoft-dex/GITHUB_CLIENT_ID" | wc -c`.

Details: [Phase 4 — 4.7](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#47-oauth-secrets-github-vcluster-incus-ui) · [`1password.md`](https://github.com/maceosoft/gitops/blob/main/argocd/secrets/1password.md)

## Monitoring: Discord webhooks, healthchecks.io ping, and Grafana (1Password → cluster Secrets)

Optional ([Phase 4 — 4.13](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional)). How to create credentials: [Discord and healthchecks.io](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/index.md). `playbook-platform-secrets.yml` creates, in the `monitoring` namespace, three Secrets from 1Password; their values never pass through Git: `homelab` is public and `gitops`, though private, doesn't store secrets.

| 1Password Item                             | Fields                                                                           | Secret in cluster                                                                                                                                                                                                                       |
| ------------------------------------------ | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `discord-webhooks`                         | `critical`, `warnings`                                                           | `monitoring/discord-webhooks`                                                                                                                                                                                                           |
| `discord-webhooks`                         | `deployments` *(optional)*                                                       | keys `discord-despliegues-url` and `discord-avisos-url` in `argocd/argocd-notifications-secret` (Argo CD notifications, [4.14](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#414-argocd-notifications-to-discord-optional)) |
| `healthchecks-ping`                        | `url`                                                                            | `monitoring/healthchecks-ping`                                                                                                                                                                                                          |
| `grafana-admin`                            | `password` (and `username`, optional)                                            | `monitoring/grafana-admin`                                                                                                                                                                                                              |
| `discord-webhooks`                         | `ai` *(optional)*                                                                | key `ai` in `monitoring/discord-webhooks` (channel `#ia` for [AI reports](https://homelab.maceo.cl/en/operacion/logs/#daily-and-weekly-reports-in-ia))                                                                                  |
| `open-webui` *(AI assistant, optional)*    | `WEBUI_SECRET_KEY`                                                               | `ai/open-webui-secret` ([how to generate](https://homelab.maceo.cl/en/operacion/ia-ollama/#create-the-open-webui-key-once-only))                                                                                                        |
| `ollama-router` *(AI assistant, optional)* | `registry_token`, `open-webui`, `k8sgpt`, `reports`, `alert-triage`, `translate` | `ai/ollama-router-secrets` (all fields) and a Secret `ollama-token-<cliente>` with key `token` in each client's namespace ([rotation](https://homelab.maceo.cl/en/operacion/rotar-credenciales/#ollama-and-internal-services-tokens))   |
| `open-terminal` *(optional)*               | `api_key`                                                                        | `ai-sandbox/open-terminal-api-key` (mounted as file in pod) and `ai/open-terminal-api-key` ([guide](https://homelab.maceo.cl/en/operacion/ia-ollama/#open-terminal-terminal-for-the-assistant))                                         |
| `alert-triage-webhook` *(optional)*        | `token`                                                                          | `monitoring/alert-triage-webhook`                                                                                                                                                                                                       |
| `mcp-api-key` *(optional)*                 | `key`                                                                            | `ai/mcp-api-key`                                                                                                                                                                                                                        |

Tokens are generated with `openssl rand -hex 32` (the script requires at least 32 characters and no spaces). The `translate` field doesn't create any Secret in the cluster: it's the value of the `OLLAMA_TOKEN` GitHub Actions secret. If the Dex client for Open WebUI turns out to be confidential (because it doesn't support PKCE —Proof Key for Code Exchange—), its secret is generated with `playbook-dex-oauth-secrets.yml`, like `VCLUSTER_CLIENT_SECRET`.

The playbook applies Secrets with `--server-side` and works with Python 3.9 or later. All AI items are optional (Open Terminal, the `#ia` channel, the tokens): if one is missing, only what depends on it is skipped. The `ai` and `ai-sandbox` namespaces are created without Pod Security Admission labels and with the `Prune=false,Delete=false` Argo CD annotations, so Argo CD doesn't delete them on sync.

Alertmanager reads URLs as mounted files (`webhook_url_file`), so they don't even appear in its config. The script requires both Discord URLs to have the format `https://discord.com/api/webhooks/<id>/<token>`. To rotate a webhook, delete it and create a new one in Discord, update the item, and run the playbook again.

## Local AI: create tokens with `op`

Local AI (Ollama router, Open WebUI, Open Terminal, reports) uses random tokens that live in 1Password, vault `HomeLab`, and that `playbook-platform-secrets.yml` brings to the cluster. With the 1Password CLI (`op`) and the app unlocked, each item is created with a command; each field receives a distinct `openssl rand -hex 32` value (64 hex characters):

| Item                   | Fields                                                                           | Purpose                                                   |
| ---------------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------- |
| `ollama-router`        | `registry_token`, `open-webui`, `k8sgpt`, `reports`, `alert-triage`, `translate` | One token per router client and one for node registration |
| `alert-triage-webhook` | `token`                                                                          | Alertmanager sends it to `alert-triage`                   |
| `mcp-api-key`          | `key`                                                                            | Key for Open WebUI MCP tools                              |
| `open-terminal`        | `api_key`                                                                        | Key for Open Terminal sandbox                             |
| `open-webui`           | `WEBUI_SECRET_KEY`                                                               | Key with which Open WebUI signs sessions                  |

```
op item create --category "Secure Note" --vault HomeLab --title ollama-router \
  "registry_token[password]=$(openssl rand -hex 32)" \
  "open-webui[password]=$(openssl rand -hex 32)" \
  "k8sgpt[password]=$(openssl rand -hex 32)" \
  "reports[password]=$(openssl rand -hex 32)" \
  "alert-triage[password]=$(openssl rand -hex 32)" \
  "translate[password]=$(openssl rand -hex 32)"

op item create --category "Secure Note" --vault HomeLab --title alert-triage-webhook "token[password]=$(openssl rand -hex 32)"

op item create --category "Secure Note" --vault HomeLab --title mcp-api-key "key[password]=$(openssl rand -hex 32)"

op item create --category "Secure Note" --vault HomeLab --title open-terminal "api_key[password]=$(openssl rand -hex 32)"

op item create --category "Secure Note" --vault HomeLab --title open-webui "WEBUI_SECRET_KEY[password]=$(openssl rand -hex 32)"
```

- If you have more than one 1Password account, export `OP_ACCOUNT` with yours first (it's the standard `op` variable): the items must end up in the account that `apply_platform_secrets.py` reads.
- `op` creates each field loose (no section) and with the exact name, as the playbook requires. Note: values passed as command arguments are visible for a moment in `ps` on your own machine; it's a minor risk on a personal workstation, but don't use these commands on a shared machine.
- To see what `op` would do without creating anything, add `--dry-run` to each command.

Then verify the fields, without showing values, and apply them:

```
python3 ansible/scripts/apply_platform_secrets.py --check
cd ansible && ansible-playbook -i inventory.ini playbook-platform-secrets.yml
```

The `#ia` channel is separate: its webhook is added manually to the `ai` field of the `discord-webhooks` item (see [Discord and healthchecks.io](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/index.md)).

### GitHub secret for translation

Automated translation (`translate.yml`) needs the GitHub Actions secret `OLLAMA_TOKEN`, with the value of the `translate` field. The value travels through a pipe, without being printed or remaining in terminal history:

```
op read "op://HomeLab/ollama-router/translate" | gh secret set OLLAMA_TOKEN --repo maceosoft/homelab
# debe aparecer OLLAMA_TOKEN
gh secret list --repo maceosoft/homelab
```

### Rotate a token

Edit the field in the 1Password app (use the password generator) or create a new value with `op`:

```
op item edit ollama-router "reports[password]=$(openssl rand -hex 32)"
```

Then run `playbook-platform-secrets.yml` again and restart the pods that use it (see [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md)). If you change `translate`, repeat the `gh secret set` command.

## Root password rotation (local 1Password SDK)

[`playbook-rotate-root-passwords.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-rotate-root-passwords.yml) generates a new password per node (not idempotent on purpose — each run rotates), applies it in the local `root` of each host (`become` + `ansible.builtin.user`, hashed with `password_hash('sha512')`) and stores it in the **`HomeLab`** vault, one item per host (`root@invincible`, `root@oliver`, `root@deborah`, field `password`). It doesn't touch SSH — `root` login via SSH is already disabled by `host_hardening` (`PermitRootLogin no`); this is defense in depth for console/`su -`.

```
cd ansible
ansible-playbook -i inventory.ini playbook-rotate-root-passwords.yml
ansible-playbook -i inventory.ini playbook-rotate-root-passwords.yml \
  --limit deborah
```

To read a password when you need it (for example, via console), with the 1Password CLI and without leaving it in history: `op read "op://HomeLab/root@deborah/password" | pbcopy` (on Linux, `xclip -selection clipboard` or `wl-copy`). If you have more than one 1Password account, export `OP_ACCOUNT` first.

Trade-off analysis

Sealed Secrets

Encrypted in Git; lightweight controller (~30 MB)

Cluster-only; doesn't work pre-bootstrap

SOPS + age

Zero pods; encrypted files in repo

Age key on workstation; manual workflow

1Password SDK

OAuth outside Git; DesktopAuth

App unlocked; not for SealedSecret in CAPN

Vault / ESO discarded

Enterprise-grade

Excessive RAM and complexity for HomeLab

## Discarded

- **HashiCorp Vault** — too heavy for the M700.
- **1Password + ESO (External Secrets Operator) in cluster** — requires Business plan; we use **local Python SDK (Software Development Kit)** (DesktopAuth) from the workstation.
- **RustyVault** — young project, no confirmed ARM64 (64-bit ARM architecture).

**This decision in other environments**

Secrets live in 1Password and reach the cluster via a playbook; what goes to Git is encrypted with Sealed Secrets or SOPS + age, never in plaintext. No need to maintain a secrets server, and for a single person it's enough.

- **Benefits:** the same pattern (a central manager, nothing in plaintext in Git, one secret per client) gives a single place to see who has what, and revoke or rotate a credential without touching code.
- **Central manager with auditing:** uses [HashiCorp Vault](https://developer.hashicorp.com/vault), [1Password Business](https://1password.com/business-security) or [CyberArk](https://www.paloaltonetworks.com/idira/machine/secrets-management) (now part of Palo Alto Networks), with a log of every read and per-team permissions.
- **Automatic synchronization:** instead of running a playbook from a workstation, the cluster reads secrets with [External Secrets Operator](https://external-secrets.io/) and keeps them up to date automatically.
- **Rotation and auditing:** credentials expire and rotate by policy, and compliance (SOC 2, ISO 27001) requires proving who read what and when.
- **When the HomeLab approach makes sense:** on small teams or isolated sites, Sealed Secrets or SOPS + age work without another server to keep available.

| Cloud              | Equivalent Service                                                          |
| ------------------ | --------------------------------------------------------------------------- |
| GCP                | [Secret Manager](https://cloud.google.com/security/products/secret-manager) |
| AWS                | [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/)              |
| Azure              | [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault)     |
| Oracle Cloud (OCI) | [OCI Vault](https://www.oracle.com/security/cloud-security/key-management/) |

Additionally, each cloud's KMS can encrypt Secrets inside etcd: [GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/encrypting-secrets), [EKS](https://docs.aws.amazon.com/eks/latest/userguide/envelope-encryption.html), [AKS](https://learn.microsoft.com/en-us/azure/aks/use-kms-etcd-encryption) and [OKE](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengencryptingdata.htm). You gain a managed service, with rotation, auditing, and IAM permissions included. You lose portability: each cloud has its own API, and you pay per secret and per read.

Full stack: **[Tech Stack](https://homelab.maceo.cl/en/implementacion/stack-tecnologico/index.md)**.

# maceosoft Platform — Discord and healthchecks.io

How to create, **manually and just once**, the credentials for [monitoring](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional): the Discord webhooks where alerts arrive and the healthchecks.io check that notifies you if monitoring itself goes down. At the end they are stored in 1Password and [`playbook-platform-secrets.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-platform-secrets.yml) brings them to the cluster.

Menu names are in English

This guide uses the texts exactly as they appear in the English UI of Discord and healthchecks.io (**Server Settings**, **Add Check**…). If you use Discord in Spanish, the menus have different names but the same order. These platforms change their UI from time to time: if a button doesn't match, look for the equivalent on that same screen.

## What gets created

| What                                                                                    | Where created   | 1Password item      | Used by                                                                                                                            |
| --------------------------------------------------------------------------------------- | --------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| One **webhook** per alert channel: `critical` and `warnings`                            | Discord         | `discord-webhooks`  | Alertmanager posts alerts                                                                                                          |
| A third **optional** webhook, `ai`, for the `#ia` channel (daily and weekly AI reports) | Discord         | `discord-webhooks`  | Reports publish there ([how to create it](https://homelab.maceo.cl/en/operacion/logs/#create-the-ia-channel-and-its-webhook-once)) |
| The **ping URL** of a check                                                             | healthchecks.io | `healthchecks-ping` | Alertmanager sends a ping every 5 minutes                                                                                          |
| Grafana admin password                                                                  | 1Password       | `grafana-admin`     | Log in to Grafana                                                                                                                  |

A webhook is a URL that lets anyone **post messages** to that channel; it does not allow reading anything. The ping URL allows sending pings to the check. Both are secrets: don't paste them in Git, chat, or a ticket.

## Part 1 — Discord

You need to be a server admin (or have the **Manage Webhooks** permission) to create the webhooks.

This part is done **only from Discord** (web or app): creating roles, channels, or webhooks via API requires a bot with permissions in the server, and the HomeLab doesn't use any. The only thing you can do from the console is test a webhook and delete it.

### 1. A role for who sees alerts

*Server Settings → Roles → Create Role*, named `devops`. In the role's **Members** tab, add the people who should receive alerts. If the role already exists, skip this step.

### 2. A private category

Alerts include node and service names, and your server has people outside infrastructure, so don't use the existing channels (`despliegues`, `general`…).

1. Open the server menu (the name, top left) → **Create Category**.
1. **Category Name:** `🖥️ Infraestructura`. Enable **Private Category** and click **Next**.
1. Choose the `devops` role (and not `@everyone`) and finish with **Create Category**.

Channels created inside inherit this: only `devops` can see them.

### 3. Two channels

Hover over the category, click the **+** (**Create Channel**), choose **Text** and create:

| Channel Name       | Purpose                                                                                     |
| ------------------ | ------------------------------------------------------------------------------------------- |
| `alertas-criticas` | 🔴 Something needs attention now (node down, critical temperature, volume without replicas) |
| `alertas-avisos`   | 🟡 Worth a look, no rush                                                                    |

### 4. One webhook per channel

Repeat this for each channel:

1. *Server Settings → Integrations → Webhooks →* *Create Webhook* **(or** New Webhook\*\* if there are already some). A new webhook appears (usually named `Captain Hook`).
1. Open it and change:
   - **Name:** `Alertmanager críticas` (or `Alertmanager avisos`).
   - **Channel:** the corresponding channel (`alertas-criticas` or `alertas-avisos`). This is the field that decides where messages go: verify it didn't stay on the default channel.
1. Click **Copy Webhook URL** and paste it directly into the 1Password field (part 3). **Save Changes**.

The URL has this format, and the 1Password script rejects any other:

```
https://discord.com/api/webhooks/<id numérico>/<token>
```

To revoke a webhook:

Delete it with **Delete Webhook** on that same screen: the URL stops working instantly.

The Discord API lets you delete a webhook with its own URL, no bot or other credential needed. The URL is pasted without echo so it doesn't stay in history:

```
# pega la URL del webhook y Enter (no se ve)
read -rs HOOK
curl -fsS -X DELETE "$HOOK"
unset HOOK
```

`curl` prints nothing if it was deleted (Discord responds `204`); a `404` means it already didn't exist.

### 5. Get critical alerts on your phone

Right-click the channel → **Notification Settings** (or **Mute Channel** → options):

| Channel            | Setting                                           |
| ------------------ | ------------------------------------------------- |
| `alertas-criticas` | **All Messages** (reaches your phone)             |
| `alertas-avisos`   | **Only @mentions**, or muted: check when you want |

### 6. Test the webhook (optional)

Post a test message without leaving the URL in terminal history (it prompts silently):

```
# pega la URL y pulsa Enter (no se ve)
read -rs HOOK
curl -sS -H 'Content-Type: application/json' -d '{"content":"Prueba del webhook"}' "$HOOK"
unset HOOK
```

"Prueba del webhook" should appear in the channel; `curl` prints nothing if all goes well. A `404` error means the webhook doesn't exist (badly copied URL or deleted).

### 7. A webhook for the deployments channel (optional)

Only if you want [Argo CD notifications](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#414-argocd-notifications-to-discord-optional). Here we use your `despliegues` channel, which already exists and **is not in the private category**: the whole team sees it. It's another webhook, like the ones in step 4:

1. *Server Settings → Integrations → Webhooks →* *Create Webhook* **(or** New Webhook\*\*).
1. **Name:** `ArgoCD`. **Channel:** `despliegues`.
1. **Copy Webhook URL** and save it in the `deployments` field of the `discord-webhooks` item (part 3).

Sync failure alerts don't need another webhook: they reuse the `alertas-avisos` one.

## Part 2 — healthchecks.io

[healthchecks.io](https://healthchecks.io/) works as a **dead man's switch**: it expects to receive a ping every so often and, if it stops arriving, it notifies you. It's used to find out that monitoring (or the whole cluster) went down, something monitoring itself couldn't tell you. The free plan is enough.

### 1. Create an account

Sign up at [healthchecks.io](https://healthchecks.io/) with an email you have access to. A default project is created. This is **web-only**.

If you are going to create the check from the console (step 2), in the web UI go to the project **Settings** and create a **read-write API key**: creating it is also web-only. It is a secret: do not store it in a repo file.

### 2. Create the check

On the **Checks** page, click **Add Check** and fill in:

| Field          | Value               | Why                                                    |
| -------------- | ------------------- | ------------------------------------------------------ |
| **Name**       | `homelab-monitoreo` | This is the name you will see in the Discord alert     |
| **Schedule**   | **Simple**          | A ping at fixed intervals                              |
| **Period**     | `5 minutes`         | Alertmanager sends the ping every 5 minutes            |
| **Grace Time** | `10 minutes`        | Tolerates a late ping before marking the check as down |

The [healthchecks.io API](https://healthchecks.io/docs/api/) creates the check with the same values, in seconds (`timeout` is the *Period* and `grace` the *Grace Time*). The API key from step 1 is pasted without echo, so it does not remain in history:

```
# pega la API key (read-write) y Enter (no se ve)
read -rs HC_KEY
curl -fsS -H "X-Api-Key: $HC_KEY" \
  -d '{"name":"homelab-monitoreo","timeout":300,"grace":600}' \
  https://healthchecks.io/api/v3/checks/ | jq -r .ping_url
unset HC_KEY
```

Print the **ping URL** of the new check (the `ping_url` field in the response): this is the one for step 3.

With these values, the alert arrives about 15 minutes after the ping stops. Save the check.

### 3. Copy the ping URL

In the check, copy its **ping URL**. It has this format, and the 1Password script requires it to start with `https://`:

```
https://hc-ping.com/<uuid>
```

Paste it into the `url` field of the `healthchecks-ping` item (part 3). Anyone with this URL can send pings and keep the check green even if monitoring is down: treat it as a secret.

### 4. Connect the check with Discord

This step is **web-only**: the Discord integration is authorized in the browser, and the healthchecks.io API only allows listing integrations, not creating them.

1. At the top, go to **Integrations** and click **Add Integration**.
1. Choose **Discord** and click **Connect Discord**.
1. Discord asks you to authorize the healthchecks.io application: choose your server and the **`alertas-criticas`** channel, and authorize.
1. Back in healthchecks.io, the integration appears in the list; click **Test** to send a test message to that channel.
1. In **Checks**, verify that the Discord integration is enabled (**ON**) for `homelab-monitoreo`.

Not the same channel Alertmanager uses

healthchecks.io alerts arrive via its **own** Discord integration, not via the webhooks from part 1. This way they keep working even if the cluster is down, which is exactly the case this exists for.

### 5. First ping

The check shows as **New** until it receives its first ping. If you want to verify it manually, send one (replace `<ping URL>`):

```
read -rs PING; curl -fsS -m 10 --retry 3 "$PING"; unset PING
```

It should respond `OK` and the check switches to **Up**. Once monitoring is deployed, Alertmanager keeps it that way every 5 minutes.

## Part 3 — Store in 1Password

Items go in the **`HomeLab`** vault, as a **Secure Note**, with fields **outside any section** and with the exact name, because the script looks them up by name. How to create an item with custom fields: [maceosoft Platform — GitHub](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields).

| Item                | Field         | Content                                                                                                                                                  |
| ------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `discord-webhooks`  | `critical`    | `alertas-criticas` webhook (part 1, step 4)                                                                                                              |
|                     | `warnings`    | `alertas-avisos` webhook                                                                                                                                 |
|                     | `deployments` | *(optional)* `despliegues` channel webhook, for Argo CD notifications (part 1, step 7)                                                                   |
|                     | `ai`          | *(optional)* `#ia` channel webhook, for AI reports ([create it](https://homelab.maceo.cl/en/operacion/logs/#create-the-ia-channel-and-its-webhook-once)) |
| `healthchecks-ping` | `url`         | The check ping URL (part 2, step 3)                                                                                                                      |
| `grafana-admin`     | `password`    | A password generated with the 1Password generator                                                                                                        |
|                     | `username`    | *(optional)* admin username; defaults to `admin`                                                                                                         |

Create each item manually with the fields from the table ([how](https://homelab.maceo.cl/en/maceosoft/github/#how-to-create-an-item-with-custom-fields)). For `grafana-admin`, generate the password with the 1Password generator.

With the 1Password app unlocked and the CLI `op` installed, the three items are created with one command each. The URLs are secrets: they are read with `read -rs` (not visible when pasted and not kept in history) and cleared from the variable when done. If you have more than one 1Password account, export `OP_ACCOUNT` first.

```
# pega el webhook de alertas-criticas y Enter
read -rs DC_CRITICAL
# pega el webhook de alertas-avisos y Enter
read -rs DC_WARNINGS
op item create --category "Secure Note" --vault HomeLab --title discord-webhooks \
  "critical[password]=$DC_CRITICAL" "warnings[password]=$DC_WARNINGS"
unset DC_CRITICAL DC_WARNINGS

# Campos opcionales, en el mismo item (despliegues e #ia):
read -rs DC_DEPLOY; op item edit discord-webhooks "deployments[password]=$DC_DEPLOY"; unset DC_DEPLOY
read -rs DC_AI;     op item edit discord-webhooks "ai[password]=$DC_AI";                unset DC_AI

# pega la ping URL de healthchecks.io y Enter
read -rs HC_URL
op item create --category "Secure Note" --vault HomeLab --title healthchecks-ping "url[password]=$HC_URL"
unset HC_URL

op item create --category "Secure Note" --vault HomeLab --title grafana-admin \
  "password[password]=$(openssl rand -base64 24)" "username[text]=admin"
```

To view the Grafana password when you need it: `op read "op://HomeLab/grafana-admin/password"`. The item and field names are exactly what `apply_platform_secrets.py` reads; do not change them. Discord webhooks are created **web-only** (part 1); the healthchecks.io check can also be created via API (part 2, step 2).

Verify they are set correctly, without showing any values (from `ansible/`):

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
../.venv/bin/python scripts/apply_platform_secrets.py --check
```

It should show `✓` in `discord-webhooks`, `healthchecks-ping` and `grafana-admin`. If any Discord URL does not have the expected format, it reports it with `✗`.

## Next

With the items ready, continue at [4.13 — Monitoring and alerts](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional) (step 4: create the Secrets in the cluster). To rotate a webhook or the ping URL later: [Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/#monitoring-discord-webhooks-healthchecksio-ping-and-grafana).

# maceosoft Platform — GitHub

How the GitHub org **`maceosoft`** connects to the HomeLab cluster:

- **Argo CD** deploys from the private repo `maceosoft/gitops`.
- **ARC (Actions Runner Controller)** runs a self-hosted GitHub Actions runner inside the cluster (one scale set per organization: `arc-runners` for `maceosoft` and `arc-runners-symintel` for the legacy org; see [Two organizations](#two-organizations-one-scale-set-per-org)).
- **OpenTofu** manages the org from a pipeline that runs on one of those runners, and stores its state in the cluster.

Where it runs in the implementation guide

- **Steps 1 and 2 (manual):** [Phase 0 — GitHub and 1Password](https://homelab.maceo.cl/en/implementacion/fase-0-preparacion/#github-and-1password-maceosoft-org-manual-one-time-only)
- **Step 3 (platform secrets):** [Phase 4 — 4.2](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#42-bootstrap-gitops-applicationset), before the bootstrap ApplicationSet
- **Steps 4 and 5 (ARC and pipeline):** [Phase 4 — 4.11](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#411-arc-and-opentofu-pipeline)

This page is the complete reference: permissions, fields, verification, and rotation.

Repos involved

| Repo                                                                      | What it contains                                                                 |
| ------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| [`maceosoft/homelab`](https://github.com/maceosoft/homelab)               | This site. Ansible: k3s, Argo CD, and **platform secrets** (1Password → cluster) |
| [`maceosoft/gitops`](https://github.com/maceosoft/gitops)                 | Argo CD Applications, including ARC and the OpenTofu runner                      |
| [`maceosoft/core-pipelines`](https://github.com/maceosoft/core-pipelines) | Reusable workflows (`tofu.yml`, CI, FTP deploy)                                  |
| [`maceosoft/infra`](https://github.com/maceosoft/infra)                   | OpenTofu: org repos, environments, and secrets                                   |

## How it fits

```
flowchart LR
  OP[1Password<br/>bóveda HomeLab] -->|playbook-platform-secrets.yml| K8S
  subgraph K8S[Clúster k3s]
    ARGO[ArgoCD] -->|App maceosoft-argocd| GITOPS[(maceosoft/gitops)]
    ARGO --> ARC[ARC controller]
    ARC --> RT[arc-runners]
    RT --> STATE[(ns terraform<br/>Secret tfstate-default-github)]
  end
  INFRA[maceosoft/infra] -->|tofu.yml de core-pipelines| RT
  RT -->|App maceosoft-terraform| GH[org maceosoft]
```

Some pieces **must exist before** Argo CD or the pipeline can work. Those are created **by hand, once**, and OpenTofu neither manages nor imports them:

| What                                      | Why manual                                                                                     |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Repos `gitops`, `infra`, `core-pipelines` | Without them there's nowhere to deploy from and no pipeline to run OpenTofu                    |
| App `maceosoft-argocd`                    | Argo CD needs it to read the private repo that deploys everything else                         |
| GitHub Apps and OAuth App                 | GitHub doesn't allow creating OAuth Apps via API, and GitHub Apps require browser confirmation |

Everything else (repo `api` and future repos) is created by the `infra` pipeline.

## 1Password Items

All in the **`HomeLab`** vault. Scripts read them through the 1Password desktop app, which must be unlocked and have *Settings → Developer → Integrate with other apps* enabled. To indicate **which account** to use, define **`ONEPASSWORD_ACCOUNT_NAME`** with your account name as it appears in the top left of the app sidebar (for example `Mi Cuenta`; if not defined, it uses `Personal`):

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
```

| Item                                | Item type   | Fields                                                                                                                                                                | Used by                                                                                                                                                                                                                                                         |
| ----------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `maceosoft-terraform`               | Secure Note | `app_id`, `installation_id`, `private_key`                                                                                                                            | OpenTofu runner (`TF_VAR_github_app_*`)                                                                                                                                                                                                                         |
| `maceosoft-arc-runners`             | Secure Note | `app_id`, `installation_id`, `private_key`                                                                                                                            | ARC (runner registration)                                                                                                                                                                                                                                       |
| `maceosoft-argocd`                  | Secure Note | `app_id`, `installation_id`, `private_key`                                                                                                                            | Argo CD reads `maceosoft/gitops` (`argocd/repo-gitops`)                                                                                                                                                                                                         |
| `symintel-arc-runners` *(optional)* | Secure Note | `app_id`, `installation_id`, `private_key`                                                                                                                            | ARC for the **legacy org** (`symintel`): its scale set registration. See [Two organizations](#two-organizations-one-scale-set-per-org)                                                                                                                          |
| `symintel-terraform` *(optional)*   | Secure Note | `app_id`, `installation_id`, `private_key`                                                                                                                            | OpenTofu for the legacy org; without it, those runners lack OpenTofu credentials                                                                                                                                                                                |
| `sealed-secrets`                    | Secure Note | `certificate`, `private_key` *(optional until Phase 6)*                                                                                                               | Sealed Secrets encryption key: survives K3s reinstall                                                                                                                                                                                                           |
| `maceosoft-dex`                     | Secure Note | `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET` (+ `VCLUSTER_CLIENT_SECRET`, generated by the playbook)                                                                    | Dex (GitHub login)                                                                                                                                                                                                                                              |
| `discord-webhooks`                  | Secure Note | `critical`, `warnings`: Discord webhook URL per channel; `deployments` *(optional)*: the `despliegues` channel one for Argo CD notifications *(optional: monitoring)* | Alertmanager sends alerts ([4.13](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#413-monitoring-and-alerts-optional)). How to create them: [Discord and healthchecks.io](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#part-1-discord) |
| `healthchecks-ping`                 | Secure Note | `url`: healthchecks.io check ping URL *(optional: monitoring)*                                                                                                        | Alertmanager pings it every 5 minutes to signal monitoring is alive. How to create it: [Discord and healthchecks.io](https://homelab.maceo.cl/en/maceosoft/discord-healthchecks/#part-2-healthchecksio)                                                         |
| `grafana-admin`                     | Secure Note | `password` and, optionally, `username` (default `admin`) *(optional: monitoring)*                                                                                     | Grafana admin                                                                                                                                                                                                                                                   |
| `ftp-<repo>`                        | Secure Note | `dev_host`, `dev_user`, `dev_password`, `dev_remote_dir`, `prod_host`, `prod_user`, `prod_password`, `prod_remote_dir`                                                | OpenTofu → `<repo>` FTP secrets (`TF_VAR_ftp`)                                                                                                                                                                                                                  |

### How to create an item with custom fields

The playbook looks up each value by its path `op://HomeLab/<item>/<campo>`, so **the item title and every field name must be exact** (lowercase, with underscores).

1. In 1Password open the **`HomeLab`** vault → **New Item** → **Secure Note**.
1. In **Title** type the item name, for example `maceosoft-terraform`.
1. For each field: **+ Add more** → choose the type (**Text** or **Password**) → in the field label type the exact name (for example `app_id`) → paste the value.
1. Add the fields directly on the item. **Do not create a section** to group them: with a section the path changes and the playbook won't find them.
1. Save.

Use **Password** for secret values (they are hidden) and **Text** for the rest. Each table below tells which type each field uses.

You can also create the item with the 1Password CLI (`op`), with 1Password unlocked. Each field is written as `nombre[tipo]=valor`, where the type is `password` (secret value) or `text`; `op` creates loose fields, without a section. Secret values are not typed in the command: they are pasted without echo using `read -rs`:

```
# pega la contraseña y Enter (no se ve)
read -rs FTP_DEV_PASS
op item create --category "Secure Note" --vault HomeLab --title ftp-mi-repo \
  "dev_host[text]=ftp.ejemplo.com" "dev_user[text]=usuario" "dev_password[password]=$FTP_DEV_PASS"
unset FTP_DEV_PASS
```

If you have more than one 1Password account, export `OP_ACCOUNT` with yours first. For a random value use `$(openssl rand -hex 32)`. Local AI items (router tokens, Open Terminal…) already have their commands in [Secrets Management](https://homelab.maceo.cl/en/secrets/#local-ai-create-tokens-with-op).

## Step 1 — GitHub Apps and OAuth App (manual)

Do this with an **organization owner** account `maceosoft`.

### `maceosoft-terraform`

[maceosoft → Settings → Developer settings → GitHub Apps → New GitHub App](https://github.com/organizations/maceosoft/settings/apps/new)

| Field                                   | Value                          |
| --------------------------------------- | ------------------------------ |
| GitHub App name                         | `maceosoft-terraform`          |
| Homepage URL                            | `https://github.com/maceosoft` |
| Webhook → Active                        | unchecked                      |
| Where can this GitHub App be installed? | Only on this account           |

The rest of the form (Callback URL, Setup URL, *Request user authorization*, events) is left empty or unchecked.

| Repository permission | Access         | Purpose                            |
| --------------------- | -------------- | ---------------------------------- |
| Actions               | Read and write | Actions management in repos        |
| Administration        | Read and write | Create repos and branch protection |
| Contents              | Read and write | Repo files                         |
| Environments          | Read and write | Environments `dev` / `prod`        |
| Metadata              | Read-only      | Required                           |
| Secrets               | Read and write | FTP secrets per environment        |
| Variables             | Read and write | `PROJECT_STACK`, `FTP_PROTOCOL`    |
| Workflows             | Read and write | `.github/workflows/*` files        |

| Organization permission | Access         | Purpose                 |
| ----------------------- | -------------- | ----------------------- |
| Administration          | Read and write | Create repos in the org |

All other permissions remain at *No access*.

After *Create GitHub App*, gather three pieces of data and save them in the **`maceosoft-terraform`** item in 1Password (Secure Note, see [how to create it](#como-crear-un-item-con-campos-propios)):

| Field in 1Password | Type     | Data            | Where in GitHub                                                                                                                                                                                                                                                                    |
| ------------------ | -------- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `app_id`           | Text     | App ID          | App page, *General* tab: **App ID** line (a number, e.g. `1234567`)                                                                                                                                                                                                                |
| `private_key`      | Password | Private key     | Same page, bottom under *Private keys* → **Generate a private key**. A `maceosoft-terraform.<fecha>.private-key.pem` file downloads. Open it with a text editor and copy **everything**, including the `-----BEGIN RSA PRIVATE KEY-----` and `-----END RSA PRIVATE KEY-----` lines |
| `installation_id`  | Text     | Installation ID | App left menu → **Install App** → *Install* next to `maceosoft` → **All repositories** → *Install*. GitHub takes you to `https://github.com/organizations/maceosoft/settings/installations/<número>`: that final number                                                            |

The item looks like this:

```
maceosoft-terraform        (Nota segura · bóveda HomeLab)
├── app_id                1234567
├── installation_id       87654321
└── private_key           ••••••••   (el .pem completo)
```

**Done only from the web:** create the App, generate its private key (*Generate a private key*) and install it. GitHub offers no CLI or API for this (manifest-based App creation also requires browser confirmation). Saving the data in 1Password can be done either way:

When pasting the `.pem` into a Password field, line breaks are lost. Doesn't matter: the playbook reconstructs the key. Once the item is saved, **delete the `.pem`** from Downloads.

What *is* automated is reading the two numbers and saving the item.

1. List installed Apps in the org and extract `App ID` and `Installation ID` (needs the organization permission: `gh auth refresh -s admin:read:org` once):

   ```
   gh api orgs/maceosoft/installations --jq '.installations[] | [.app_slug, .app_id, .id, .repository_selection] | @tsv'
   ```

1. With the downloaded `.pem`, create the item (1Password unlocked; if you have more than one 1Password account, export `OP_ACCOUNT` first). The `.pem` value is read from the file and never typed in the command:

   ```
   APP=maceosoft-terraform
   read -r APP_ID INST_ID < <(gh api orgs/maceosoft/installations \
     --jq ".installations[] | select(.app_slug==\"$APP\") | \"\(.app_id) \(.id)\"")
   op item create --category "Secure Note" --vault HomeLab --title "$APP" \
     "app_id[text]=$APP_ID" "installation_id[text]=$INST_ID" \
     "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
   ```

1. Verify without showing values and delete the `.pem`:

   ```
   python3 ansible/scripts/apply_platform_secrets.py --check
   # en macOS: rm -P
   shred -u ~/Downloads/$APP.*.private-key.pem
   ```

The same command works for the other Apps: just change `APP` (`maceosoft-arc-runners`, `maceosoft-argocd`, and `symintel-arc-runners` / `symintel-terraform` if using the old org, changing `orgs/maceosoft` to `orgs/symintel`).

### `maceosoft-arc-runners`

This is the App ARC uses to register the runner `arc-runners` in the org. It is created in the same form as `maceosoft-terraform`, but it is a **separate App**: its own name, permissions, and credentials.

[maceosoft → Settings → Developer settings → GitHub Apps → New GitHub App](https://github.com/organizations/maceosoft/settings/apps/new)

| Field                                   | Value                          |
| --------------------------------------- | ------------------------------ |
| GitHub App name                         | `maceosoft-arc-runners`        |
| Homepage URL                            | `https://github.com/maceosoft` |
| Webhook → Active                        | unchecked                      |
| Where can this GitHub App be installed? | Only on this account           |

The rest of the form (Callback URL, Setup URL, *Request user authorization*, events) is left empty or unchecked.

| Repository permission | Access         | Purpose                                  |
| --------------------- | -------------- | ---------------------------------------- |
| Actions               | Read-only      | View queued jobs the runner must pick up |
| Administration        | Read and write | Register runners (requested by ARC)      |
| Metadata              | Read-only      | Required                                 |

| Organization permission | Access         | Purpose                                                 |
| ----------------------- | -------------- | ------------------------------------------------------- |
| Self-hosted runners     | Read and write | Register and delete the runner `arc-runners` in the org |

All other permissions remain at *No access*.

After *Create GitHub App*, save its three pieces of data in the **`maceosoft-arc-runners`** item in 1Password (Secure Note, see [how to create it](#como-crear-un-item-con-campos-propios)):

| 1Password field   | Type     | Data            | Where in GitHub                                                                                                                                                                                                                                                            |
| ----------------- | -------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `app_id`          | Text     | App ID          | App page, *General* tab: **App ID** line                                                                                                                                                                                                                                   |
| `private_key`     | Password | Private key     | Same page, bottom under *Private keys* → **Generate a private key**. Downloads `maceosoft-arc-runners.<fecha>.private-key.pem`. Open with a text editor and copy **everything**, including the `-----BEGIN RSA PRIVATE KEY-----` and `-----END RSA PRIVATE KEY-----` lines |
| `installation_id` | Text     | Installation ID | App left menu → **Install App** → *Install* next to `maceosoft` → **All repositories** → *Install*. The number at the end of the URL `https://github.com/organizations/maceosoft/settings/installations/<número>`                                                          |

The item looks like this:

```
maceosoft-arc-runners      (Nota segura · bóveda HomeLab)
├── app_id                2345678
├── installation_id       98765432
└── private_key           ••••••••   (el .pem completo)
```

Creating the App, its key, and installing it is **only done from the web**; saving the item can be done either way:

The three values are **different** from those of `maceosoft-terraform`: each App has its own App ID, its own key, and its own installation. Do not copy the ones from the other App. Once the item is saved, **delete the `.pem`** from Downloads.

The same [`maceosoft-terraform`](#maceosoft-terraform) command with `APP=maceosoft-arc-runners`: `gh api` outputs this App's numbers, so they don't get mixed up with the other one.

```
APP=maceosoft-arc-runners
read -r APP_ID INST_ID < <(gh api orgs/maceosoft/installations \
  --jq ".installations[] | select(.app_slug==\"$APP\") | \"\(.app_id) \(.id)\"")
op item create --category "Secure Note" --vault HomeLab --title "$APP" \
  "app_id[text]=$APP_ID" "installation_id[text]=$INST_ID" \
  "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
python3 ansible/scripts/apply_platform_secrets.py --check
# en macOS: rm -P
shred -u ~/Downloads/$APP.*.private-key.pem
```

### OAuth App `Maceosoft Dex`

[maceosoft → Settings → Developer settings → OAuth Apps → New OAuth App](https://github.com/organizations/maceosoft/settings/applications/new)

| Field            | Value                                                           |
| ---------------- | --------------------------------------------------------------- |
| Application name | `Maceosoft Dex`                                                 |
| Homepage URL     | `https://argocd.homelab.local`                                  |
| Redirect URIs    | `https://argocd.homelab.local/api/dex/callback` (only this one) |

After *Register application*, create the **`maceosoft-dex`** item in the `HomeLab` vault (Secure Note, see [how to create it](#como-crear-un-item-con-campos-propios)) with these two fields:

| 1Password field        | Type     | Data          | Where in GitHub                                                                                     |
| ---------------------- | -------- | ------------- | --------------------------------------------------------------------------------------------------- |
| `GITHUB_CLIENT_ID`     | Text     | Client ID     | OAuth App page: **Client ID** line                                                                  |
| `GITHUB_CLIENT_SECRET` | Password | Client secret | Same page → **Generate a new client secret**. GitHub shows it **only once**: copy it at that moment |

The item looks like this (the last field is added automatically by `playbook-dex-oauth-secrets.yml` the first time it runs: it's the vCluster Dex client secret, not from GitHub):

```
maceosoft-dex              (Nota segura · bóveda HomeLab)
├── GITHUB_CLIENT_ID      Ov23li…
├── GITHUB_CLIENT_SECRET  ••••••••
└── VCLUSTER_CLIENT_SECRET ••••••••  (lo genera el playbook)
```

**Creating the OAuth App and generating its client secret is only done from the web**: GitHub has no API for OAuth Apps. Saving the two values in 1Password can be done either way:

Create the Secure Note `maceosoft-dex` and paste each value from the table into its field ([how to create it](#como-crear-un-item-con-campos-propios)).

Once you have the two values, save them without displaying them on screen (if you have more than one 1Password account, export `OP_ACCOUNT` first):

```
# pega el Client ID y Enter
read -r  GH_CID
# pega el Client secret y Enter
read -rs GH_CSEC
op item create --category "Secure Note" --vault HomeLab --title maceosoft-dex \
  "GITHUB_CLIENT_ID[text]=$GH_CID" "GITHUB_CLIENT_SECRET[password]=$GH_CSEC"
unset GH_CID GH_CSEC
```

To verify someone is in the team `devops` (or add them) from the console: `gh api orgs/maceosoft/teams/devops/memberships/<usuario>` (returns `"state": "active"`) and, to add them as a member, `gh api -X PUT orgs/maceosoft/teams/devops/memberships/<usuario> -f role=member`.

Only the **`devops`** team from `maceosoft` can enter, and its members become Argo CD admins. This is applied in [Phase 4 step 4.7](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#47-oauth-secrets-github-vcluster-incus-ui).

## Step 2 — Repos, Argo CD App & FTP (manual)

1. **Repos** (private), created in `maceosoft` with their content pushed via git:

   - `gitops`.
   - `core-pipelines`: afterwards, make it accessible to the other repos in the org (below). While testing, the other repos use it with `@main`; the tag `v1` is created once everything works (see the README of `core-pipelines`).
   - `infra`.

   Create each repo in the org with **New repository** (private) and push its content with git. Then, in `core-pipelines`: *Settings → Actions → General → Access*, choose "Accessible from repositories in the 'maceosoft' organization".

   With the GitHub CLI, from each project's folder:

   ```
   gh repo create maceosoft/gitops --private --source . --remote origin --push
   # igual para core-pipelines e infra
   gh repo list maceosoft --json name,visibility --jq '.[] | [.name, .visibility] | @tsv'
   # core-pipelines accesible desde los demás repos de la org
   gh api -X PUT repos/maceosoft/core-pipelines/actions/permissions/access -f access_level=organization
   ```

   1. **GitHub App `maceosoft-argocd`**, used by Argo CD to read the private repo `gitops`. Comes after the repos because it is installed **only** on `gitops`. (Deploy keys are disabled in the organization, so an App is used.)

   [maceosoft → Settings → Developer settings → GitHub Apps → New GitHub App](https://github.com/organizations/maceosoft/settings/apps/new)

   | Field                                   | Value                          |
   | --------------------------------------- | ------------------------------ |
   | GitHub App name                         | `maceosoft-argocd`             |
   | Homepage URL                            | `https://github.com/maceosoft` |
   | Webhook → Active                        | unchecked                      |
   | Where can this GitHub App be installed? | Only on this account           |

   The rest of the form is left empty or unchecked.

   | Repository permission | Access    | Purpose        |
   | --------------------- | --------- | -------------- |
   | Contents              | Read-only | Clone the repo |
   | Metadata              | Read-only | Required       |

   No organization permissions. All others remain *No access*.

   After *Create GitHub App*, save its three pieces of data in the **`maceosoft-argocd`** item in 1Password (Secure Note, see [how to create it](#como-crear-un-item-con-campos-propios)):

   | 1Password field   | Type     | Data            | Where in GitHub                                                                                                                                                                                    |
   | ----------------- | -------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | `app_id`          | Text     | App ID          | App page, *General* tab: **App ID** line                                                                                                                                                           |
   | `private_key`     | Password | Private key     | Same page, bottom under *Private keys* → **Generate a private key**. Copy the **entire** downloaded `.pem`, including the `BEGIN`/`END` lines                                                      |
   | `installation_id` | Text     | Installation ID | Left menu → **Install App** → *Install* next to `maceosoft` → **Only select repositories** → choose **`gitops`** → *Install*. The number at the end of the URL `…/settings/installations/<número>` |

   Install it **only on `gitops`**: with that key you can read the code of all repos where it is installed. Delete the `.pem` when finished.

   Creating the App, its key, and installing it is **web-only**; saving the item can be done either way:

   Create the Secure Note `maceosoft-argocd` and paste each piece of data from the table into its field ([how to create it](#como-crear-un-item-con-campos-propios)).

   The same command from [`maceosoft-terraform`](#maceosoft-terraform), with `APP=maceosoft-argocd`:

   ```
   APP=maceosoft-argocd
   read -r APP_ID INST_ID < <(gh api orgs/maceosoft/installations \
     --jq ".installations[] | select(.app_slug==\"$APP\") | \"\(.app_id) \(.id)\"")
   op item create --category "Secure Note" --vault HomeLab --title "$APP" \
     "app_id[text]=$APP_ID" "installation_id[text]=$INST_ID" \
     "private_key[password]=$(cat ~/Downloads/$APP.*.private-key.pem)"
   # en macOS: rm -P
   shred -u ~/Downloads/$APP.*.private-key.pem
   ```

1. **FTP (File Transfer Protocol) credentials for repo `api`**, the first project in the map with `deploy = true`: title **`ftp-api`**, type Secure Note. For each repo you add later, follow the [procedure — Add a repo](https://homelab.maceo.cl/en/operacion/agregar-repo/index.md). Fields, [created the same way as the others](#como-crear-un-item-con-campos-propios):

   | Field             | Type     | Example                                   |
   | ----------------- | -------- | ----------------------------------------- |
   | `dev_host`        | Text     | `dev.tu-cpanel.example.com`               |
   | `dev_user`        | Text     | `usuario_dev`                             |
   | `dev_password`    | Password |                                           |
   | `dev_remote_dir`  | Text     | `/home/usuario/dev.maceosoft.example.com` |
   | `prod_host`       | Text     | `tu-cpanel.example.com`                   |
   | `prod_user`       | Text     | `usuario_prod`                            |
   | `prod_password`   | Password |                                           |
   | `prod_remote_dir` | Text     | `/home/usuario/public_html`               |

   Change the hosts, users, and folders to yours; the two passwords are pasted without echo (if you have more than one 1Password account, export `OP_ACCOUNT` first):

   ```
   # pega cada contraseña y Enter
   read -rs FTP_DEV_PASS;  read -rs FTP_PROD_PASS
   op item create --category "Secure Note" --vault HomeLab --title ftp-api \
     "dev_host[text]=dev.tu-cpanel.example.com" "dev_user[text]=usuario_dev" \
     "dev_password[password]=$FTP_DEV_PASS" "dev_remote_dir[text]=/home/usuario/dev.maceosoft.example.com" \
     "prod_host[text]=tu-cpanel.example.com" "prod_user[text]=usuario_prod" \
     "prod_password[password]=$FTP_PROD_PASS" "prod_remote_dir[text]=/home/usuario/public_html"
   unset FTP_DEV_PASS FTP_PROD_PASS
   ```

1. **Sealed Secrets key** *(optional: needed when you start storing `SealedSecret` in Git, e.g., in Phase 6)*. Sealed Secrets encrypts secrets with a key pair that, by default, the controller generates inside the cluster: if you reinstall K3s it is lost, and the `SealedSecret` already in Git can no longer be decrypted. That's why it is generated **once**, saved in 1Password, and the secrets playbook brings it to the cluster before the controller starts.

   Generate the pair in a temporary folder (`openssl` ships with macOS and Linux):

   ```
   cd "$(mktemp -d)"
   openssl req -x509 -days 3650 -nodes -newkey rsa:4096 \
     -keyout tls.key -out tls.crt -subj "/CN=sealed-secret/O=sealed-secret"
   ```

   Create in 1Password the item **`sealed-secrets`** (Secure Note, [created the same way as the others](#como-crear-un-item-con-campos-propios)):

   | Field         | Type     | Value                                                                                                                            |
   | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------- |
   | `certificate` | Text     | The full content of `tls.crt` (the **public** key), with the `-----BEGIN CERTIFICATE-----` and `-----END CERTIFICATE-----` lines |
   | `private_key` | Password | The full content of `tls.key` (the **private** key), with its `BEGIN`/`END` lines                                                |

   To copy a file: `pbcopy < tls.crt` (macOS) or open it with an editor and copy everything.

   With the 1Password CLI the files are read directly (if you have more than one 1Password account, export `OP_ACCOUNT` first):

   ```
   op item create --category "Secure Note" --vault HomeLab --title sealed-secrets \
     "certificate[text]=$(cat tls.crt)" "private_key[password]=$(cat tls.key)"
   python3 ansible/scripts/apply_platform_secrets.py --check
   ```

   Once the item is saved, **delete the files**:

   ```
   rm -f tls.key tls.crt
   ```

   The certificate is valid for 10 years. The playbook verifies that the certificate and key are a pair (if you copied files from different pairs, it stops with a clear message) and creates the Secret `kube-system/sealed-secrets-key-homelab`, which the controller detects by its label `sealedsecrets.bitnami.com/sealed-secrets-key: active`. While the item doesn't exist, the playbook warns and continues: Sealed Secrets would generate its own key.

## Step 3 — Cluster and platform secrets

With k3s installed ([Phase 3](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md), Argo CD included):

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
export ONEPASSWORD_VAULT="HomeLab"
cd ansible
ansible-playbook -i inventory.ini playbook-platform-secrets.yml
```

[`playbook-platform-secrets.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-platform-secrets.yml) creates the namespaces `argocd`, `arc-runners` (PSS `privileged`, for the `dind` sidecar) and `terraform` (only stores OpenTofu state), and these Secrets:

| Secret                                                   | Namespace              | Content                                                                                                                                                                                                                         |
| -------------------------------------------------------- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `repo-gitops`                                            | `argocd`               | App `maceosoft-argocd`: Argo CD can clone `https://github.com/maceosoft/gitops.git`                                                                                                                                             |
| `arc-runners-github-app`                                 | `arc-runners`          | ARC App                                                                                                                                                                                                                         |
| `terraform-app`                                          | `arc-runners`          | `TF_VAR_github_app_id`, `TF_VAR_github_app_installation_id`, `TF_VAR_github_app_private_key`                                                                                                                                    |
| `tofu-vars`                                              | `arc-runners`          | `TF_VAR_ftp` (all `ftp-<repo>` items)                                                                                                                                                                                           |
| `arc-runners-github-app`, `terraform-app`                | `arc-runners-symintel` | Same for the old org, from `symintel-arc-runners` and `symintel-terraform` (only if they exist). The namespace is created with PSS `baseline`                                                                                   |
| `sealed-secrets-key-homelab`                             | `kube-system`          | Sealed Secrets key pair (item `sealed-secrets`, if exists)                                                                                                                                                                      |
| `discord-webhooks`, `healthchecks-ping`, `grafana-admin` | `monitoring`           | Discord webhooks, healthchecks.io ping URL, and Grafana password (items with the same names; only if all three exist). The namespace is created with PSS `privileged`, because `node-exporter` uses the node's disk and network |

Single runner

It's a HomeLab: the single scale set (`arc-runners`, Default group) runs CI (continuous integration), deploys and OpenTofu, and has the OpenTofu credentials as environment variables. Any `maceosoft` repo that runs a job there can reach them. Keep the org with only your repos.

Always against the HomeLab

The playbook uses `~/.kube/homelab-k3s.yaml` (the one left by `k3s_fetch_kubeconfig`), **not** the current `kubectl` context. This way it doesn't write secrets to another cluster. For another kubeconfig: `-e platform_kubeconfig=/ruta`.

It's idempotent: run it again whenever you rotate something in 1Password or add a `ftp-<repo>` item.

## Two organizations: one scale set per org

An ARC scale set registers with **a single** GitHub organization (`githubConfigUrl`). For the same runners to serve the new org (`maceosoft`) and the old org (`symintel`) there are **two scale sets**, with a single controller (`arc-controller`) serving both:

|                                | Org `maceosoft`                    | Org `symintel` (old)                     |
| ------------------------------ | ---------------------------------- | ---------------------------------------- |
| Application                    | `arc-runners`                      | `arc-runners-symintel`                   |
| Namespace                      | `arc-runners`                      | `arc-runners-symintel`                   |
| ARC App (1Password item)       | `maceosoft-arc-runners`            | `symintel-arc-runners`                   |
| Name in `runs-on:`             | `arc-runners`                      | `arc-runners` (the same)                 |
| Cache `/cache` (RWX, Longhorn) | 20 GiB                             | 10 GiB, separate                         |
| Docker (`dind`)                | Yes, with a privileged sidecar     | Yes, the same                            |
| OpenTofu and FTP               | Yes (`terraform-app`, `tofu-vars`) | Only if item `symintel-terraform` exists |

Both use **the same runner template** ([`values-scaleset.yaml`](https://github.com/maceosoft/gitops/blob/main/arc/values-scaleset.yaml)); the old org changes `githubConfigUrl` ([`values-scaleset-symintel.yaml`](https://github.com/maceosoft/gitops/blob/main/arc/values-scaleset-symintel.yaml)). Each namespace has its own Secrets, with **the same name**, so those of one org aren't visible to the other, and the cache isn't shared either (a poisoned cache in one org doesn't reach the other).

To enable the old org: create the ARC App in `symintel` (same as [`maceosoft-arc-runners`](#maceosoft-arc-runners), but in that org) and save it in item `symintel-arc-runners`; then run `playbook-platform-secrets.yml`. If the item doesn't exist, the script doesn't create the namespace or the Secret and the old org's scale set doesn't start. To remove it, delete the Application `arc-runners-symintel` and the items.

## Step 4 — GitOps

```
kubectl apply -f gitops/bootstrap/root-appset.yaml
```

Argo CD syncs, among others, these Applications:

| Application      | What it deploys                                                                      |
| ---------------- | ------------------------------------------------------------------------------------ |
| `arc-controller` | ARC Controller (`arc-systems`)                                                       |
| `terraform-rbac` | ServiceAccount `tofu-runner` + Role (only Secrets/Leases from namespace `terraform`) |
| `arc-runners`    | The runner (up to 3 in parallel), with namespace Secrets as environment variables    |

## Step 5 — First OpenTofu pipeline

Push to `main` of `infra`, or trigger it from *Actions → tofu → Run workflow* (with the CLI: `gh workflow run tofu.yml --repo maceosoft/infra --ref main` and `gh run watch --repo maceosoft/infra`). The workflow [`tofu.yml`](https://github.com/maceosoft/core-pipelines/blob/main/.github/workflows/tofu.yml) runs on `arc-runners`: it does `plan` on PRs (pull requests) and `apply` on `main`. On the first run it creates `api`.

## Verify

Before the cluster, ensure 1Password has everything (from the root of `homelab`):

```
# tu cuenta
export ONEPASSWORD_ACCOUNT_NAME="Mi Cuenta"
# items y campos: todo ✓
.venv/bin/python ansible/scripts/apply_platform_secrets.py --check
# Secrets que se van a crear, sin valores
.venv/bin/python ansible/scripts/apply_platform_secrets.py --dry-run
```

After the playbook and GitOps:

```
kubectl -n argocd get secret repo-gitops
kubectl -n arc-runners get secret \
  arc-runners-github-app terraform-app tofu-vars
# org antigua, si la usas:
kubectl -n arc-runners-symintel get secret arc-runners-github-app
kubectl -n argocd get applications | grep -E 'arc|terraform'
# controller + listener del scale set
kubectl -n arc-systems get pods
# después del primer apply de OpenTofu
kubectl -n terraform get secret tfstate-default-github
```

## Rotate credentials

Full step-by-step: [Operations — Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md).

| What                                                           | How                                                                                                                                                                                                                                      |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Private key of a GitHub App                                    | Generate a new one in GitHub (web only), replace `private_key` in 1Password (`op item edit <item> "private_key[password]=$(cat llave.pem)"`), re-run `playbook-platform-secrets.yml` and delete the old one in GitHub                    |
| Private key of `maceosoft-argocd`                              | Same as other Apps: new key in GitHub, replace `private_key`, re-run the playbook and delete the old one                                                                                                                                 |
| FTP credentials                                                | Edit `ftp-<repo>` (`op item edit ftp-<repo> "prod_password[password]=..."`), re-run the playbook and trigger the `infra` pipeline (`gh workflow run tofu.yml --repo maceosoft/infra --ref main`)                                         |
| Discord webhook, healthchecks.io ping URL, or Grafana password | Change the field in its item, re-run `playbook-platform-secrets.yml` and follow [Operations — Rotate credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/#monitoring-discord-webhooks-healthchecksio-ping-and-grafana) |
| Dex client secret                                              | New secret in the OAuth App, update `GITHUB_CLIENT_SECRET` and run `playbook-dex-oauth-secrets.yml`                                                                                                                                      |

# vCluster Platform

Part of the implementation guide

**[Phase 5 — vCluster](https://homelab.maceo.cl/en/implementacion/fase-5-vcluster/index.md)** — executable steps. Requires **[Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)** (Dex, Gateway).

Deep dive supplement. vCluster Platform offers **UI (web interface)**, **SSO (single sign-on) via Argo CD Dex**, and **kubeconfig download** for the management K3s (connected cluster) and vClusters, without relying solely on CLI.

## What it does

| Capability       | Description                                                                                              |
| ---------------- | -------------------------------------------------------------------------------------------------------- |
| Virtual clusters | Isolated Kubernetes API within the management K3s, without lifting Incus VMs                             |
| Web UI           | Create/delete vClusters, view status, connect from browser                                               |
| SSO              | Login OIDC via **Dex** → **GitHub** (team `devops` of org `maceosoft`); credentials in 1Password         |
| Kubeconfig       | **Download kubeconfig** / **Connect** for the **host K3s** (connected cluster) and for each **vCluster** |

Sync **manual** — Application `vcluster-platform` (commented in the ApplicationSet `homelab-root`: manually activated).

## Components involved

| Resource                                                                                                       | Purpose                                                                                                     |
| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| [`vcluster-platform.yaml`](https://github.com/maceosoft/gitops/blob/main/argocd/apps/vcluster-platform.yaml)   | Helm Application that installs Loft Platform in `vcluster-platform`                                         |
| [`vcluster/values/platform.yaml`](https://github.com/maceosoft/gitops/blob/main/vcluster/values/platform.yaml) | HTTPRoute `vcluster.homelab.local` (`vcluster/route`), OIDC issuer → Argo CD Dex                            |
| [`argocd/config/dex.config`](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config)           | GitHub connector + OAuth client `vcluster-platform`                                                         |
| [`argocd/secrets/1password.md`](https://github.com/maceosoft/gitops/blob/main/argocd/secrets/1password.md)     | 1Password items → `argocd-secret`                                                                           |
| **`argocd-route`**                                                                                             | Publishes Argo CD/Dex in `argocd.homelab.local` (see [GitOps](https://homelab.maceo.cl/en/gitops/index.md)) |

## Authentication flow

```
sequenceDiagram
  participant U as Usuario
  participant VP as vcluster_homelab_local
  participant Dex as argocd_api_dex
  U->>VP: Login SSO
  VP->>Dex: OIDC authorize
  Dex->>U: GitHub OAuth
  Dex->>VP: token
  VP->>U: UI + kubeconfig
```

| URL                                                                | Role                                 |
| ------------------------------------------------------------------ | ------------------------------------ |
| [`https://argocd.homelab.local`](https://argocd.homelab.local)     | Argo CD UI; Dex issuer in `/api/dex` |
| [`https://vcluster.homelab.local`](https://vcluster.homelab.local) | vCluster Platform UI                 |

## Implementation (summary)

Follow the **[Phase 4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md)** until OAuth Dex and the Gateway are complete. Then:

### 1. OAuth Secrets (step 4.7 of the GitOps guide)

Ansible generates and applies the local secrets; manually create `GITHUB_CLIENT_ID` and `GITHUB_CLIENT_SECRET` in 1Password:

```
cd ansible
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml
# Tras desplegar Platform (paso 2):
ansible-playbook -i inventory.ini playbook-dex-oauth-secrets.yml \
  --tags vcluster -e dex_oauth.apply_vcluster=true
```

### 2. Deploy Platform

```
kubectl apply -f gitops/argocd/apps/vcluster-platform.yaml
```

Argo CD UI → `vcluster-platform` → **Sync**.

With the Argo CD CLI:

```
argocd login argocd.homelab.local --grpc-web
argocd app sync vcluster-platform
```

### 3. Validate GitHub SSO

After syncing `argocd`, login to Argo CD or Platform should offer **GitHub**. If it fails, check the Redirect URIs of the OAuth App and the team `maceosoft/devops` in [`dex.config`](https://github.com/maceosoft/gitops/blob/main/argocd/config/dex.config).

With SSO established, put `auth.password.disabled: true` in `platform.yaml`.

### 4. DNS

`vcluster.homelab.local` and `argocd.homelab.local` are resolved by the LAN DNS (BIND in `deborah`) to the fixed IP of the Gateway, `192.168.23.200` ([Phase 4 — 4.8](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#48-homelablocal-dns)). Use `192.168.20.5` as DNS; if not possible, the backup is `/etc/hosts`:

```
192.168.23.200  argocd.homelab.local vcluster.homelab.local
192.168.20.6    incus.homelab.local
```

## Daily use

1. Open [`https://vcluster.homelab.local`](https://vcluster.homelab.local)
1. **Login with SSO** (GitHub) or local admin until password is disabled
1. **New Virtual Cluster** → name → **Create**
1. In the vcluster: **Connect** / **Download kubeconfig**
1. `export KUBECONFIG=~/Downloads/kubeconfig.yaml` and `kubectl get nodes`

### Kubeconfig of the management K3s (connected cluster)

The host cluster where Platform runs appears in **Clusters** as connected cluster. Same SSO flow → **Connect** / **Download kubeconfig**. Without **Cluster Access** explicitly granted, the authenticated user **does not see the cluster** (default deny). Grant permissions to the user/team `loft-*` in the Platform UI.

Alternative without web: [`fase-4-gitops.md` 4.1](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/#41-kubectl-access).

CLI alternative:

```
vcluster connect <nombre> -n <namespace>
```

## vCluster vs CAPN

|              | vCluster Platform           | CAPN                  |
| ------------ | --------------------------- | --------------------- |
| Auth         | Dex Argo CD + UI kubeconfig | Secret Incus sealed   |
| Runtime      | Pods in management K3s      | Incus instances       |
| RAM          | Moderate (1 vcluster)       | High per kubeadm node |
| Argo CD Sync | manual                      | manual                |

vCluster vs CAPN

vCluster Platform

Pods in K3s; web kubeconfig; SSO Dex

Overhead per vcluster; Platform permissions

CAPN

Real kubeadm clusters in Incus

High RAM per node; secret Incus sealed

Web kubeconfig (Platform)

No `scp`; centralized audit

Requires explicit Cluster Access

`scp` / CLI

Works before Phase 5

Local file; no SSO in the flow

## Dependencies

Must be **Healthy** before syncing Platform:

- `openebs`, `homelab-storage` (PVCs)
- `metallb`, `metallb-config`, `cert-manager-config` and the Gateway controller (`kong`) (HTTPS URLs)
- `argocd-route`, `argocd` (Dex + staticClient)

# CAPN (Cluster API Provider Incus)

Part of the implementation guide

**[Phase 6 — CAPN](https://homelab.maceo.cl/en/implementacion/fase-6-capn/index.md)** — runnable walkthrough.

The existing K3s (lightweight Kubernetes distribution) acts as the *management cluster* for [Cluster API](https://cluster-api.sigs.k8s.io/). Provider: [CAPN](https://capn.linuxcontainers.org/).

## Installation

```
curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.10.10/clusterctl-linux-amd64 -o clusterctl
chmod +x clusterctl && sudo mv clusterctl /usr/local/bin/

mkdir -p ~/.cluster-api
curl -o ~/.cluster-api/clusterctl.yaml https://capn.linuxcontainers.org/static/v0.1/clusterctl.yaml
clusterctl init -i incus
```

## Incus credentials as a Secret

See [`gitops/sealed-secrets/`](https://github.com/maceosoft/gitops/tree/main/sealed-secrets) — the `lxc-secret` secret is encrypted with Sealed Secrets before committing, never in plain text.

## Generate a workload cluster

```
export LXC_SECRET_NAME=lxc-secret
export LOAD_BALANCER='lxc: {}'
export DEPLOY_KUBE_FLANNEL=true
export LXC_IMAGE_NAME=kubeadm/v1.36.1
export CONTROL_PLANE_MACHINE_TARGET="@x86-nodes"
export CONTROL_PLANE_MACHINE_TYPE=container
export WORKER_MACHINE_TARGET="@arm64-nodes"
# arm64: solo container, sin imágenes VM oficiales
export WORKER_MACHINE_TYPE=container

clusterctl generate cluster demo -i incus \
  --kubernetes-version v1.36.2 \
  --control-plane-machine-count 1 \
  --worker-machine-count 2 > gitops/capn/clusters/demo.yaml

kubectl apply -f gitops/capn/clusters/demo.yaml
```

> Note: CAPN provisions vanilla **kubeadm** clusters (not nested K3s) — heavier per node. Good for ephemeral dev/test clusters, not for 24/7 on oliver.

The generated manifest is synced by Argo CD with manual sync — see [`capn-demo.yaml`](https://github.com/maceosoft/gitops/blob/main/argocd/apps/capn-demo.yaml).

# Ansible — role catalog

Inventory of roles in [`ansible/roles/`](https://github.com/maceosoft/homelab/tree/main/ansible/roles). Each role has its `README.md` with variables, tags, and verification.

From zero to operational node?

This page is the catalog by role/phase. The **complete order** to take a node from bare-metal to operational (includes hardening, BIND, WiFi — currently outside phases 1-4 below) is in [`ansible/bootstrap/README.md`](https://github.com/maceosoft/homelab/blob/main/ansible/bootstrap/README.md).

## By implementation phase

| Phase                                                                           | Role(s)                                      | Playbook                                                |
| ------------------------------------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------- |
| [1 — Network](https://homelab.maceo.cl/en/implementacion/fase-1-red/index.md)   | `netplan_bridge`                             | `playbook-set-static-ip.yml`                            |
| [2 — Incus](https://homelab.maceo.cl/en/implementacion/fase-2-incus/index.md)   | `incus_install`, `incus_cluster`, `incus_ui` | `playbook-bootstrap.yml` + `playbook-incus-cluster.yml` |
| [3 — K3s](https://homelab.maceo.cl/en/implementacion/fase-3-k3s/index.md)       | `k3s_*` (10 roles)                           | `playbook-k3s.yml`                                      |
| [4 — GitOps](https://homelab.maceo.cl/en/implementacion/fase-4-gitops/index.md) | — (inline playbook)                          | `playbook-dex-oauth-secrets.yml`                        |

## Roles

| Role                                                                                                                  | Purpose                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | Playbook                         | Tags                                       |
| --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ------------------------------------------ |
| [`netplan_bridge`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/netplan_bridge/README.md)             | Static IP on `br0` (Netplan) and static `/etc/resolv.conf`                                                                                                                                                                                                                                                                                                                                                                                                                                                                    | `playbook-set-static-ip.yml`     | —                                          |
| [`incus_install`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_install/README.md)               | Incus package (Zabbly)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        | `playbook-bootstrap.yml`         | —                                          |
| [`incus_cluster`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_cluster/README.md)               | Incus bootstrap/join/groups                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `playbook-incus-cluster.yml`     | `bootstrap`, `join`, `groups`, `scheduler` |
| [`incus_ui`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_ui/README.md)                         | Incus native UI                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `playbook-incus-cluster.yml`     | `ui`                                       |
| [`incus_ovn`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/incus_ovn/README.md)                       | OVN for Incus (optional, off by default)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      | `playbook-incus-cluster.yml`     | `ovn`                                      |
| [`bind_dns`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/bind_dns/README.md)                         | BIND on deborah: LAN DNS and `mco.local` zones, its reverse and `homelab.local`                                                                                                                                                                                                                                                                                                                                                                                                                                               | `playbook-bind-dns.yml`          | —                                          |
| — (no role)                                                                                                           | Registers in Incus the metrics certificate read by vmagent (issued by cert-manager; read from Secret `monitoring/incus-metrics`)                                                                                                                                                                                                                                                                                                                                                                                              | `playbook-incus-metrics.yml`     | —                                          |
| [`node_apt_metrics`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/node_apt_metrics/README.md)         | Exposes pending apt patches and whether the node needs a reboot to Prometheus (only alerts, installs nothing)                                                                                                                                                                                                                                                                                                                                                                                                                 | `playbook-apt-metrics.yml`       | —                                          |
| [`node_units_metrics`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/node_units_metrics/README.md)     | Exposes host service memory and CPU (Ollama, k3s, incus, BIND) and top processes outside Kubernetes to Prometheus, to know what consumes when a node runs out of memory (only measures, changes nothing)                                                                                                                                                                                                                                                                                                                      | `playbook-units-metrics.yml`     | —                                          |
| [`k3s_prereqs`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_prereqs/README.md)                   | Swap, kernel modules, sysctl, Longhorn requirements (open-iscsi, NFS)                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `playbook-k3s.yml`               | `prereqs`, `always`                        |
| [`k3s_server`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_server/README.md)                     | K3s control-plane                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             | `playbook-k3s.yml`               | `server`, `core`                           |
| [`k3s_cni`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_cni/README.md)                           | CNI Canal/Calico/Cilium                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       | `playbook-k3s.yml`               | `cni`, `core`                              |
| [`k3s_agent`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_agent/README.md)                       | K3s workers                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `playbook-k3s.yml`               | `agent`, `core`                            |
| [`k3s_kube_vip`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_kube_vip/README.md)                 | API VIP `:6443`                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               | `playbook-k3s.yml`               | `kube_vip`                                 |
| [`k3s_cleanup_traefik`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_cleanup_traefik/README.md)   | Cleans Traefik/svclb                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          | `playbook-k3s.yml`               | `traefik`                                  |
| [`k3s_node_labels`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_node_labels/README.md)           | Node labels                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | `playbook-k3s.yml`               | `labels`                                   |
| [`k3s_argocd`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_argocd/README.md)                     | Argo CD controller                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            | `playbook-k3s.yml`               | `argocd`, `gitops`                         |
| [`k3s_fetch_kubeconfig`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_fetch_kubeconfig/README.md) | Local kubeconfig                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              | `playbook-k3s.yml`               | `kubeconfig`                               |
| [`k3s_os_hardening`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/k3s_os_hardening/README.md)         | unattended-upgrades                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `playbook-k3s.yml`               | `hardening`                                |
| [`host_hardening`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/host_hardening/README.md)             | SSH, default accounts, sysctl, auditd                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         | `playbook-hardening.yml`         | —                                          |
| [`opkssh`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/opkssh/README.md)                             | SSH to hosts with Dex identity (team `devops`)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                | `playbook-opkssh.yml`            | —                                          |
| [`ollama_ondemand`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/ollama_ondemand/README.md)           | Multi-node Ollama (group `llm_nodes`: `deborah`, `invincible`, `oliver`) that only starts when a request arrives: socket + systemd proxy, downloads per architecture, `MemoryMax` per node, CPU cap (`CPUQuota`), and a guard that stops and alerts if it saturates the node for 2 hours, hardened systemd units, model downloads through the socket, models and priority by `host_vars`, and the agent that registers the node with the router ([guide](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md)) | `playbook-ollama.yml`            | `models`, `cpu_guard`                      |
| [`host_firewall`](https://github.com/maceosoft/homelab/tree/main/ansible/roles/host_firewall)                         | Host firewall with `nftables` (own table), applied with auto-rollback at 300 seconds if not confirmed; includes VXLAN, Geneve, etcd, and calico-node metrics ([README](https://github.com/maceosoft/homelab/blob/main/ansible/roles/host_firewall/README.md), [guide](https://homelab.maceo.cl/en/security/hardening-post-incidente/index.md))                                                                                                                                                                                | `playbook-firewall.yml`          | —                                          |
| [`ollama_node`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/ollama_node/README.md)                   | Ollama registration agent as systemd service (`ollama-node-register`), running as user `ollama-node` (supplementary group `ollama`) and announcing node models to the router ([guide](https://homelab.maceo.cl/en/operacion/agregar-nodo-ollama/index.md))                                                                                                                                                                                                                                                                    | `playbook-ollama.yml`            | `register`                                 |
| [`wifi_failover`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/wifi_failover/README.md)               | Cable backup WiFi (nodes with WiFi)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           | `playbook-wifi-failover.yml`     | —                                          |
| [`storage_provision`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/storage_provision/README.md)       | Formats and mounts a free disk (destructive)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  | `playbook-storage-provision.yml` | —                                          |

## Role-less playbooks (inline tasks)

| Playbook                                                                                                                          | Phase    | Description                                                                                                                                |
| --------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| [`setup_sudo.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/bootstrap/setup_sudo.yml)                               | Day 0    | Installs python3 and sudo and gives your admin user sudo (connect as root; user can be specified with `-e bootstrap_admin_user=<usuario>`) |
| [`playbook-discovery.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-discovery.yml)                         | 0        | Hardware inventory → `ansible/reports/`                                                                                                    |
| [`playbook-incus-cluster.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-incus-cluster.yml)                 | 2        | Incus bootstrap/join/groups/UI                                                                                                             |
| [`playbook-dex-oauth-secrets.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-dex-oauth-secrets.yml)         | 4        | Dex OAuth: 1Password + Argo CD + Incus/vCluster                                                                                            |
| [`playbook-rotate-root-passwords.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-rotate-root-passwords.yml) | Security | Rotates root password → 1Password (HomeLab vault)                                                                                          |
| [`playbook-k3s-uninstall.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/playbook-k3s-uninstall.yml)                 | Appendix | Uninstalls K3s on the 3 nodes                                                                                                              |

OAuth documentation: [`gitops/argocd/secrets/1password.md`](https://github.com/maceosoft/gitops/blob/main/argocd/secrets/1password.md)

## K3s execution order

```
flowchart LR
  prereqs[k3s_prereqs] --> server[k3s_server]
  server --> cni[k3s_cni_opcional]
  cni --> kube_vip[k3s_kube_vip_opcional]
  server --> kube_vip
  kube_vip --> agent[k3s_agent]
  agent --> traefik[k3s_cleanup_traefik]
  traefik --> labels[k3s_node_labels_opcional]
  labels --> argocd[k3s_argocd_opcional]
  argocd --> kubeconfig[k3s_fetch_kubeconfig_opcional]
  kubeconfig --> hardening[k3s_os_hardening_opcional]
```

## Global configuration

| File                                                                                                                                          | Contents                      |
| --------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- |
| [`group_vars/incus_cluster/k3s_install.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/k3s_install.yml) | K3s playbook flags            |
| [`group_vars/dex_oauth_secrets.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/dex_oauth_secrets.yml)                 | Dex OAuth                     |
| [`group_vars/incus_cluster/vars.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/incus_cluster/vars.yml)               | Incus storage, groups, UI     |
| [`group_vars/all.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/group_vars/all.yml)                                             | Gateway, control plane IP     |
| [`host_vars/*.yml`](https://github.com/maceosoft/homelab/tree/main/ansible/host_vars)                                                         | Labels, kubelet args per node |

## References

- [K3s playbook options](https://homelab.maceo.cl/en/k3s/playbook-options/index.md)
- [Implementation guide](https://homelab.maceo.cl/en/implementacion/index.md)

# Post-incident hardening

What happened (2026-09)

`deborah` (Orange Pi 5 Plus, K3s control plane) was left exposed directly to the internet — DMZ (demilitarized zone)/port-forward on the router to its IP — still with the manufacturer image's default password. It was compromised and had to be reinstalled from scratch.

This page is the guide to rebuild `deborah` securely and to prevent this incident from recurring on `invincible`/`oliver`. The playbook and the role that accompany it are in [`ansible/roles/host_hardening/`](https://github.com/maceosoft/homelab/blob/main/ansible/roles/host_hardening/README.md).

## Root cause (and why order matters)

Two independent failures, either one alone was already enough:

1. **Direct internet exposure.** A port-forward/DMZ on the router sends internet traffic straight to a LAN (local area network) host, without any intermediate control. Any service with a vulnerability, or any administration port (SSH included), becomes reachable by anyone on the internet.
1. **Unchanged default password.** SBC —Single Board Computer— images (Orange Pi, Raspberry Pi, etc.) ship with publicly documented username/password in the manufacturer's repo — it's the first thing an automated scanner tries.

The combination is lethal: no zero-day vulnerability is needed, a cheap scanner sweeping internet IPs for SSH (Secure Shell) with default credentials is enough.

## The principle that replaces the DMZ: remote access only via the router's VPN

The WireGuard server is the router. This repo stores no WireGuard configuration or keys.

- **Never** forward an administration port (SSH, Incus UI, K3s API) on the router directly to a LAN host.
- The **only** traffic that should reach your network from the internet is the router's VPN (virtual private network) traffic. Everything else (SSH, Incus, K3s, BIND) is administered *through* the tunnel or from the LAN — never directly from the internet.
- If you ever need to genuinely publish something to the internet (a web service, not an administration service), the standard practice is a reverse proxy in a real, isolated DMZ — not the host running the service, and never the same host that manages the cluster.

Exposure is limited at the router **and on each node** (see the next section). `roles/host_hardening` does not configure the host firewall: ufw conflicts with kube-proxy's iptables rules and is not compatible with K8s (Kubernetes); that's handled by the `host_firewall` role, which uses `nftables`.

## Host firewall, exposed services, and authentication

An audit of the repo and the live cluster found that, in addition to the incident's DMZ, the nodes had the `INPUT` chain in `ACCEPT` and several services accessible from the entire LAN and WiFi without authentication. The outcome, and what was done for each finding, is summarized here; the general rule and the exception list live in [Authentication model](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/index.md). The "Fix" column describes the design; what gets rolled out in stages (NetworkPolicy `default-deny`, PSA —Pod Security Admission— labels, `oauth2-proxy`) is marked as "to enable".

| Finding                                                                                                    | Fix                                                                                                                                                                                                                                                                                                    |
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| No managed host firewall: LAN and WiFi reached `22`, `6443`, `8443`, `10250`, `9100`, `11434`...           | `host_firewall` role with `nftables`: own table, evaluated before kube-proxy and without touching Calico chains                                                                                                                                                                                        |
| Ollama without authentication, with `/api/pull`, `/api/create`, and `/api/delete`                          | Socket only on the node IP, `nftables` with port `11434` only from K3s nodes and pods, Kong only via HTTPS and Bearer auth on the router, which also blocks management verbs. The node's Ollama remains unauthenticated: the real protection is the router token plus NetworkPolicy                    |
| Kong Manager (ports `8002` and `8445`, NodePort) and Admin API without authentication                      | Kong Manager disabled; Admin API internal only (its NetworkPolicy is to enable)                                                                                                                                                                                                                        |
| Argo CD with local `admin` user and its initial secret                                                     | `admin.enabled: "false"` and delete the initial secret; the pipeline uses a `ci` token with expiration                                                                                                                                                                                                 |
| No NetworkPolicy in `kong`, `monitoring`, `ai`, `longhorn-system`, `gateway`, and `arc-*`                  | `default-deny` manifests and explicit per-namespace permissions, **to enable** one by one (those for logs, `ai/tools`, `alert-triage`, and Open Terminal already ship with their application)                                                                                                          |
| Open WebUI with open registration and no Dex                                                               | Dex with the `devops` team, registration closed                                                                                                                                                                                                                                                        |
| Incus (`8443`) open to LAN                                                                                 | Restricted by `nftables`; specific OIDC group                                                                                                                                                                                                                                                          |
| `adbd` (Android Debug Bridge, no credentials) listening on `deborah:5555`, launched by `usbdevice.service` | Disable and mask that unit and verify the port closes                                                                                                                                                                                                                                                  |
| node-exporter and MetalLB and FRR metrics open to LAN                                                      | Listen only on the node IP and `nftables` toward the collector                                                                                                                                                                                                                                         |
| `alert-triage` and `mcpo` without token                                                                    | Bearer token or API key, NetworkPolicy                                                                                                                                                                                                                                                                 |
| BIND without zone transfers and with visible version                                                       | `allow-transfer { none; }` and version hidden. DNS serves the whole network (other router subnets, VPN): not limited by source nor rate limit enabled (opt-in with `bind_dns_restrict_clients` and `bind_dns_rate_limit_enabled`)                                                                      |
| K3s without Dex or Secrets encryption                                                                      | Flags `oidc-*` toward Dex, `--secrets-encryption`, and installer with checksum, configured in `k3s_install` (`group_vars/incus_cluster/k3s_install.yml`); the server picks them up on reinstall or when [moving the control plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md) |
| `rpcbind`, `dnsmasq`, SSH with `AllowUsers` undefined, minimum `auditd`                                    | Disable what is not used, `AllowUsers devops amaceo`, and `auditd` rules sent to [VictoriaLogs](https://homelab.maceo.cl/en/operacion/logs/index.md)                                                                                                                                                   |

Firewall applies with auto-rollback

A `nftables` mistake can lock you out of SSH. The role applies rules with a **300-second (5-minute) auto-rollback**: new connections (SSH, K3s API) are tested and must be confirmed manually; if not, a systemd timer removes the rules. It is applied node by node (`playbook-firewall.yml`): first `--check` on `oliver`, then `invincible`, and finally `deborah`. To remove it in an emergency on the node: `sudo systemctl disable --now homelab-firewall`. The role details are in its [README](https://github.com/maceosoft/homelab/blob/main/ansible/roles/host_firewall/README.md).

SSH access with the Dex identity (opkssh) only works if the policy deployed on each node matches the current team organization (`maceosoft:devops`); if the organization changes, `playbook-opkssh.yml` must be reapplied.

The shared `devops` account (with passwordless `sudo`) is kept as an SSO (single sign-on) account by explicit decision; control is `AllowUsers` and the audit trail is `auditd`.

## What the hardening does (summary)

Full detail in the [role README](https://github.com/maceosoft/homelab/blob/main/ansible/roles/host_hardening/README.md). In short: SSH without password (key only), removal of known default accounts (`orangepi`, `pi`, `ubuntu`, `debian`) with their home directories, basic network sysctl, and auditd to have a trail if something happens again. On `deborah`, **SSH host keys are also regenerated** — factory SBC images sometimes repeat both the password and the host keys across all units of the same model.

```
cd ansible
ansible-playbook -i inventory.ini playbook-hardening.yml --check --diff
ansible-playbook -i inventory.ini playbook-hardening.yml --limit deborah
```

## `deborah` rebuild order

`deborah` is the K3s control plane (`k3s_api_ip` in `group_vars/all.yml`) — it must be rebuilt in order before it becomes useful again. The complete order, step by step with the exact commands, lives in a single place to avoid duplication (and prevent it from going out of sync in two different places):

**→ [`ansible/bootstrap/README.md`](https://github.com/maceosoft/homelab/blob/main/ansible/bootstrap/README.md)**

Use `--limit deborah` at each step. The static IP (`playbook-set-static-ip.yml`) goes **before** hardening.

Specific notes:

- **Manual, before any Ansible step:** when flashing the image, change the default password and load your SSH key *before* connecting the machine to any network with internet access. `playbook-hardening.yml` assumes a key already exists in `authorized_keys` (aborts if not found) — it does not replace this manual step.
- **Hardening goes before re-adding `deborah` to the cluster** (after the static IP): there is no point in re-exposing an unhardened host while rebuilding.
- **The CA (certificate authority) and K3s certificates (lightweight Kubernetes distribution) regenerate automatically** when doing a clean `k3s server` on the reinstalled `deborah` (no etcd backup/snapshot to restore) — this is, in effect, a full rotation of the cluster's root of trust. If you happen to have an etcd snapshot from before the incident, **do not restore it** — it would bring back potentially compromised certificates.

## Rotate the root password

In addition to SSH hardening, it is advisable to rotate the local `root` password on all 3 nodes (defense in depth for console/`su -` — `root` login via SSH is already disabled). There is a dedicated playbook that generates a new password per host and stores it in 1Password (vault `HomeLab`): see [Secrets management](https://homelab.maceo.cl/en/secrets/#root-password-rotation-local-1password-sdk).

```
ansible-playbook -i inventory.ini playbook-rotate-root-passwords.yml \
  --limit deborah
```

## On the router

- Delete any DMZ/port-forward rule pointing to `192.168.20.5` (or any other node).
- Leave **only** the router's WireGuard server exposed.
- If the router supports it, this is also the time to evaluate VLANs (virtual local area networks) (see [Next hardening level](#next-hardening-level)) instead of a single flat LAN.

These changes are made **only from the router's web interface**; menus depend on the model.

## Next hardening level

- **Segmentation with VLANs** — separate the HomeLab cluster VLAN (virtual local area network) from anything else on the LAN (IoT, guests, etc.), instead of trusting a single flat LAN. Requires network hardware/config not yet declared in this repo.
- **CIS (Center for Internet Security) Kubernetes / `kube-bench` Benchmark** on K3s — K3s already passes several CIS controls by default; for those it doesn't, you must opt in explicitly (`protect-kernel-defaults`, `secrets-encryption`, `NetworkPolicy`, Pod Security Admission). See [official K3s guide](https://docs.k3s.io/security/hardening-guide) and [`kube-bench`](https://github.com/aquasecurity/kube-bench).
- **Encryption of secrets at rest** in K3s (`--secrets-encryption`): already in `k3s_install`; the server picks it up on reinstall (or when [moving the control plane](https://homelab.maceo.cl/en/operacion/mover-control-plane/index.md), if ever done).
- **NetworkPolicies** `default-deny` between namespaces (`ai`, `monitoring`, `longhorn-system`, `kong`, `gateway`, `arc-*`): manifests are ready but enabled in stages, namespace by namespace. See [What is active](https://homelab.maceo.cl/en/security/modelo-de-autenticacion/#whats-active-and-whats-pending-activation).
- **Pod Security Admission** (PSA) in `warn` and `audit` mode with the `restricted` level before making it mandatory (labels ready, pending activation).

## References consulted

- [Linux Server Hardening Checklist for 2026 — SysWard](https://sysward.com/blog/2026-05-06-linux-server-hardening-checklist/)
- [SSH Hardening Checklist for 2026 (Ubuntu/Debian) — Let's Secure Me](https://letsecure.me/ssh-hardening-checklist-2026-ubuntu-debian/)
- [CIS Hardening Guide — K3s docs](https://docs.k3s.io/security/hardening-guide)
- [CIS Self Assessment Guide — K3s docs](https://docs.k3s.io/security/self-assessment)
- [Kubernetes hardening with kube-bench — CNCF blog](https://www.cncf.io/blog/2025/04/08/kubernetes-hardening-made-easy-running-cis-benchmarks-with-kube-bench/)
- [How to Secure Remote Access to a Homelab — HomelabAddiction](https://homelabaddiction.com/homelab-security/)

# Authentication model

HomeLab general rule: **nothing is left exposed without authentication, or without an explicit and documented network restriction when authentication is not possible.**

- **People → Dex.** Anyone opening something in the browser or via SSH (*Secure Shell*, the remote terminal connection) signs in with their GitHub account through Dex (the HomeLab's identity provider, with the `devops` team of the `maceosoft` organization). If the service speaks OIDC (OpenID Connect, the login protocol over OAuth 2.0 —Open Authorization—) natively, it connects directly; otherwise, `oauth2-proxy` is placed in front.
- **Machines → own token or key + network restriction.** A program calling another uses a token (a random secret, one per client) and, additionally, can only reach it from where it belongs.
- **What cannot authenticate** (inter-node protocols, exporters) is limited by network and listed as an [exception](#network-controlled-exceptions).

**This decision in other environments**

People sign in with SSO through Dex using their GitHub account (team `devops`), and machines with their own token plus a network restriction. It's free, no local passwords to manage, and works for a team of one or two people.

- **Benefits:** a single sign-on for everything, and when someone is offboarded in one place they lose access to all services at once; with one token per machine, you revoke a client without touching the rest.
- **Corporate IdP instead of GitHub:** identity comes from [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/fundamentals/what-is-entra), [Okta](https://www.okta.com/products/workforce-identity/), or [Google Workspace](https://workspace.google.com/), connected to HR for onboarding and offboarding. Dex can still sit in the middle as an OIDC bridge.
- **Mandatory MFA** and conditional access policies (managed device, location).
- **Group-based RBAC:** permissions are granted to IdP groups, not to individuals, and are reviewed in periodic audits.
- **Machines:** instead of static tokens, short-lived identities issued by the cluster itself are preferred, with no secret to store.
- **When the HomeLab approach makes sense:** on a small team or a community already living in GitHub, Dex + GitHub provides SSO without paying for an IdP.

| Cloud              | Equivalent service                                                                               |
| ------------------ | ------------------------------------------------------------------------------------------------ |
| GCP                | [Workforce Identity Federation](https://cloud.google.com/iam/docs/workforce-identity-federation) |
| AWS                | [AWS IAM Identity Center](https://aws.amazon.com/iam/identity-center/)                           |
| Azure              | [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/fundamentals/what-is-entra)         |
| Oracle Cloud (OCI) | [OCI IAM Identity Domains](https://docs.oracle.com/en-us/iaas/Content/Identity/home.htm)         |

For machines, each cloud gives pods an identity with no stored secrets: [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity), [EKS Pod Identity](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html) or [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html), [Microsoft Entra Workload ID on AKS](https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview), and [OKE workload identity](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contenggrantingworkloadaccesstoresources.htm). You gain MFA, auditing, and revocation integrated with cloud permissions. You lose portability: those identities only work within that provider.

## Who authenticates whom

| Service                                               | Who uses it                                                      | How it authenticates                                                                                                                                                                                                                                                                                            |
| ----------------------------------------------------- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Argo CD, Grafana                                      | People                                                           | Dex (GitHub, team `devops`). An emergency local access remains (`admin` of Grafana); the local `admin` of Argo CD is disabled (`admin.enabled: "false"`) and the pipeline uses its own token (`ci`) with expiration                                                                                             |
| Open WebUI                                            | People                                                           | Dex (SSO, *Single Sign-On*: single sign-on, with GitHub): public client with PKCE (*Proof Key for Code Exchange*, a protection for clients without secret), no shared secret. SSO only: no local form nor password registration, not even for the first administrator; only the team `devops` enters            |
| vCluster, Incus                                       | People                                                           | Dex (OIDC). Incus also limits port `8443` by firewall                                                                                                                                                                                                                                                           |
| SSH to nodes                                          | People                                                           | opkssh with Dex identity (group `maceosoft:devops`); no password nor root                                                                                                                                                                                                                                       |
| K3s API (`6443`)                                      | People and machines                                              | K3s certificates for machines; Dex (flags `oidc-*`) for people                                                                                                                                                                                                                                                  |
| Ollama Router (`/api/*`, `/v1/*`)                     | Machines: Open WebUI, K8sGPT, translation, reports, alert-triage | Token `Authorization: Bearer`, **one per client** (item `ollama-router`) + ingress NetworkPolicy (when active; see [What's active](#que-esta-activo-y-que-esta-por-activar)); the router blocks model management verbs (`pull`, `create`, `delete`, `copy`, `push`) and `blobs`                                 |
| Ollama Router (`/_registry`)                          | Machines: each Ollama node                                       | Bearer token different from clients' (field `registry_token` of item `ollama-router`)                                                                                                                                                                                                                           |
| Ollama Nodes (`11434`)                                | Only the router                                                  | **Ollama does not authenticate.** Listens on the node IP and the firewall allows port `11434` from K3s nodes **and from pods** (because the router may be on the same node). The real protection is the router token plus the NetworkPolicy of namespace `ai`. See [exceptions](#network-controlled-exceptions) |
| `alert-triage`                                        | Alertmanager                                                     | Bearer token in header (`http_config.authorization`) + NetworkPolicy                                                                                                                                                                                                                                            |
| Open Terminal (`ai-sandbox`)                          | Machine: Open WebUI                                              | Bearer API key mounted as file (item `open-terminal`, field `api_key`) + NetworkPolicy that only allows Open WebUI; use restricted to administrators and group `maceosoft:devops`                                                                                                                               |
| `mcpo` and MCP tools                                  | Open WebUI                                                       | API key (`--api-key`, item `mcp-api-key`) + NetworkPolicy; tools only for administrators                                                                                                                                                                                                                        |
| Kong (gateway)                                        | All who enter via `*.homelab.local`                              | Terminates TLS. **Kong Manager** disabled and **Admin API** internal only (ClusterIP; its NetworkPolicy is pending activation)                                                                                                                                                                                  |
| Alertmanager, vmui/vmalert, Longhorn UI, VictoriaLogs | People, if needed                                                | No own login: they are internal (ClusterIP, no route). To view them in the browser, `oauth2-proxy` + Dex (optional, pending activation)                                                                                                                                                                         |
| DNS (BIND)                                            | Entire network (LAN, other subnets, VPN)                         | Does not authenticate nor limit by origin; no zone transfers, hidden version and the port is not forwarded to the internet                                                                                                                                                                                      |

## What's active and what's pending activation

Some controls are deployed in stages, to avoid cutting the cluster all at once. In the repo [`gitops`](https://github.com/maceosoft/gitops/blob/main/bootstrap/root-appset.yaml) they are **commented out** in `root-appset.yaml` until they are activated one by one, after reading their README:

| Control                                                                                                                                                                               | Status                                                                                                                                                                                                                                                                                                      |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| NetworkPolicy `default-deny` per namespace (`ai`, `monitoring`, `longhorn-system`, `kong`, `gateway`, `arc-runners`)                                                                  | **Pending activation**, one by one and in this order: `netpol-ai`, `netpol-monitoring`, `netpol-longhorn-system`, `netpol-kong` and `netpol-gateway`, `netpol-arc-runners` (read the README of [`security/networkpolicies/`](https://github.com/maceosoft/gitops/tree/main/security/networkpolicies) first) |
| Pod Security Admission labels (`warn`/`audit` with `restricted`)                                                                                                                      | **Pending activation** (application `namespace-labels`)                                                                                                                                                                                                                                                     |
| `oauth2-proxy` in front of internal interfaces (e.g. Alertmanager)                                                                                                                    | **Pending activation** and optional                                                                                                                                                                                                                                                                         |
| NetworkPolicies that travel with their own application: logs (VictoriaLogs and Vector), `ai/tools`, `alert-triage` and Open Terminal (the one of `ai-sandbox` goes alongside the pod) | **Active** when deploying that application                                                                                                                                                                                                                                                                  |
| People login to K3s API: ClusterRoleBinding `oidc-devops-view` (Application `security-rbac`)                                                                                          | **Active**, read-only (`view`); harmless until OIDC exists                                                                                                                                                                                                                                                  |
| K8sGPT                                                                                                                                                                                | **Staged**: the decision is up to the user (see below)                                                                                                                                                                                                                                                      |
| Router Bearer tokens, Open Terminal key, Dex in Open WebUI, Kong Manager disabled, `admin` of Argo CD disabled                                                                        | **Active** upon deployment                                                                                                                                                                                                                                                                                  |

While a NetworkPolicy is not active, the network control for that service is only the node firewall and the token or key of the service itself. The tables below describe the complete design; the "Control" column of the exceptions applies when the corresponding control is active.

## Rules for a new service

Before publishing anything, answer these questions. If any lacks an answer, it does not get published.

1. **Who uses it, a person or a machine?**
   - Person: does it speak OIDC? Then a Dex client (public with PKCE —Proof Key for Code Exchange— if possible, confidential with secret if not). Does it not speak OIDC? `oauth2-proxy` in front, with `--allowed-group=maceosoft:devops`.
   - Machine: its own random token (`openssl rand -hex 32`), stored in 1Password and applied with `playbook-platform-secrets.yml`. **One token per client**, so you can revoke one without touching the rest.
1. **Where is it reached from?** Internal services: ClusterIP and NetworkPolicy; no Kong route unless needed. Services on a node: listen only on the required IP and the node firewall (`nftables`) allows only who corresponds.
1. **No NodePort, LoadBalancer, or `hostPort`** unless essential and on the exception list below.
1. **Its NetworkPolicy** in the namespace: `default-deny` and explicit ingress and egress permissions.
1. **Secrets never go to Git** (`homelab` is a public repo and `gitops`, though private, doesn't store secrets either): 1Password and cluster Secrets.
1. **If it cannot carry authentication**, add it to the exception table, with the reason and the control that replaces it.

Why `oauth2-proxy` and not a Kong plugin

The Kong OIDC plugin is Enterprise edition, and Gateway API external authentication is not available in the KIC (Kong Ingress Controller) version used. `oauth2-proxy` is deployed as **backend of the HTTPRoute** for each service, with a confidential Dex client per service and PKCE (`S256`).

## Network-controlled exceptions

What **cannot or must not** carry its own authentication stays only with an explicit network restriction:

| Service                                                                                                                                                      | Why no login                                                                                                     | Replacing control                                                                                                                                                                                                                                                                                                                                                                           |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| BGP `179` (Calico/bird), FRR `2623`, `7946` (MetalLB memberlist), `5473` (Typha)                                                                             | Node-to-node protocols, no user authentication (BGP without MD5)                                                 | `nftables`: only node IPs and pod range; `rpcbind` (`111`) is disabled                                                                                                                                                                                                                                                                                                                      |
| `9100` (node-exporter), `9120`, `9140` and `9141` (MetalLB and FRR metrics)                                                                                  | Exporters without authentication in this deployment                                                              | `nftables` only to the metrics collector and the nodes; node-exporter with TLS and basic auth if the chart allows                                                                                                                                                                                                                                                                           |
| `30980` (`healthCheckNodePort` of `kong-gateway-proxy`)                                                                                                      | Standard Kubernetes mechanism with `externalTrafficPolicy: Local`: only tells how many local endpoints there are | `nftables` only from the nodes                                                                                                                                                                                                                                                                                                                                                              |
| `19443` (MetalLB webhook, `statuscleaner`)                                                                                                                   | Admission webhook: called by the API server                                                                      | `nftables` only from the nodes and the API server                                                                                                                                                                                                                                                                                                                                           |
| DNS `53` (BIND)                                                                                                                                              | DNS does not authenticate clients and serves the whole network (LAN, other subnets, VPN)                         | No zone transfers and hidden version; the port is filtered by the router (not forwarded to internet)                                                                                                                                                                                                                                                                                        |
| Ollama `11434` on each node (and on the Mac)                                                                                                                 | Ollama API has no authentication                                                                                 | Node firewall: `11434` from K3s nodes and from pods (`host_firewall_ollama_allow_pods`), and from machines added to `host_firewall_ollama_extra_cidrs` (the Mac); on the Mac, `allowed_cidrs` of the proxy. **The node does not authenticate**: what protects inference is the router's Bearer token and the `ai` namespace NetworkPolicy. Clients don't talk to a node, only to the router |
| Internal services without login (Alertmanager, vmsingle/vmui, vmalert, Longhorn UI, VictoriaLogs, Kong Admin API, `/metrics` of the router, kubelet `10250`) | Internal ClusterIPs without native authentication (kubelet does authenticate)                                    | NetworkPolicy `default-deny` with explicit permissions; for the browser, `oauth2-proxy` + Dex                                                                                                                                                                                                                                                                                               |
| Open Terminal egress (`ai-sandbox`)                                                                                                                          | Runs arbitrary code and needs to download packages                                                               | Egress NetworkPolicy only to internet on 80/443, no access to cluster or internal network; no `docker.sock`                                                                                                                                                                                                                                                                                 |
| Calico VXLAN (UDP `4789`), OVN Geneve (UDP `6081`), etcd (`2379`, `2380`), calico-node metrics (`9091`), kube-proxy healthz (`10256`)                        | Internal cluster traffic, no user authentication                                                                 | `nftables`: only nodes (and pods where needed); Geneve only between nodes                                                                                                                                                                                                                                                                                                                   |
| SSH `22`, K3s API `6443`, kubelet `10250`, Incus `8443`                                                                                                      | Have their own authentication (keys and Dex, certificates, kubelet authn/authz, OIDC), but listen on the network | `nftables`: LAN and operator workstation (SSH), nodes, pods and operator (the rest)                                                                                                                                                                                                                                                                                                         |
| `kong-gateway-proxy` service (LoadBalancer `192.168.23.200`)                                                                                                 | It is the HomeLab entry point; each route carries its own authentication                                         | HTTP listeners only redirect to HTTPS                                                                                                                                                                                                                                                                                                                                                       |
| Dex (`/api/dex`) and Kong HTTP listeners                                                                                                                     | Dex is the login mechanism itself; HTTP only redirects to HTTPS                                                  | By design                                                                                                                                                                                                                                                                                                                                                                                   |

The source of truth for this list is [`ansible/files/exposure-allowlist.yml`](https://github.com/maceosoft/homelab/blob/main/ansible/files/exposure-allowlist.yml), which the audit script reads; if this table and the file differ, the file wins.

Audit scope

The audit covers Kubernetes services, Gateway routes, and node ports. It does not cover the router's WireGuard, services inside Incus instances (`lb01`, `vm01`, `test`) nor the `deborah` Docker socket (it is a Unix socket, not network).

## Re-run the audit

[`ansible/scripts/audit_exposure.sh`](https://github.com/maceosoft/homelab/blob/main/ansible/scripts/audit_exposure.sh) is a **read-only** script: it lists NodePort and LoadBalancer Services, HTTPRoutes (and whether their backend is an auth proxy or an app with OIDC), and the NetworkPolicies of each namespace, and compares them against the [exception list](https://github.com/maceosoft/homelab/blob/main/ansible/files/exposure-allowlist.yml). It exits with code `1` if it finds anything **outside that list** (and `2` if a tool is missing; for example, if the system Python lacks PyYAML, use one that has it). The list includes temporary entries, with a review date, for namespaces that still lack a NetworkPolicy, so it serves to detect regressions:

```
export KUBECONFIG=~/.kube/homelab-k3s.yaml
ansible/scripts/audit_exposure.sh
# 0 = sin sorpresas
echo $?

# Además, los puertos que escucha cada nodo (por SSH, solo lectura):
AUDIT_NODES="deborah invincible oliver" ansible/scripts/audit_exposure.sh
```

When you add a legitimate exception, update this table **and** the script's list in the same change. The firewall that closes the node ports is described in [Post-incident hardening](https://homelab.maceo.cl/en/security/hardening-post-incidente/index.md).

## Optional checks

- **Calico and global policies.** By default, a service account without permissions should not be able to create a `GlobalNetworkPolicy`. This is verified with:

  ```
  kubectl --as=system:serviceaccount:default:default create --dry-run=server -f globalnetworkpolicy.yaml
  ```

  (using any test `GlobalNetworkPolicy`). If it responds `created`, that account can change the pods' firewall: remove the `calico-tiered-policy-passthrough` binding that allows it. Mitigation steps are in [`security/calico-policy-edit/`](https://github.com/maceosoft/gitops/tree/main/security/calico-policy-edit).

  - **K8sGPT.** Its chart ships a `*/*/*` ClusterRole (effectively, cluster admin) and its dynamic RBAC reads Secrets across the entire cluster. That is why it is **staged** (commented out in `root-appset.yaml`). To enable it there are three paths, in [`security/k8sgpt-rbac/`](https://github.com/maceosoft/gitops/tree/main/security/k8sgpt-rbac): accept the risk and uncomment the line; use the proposed scoped role (untested); or also enable the NetworkPolicy of the namespace.

## Pending decisions

The following does not activate on its own: each point is decided and executed by a person.

1. **Enable the default-deny NetworkPolicy `default-deny`** in the order indicated above, verifying each namespace before the next. `netpol-longhorn-system` affects the `arc-cache` volume of the runners (NFS): test with a CI (continuous integration) workflow.
1. **K8sGPT**: accept the risk, scope its role, or leave it disabled.
1. **Calico mitigation**: verify with `--dry-run=server` and, if it responds `created`, remove the binding.
1. **Expose internal UIs with `oauth2-proxy`** (Alertmanager, vmui, Longhorn, VictoriaLogs): requires a confidential Dex client, its Secrets, and a DNS name.
1. **Protect the `main` branch** of the `gitops` repo (mandatory review).
1. **K8sGPT in the `-skip` of the reusable workflow `core-pipelines`** (private repo, manual change) while it remains staged.

See also [Local AI Assistant](https://homelab.maceo.cl/en/operacion/ia-ollama/index.md), [Secrets Management](https://homelab.maceo.cl/en/secrets/index.md) and [Rotate Credentials](https://homelab.maceo.cl/en/operacion/rotar-credenciales/index.md).

# Cinc/Chef lab

Isolated sandbox (Incus VMs, not LXC —Linux system containers— — Chef manages the full OS state: sysctl, mounts, kernel modules) to learn Chef/CINC (free distribution of Chef) without risking production. Tested on both architectures of the cluster.

```
incus launch images:ubuntu/24.04 cinc-lab-x86 --vm --target=@x86-nodes -c limits.cpu=2 -c limits.memory=2GiB
incus launch images:ubuntu/24.04 cinc-lab-arm --vm --target=@arm64-nodes -c limits.cpu=2 -c limits.memory=2GiB

incus snapshot create cinc-lab-x86 clean-cinc
# restaurar con: incus snapshot restore <vm> clean-cinc
incus snapshot create cinc-lab-arm clean-cinc
```

Cookbooks in [`cinc-lab/cookbooks/`](https://github.com/maceosoft/homelab/tree/main/cinc-lab/cookbooks).

Also evaluated: installing Cinc/Chef as configuration management for the whole HomeLab — rejected in favor of **Ansible** (agentless, YAML, gentler learning curve)

- **OpenTofu** (Incus provider) for 3 nodes. Cinc stays as a learning lab, not as a production component for now.