Phase 2 — Incus¶

Objective¶

Incus cluster of 3 nodes with quorum, cluster groups by architecture, and oliver in manual scheduler.

What you learn¶

Incus as a shared hypervisor: dqlite quorum across 3 members, cluster groups to steer x86 vs ARM64 (64-bit ARM architecture) workloads, and manual scheduler to protect RAM.

Learn this phase

In short: Incus is an apartment building. Each apartment is an instance: a container (a lightweight room sharing the building's utilities) or a virtual machine (a full apartment with its own kitchen). With three coordinated buildings, if one closes the other two keep running.

Key concepts

Container and virtual machine: the container is lighter; the VM (Virtual Machine) isolates more.
Cluster and quorum: with 3 members, the cluster decides by majority and tolerates the loss of one.
Cluster group: a label to direct workloads to a node type (for example, ARM64 only).
x86_64 and ARM64: two distinct processor architectures; they don't mix in the same instance.
Manual scheduler: the oliver node does not receive instances automatically; protects its memory.

Practical challenge (read-only): view cluster members and their instances.

incus cluster list
incus list -c ns4

Why a three-member cluster and not two?

With two, if one falls there is no majority and the cluster blocks. With three, the majority is two and it can keep running.

When do you prefer a container and when a VM?

Container for light, fast Linux workloads; VM when you need another kernel or more isolation.

What are cluster groups for?

To decide on which nodes an instance runs, for example keeping ARM64 workloads on deborah.

Questions? Use the Learn with AI button next to each heading, or the Learn page.

Stack for this phase¶

flowchart TB
  subgraph cluster [Incus_cluster_3_nodos]
    INV[invincible_x86]
    NOL[oliver_x86_quorum]
    DEB[deborah_arm64]
  end
  INV --> CGx86[x86_nodes]
  DEB --> CGarm[arm64_nodes]
  NOL --> CGx86

Incus — LXC virtualization (Linux system containers)/VM; Phase 2.
Zabbly Incus stable — Debian packages; Phase 2.
Ansible — Installs package, bootstrap, join, groups, UI (user interface).
Roles: incus_install, incus_cluster, incus_ui

Deep dive: Incus Cluster

This decision in other environments

🏠 At home🏢 In a company☁️ In the cloud

An Incus cluster with 3 members (dqlite quorum) on hardware you already had, free and with containers and VMs in the same tool. If a member falls, the other two keep deciding; oliver only votes and receives no workloads.

Benefits: a virtualization cluster pools several servers into a single resource: move VMs between nodes for maintenance without cutting service, restart workloads on another node if one fails, and manage everything from a single API or console, with permissions and audit trail of who did what.
Commercial products: the most common is VMware vSphere (now Broadcom, with licensing changes driving many companies to seek alternatives), Nutanix AHV or Proxmox VE with support subscription. For large private clouds, OpenStack.
Shared storage: instead of the local dir driver, shared or distributed storage (Ceph, SAN) is used, so a VM can move or start on another node without copying its disk.
Incus support: Incus also has commercial support (Zabbly, FuturFusion); the real cost is in the people who know how to operate it.
When Incus makes sense: escaping expensive VMware licenses, edge or branch sites with few servers, and labs or CI that need to create and destroy many instances quickly.

Cloud	Equivalent service
GCP	Compute Engine
AWS	Amazon EC2
Azure	Azure Virtual Machines
Oracle Cloud (OCI)	OCI Compute

You get VMs instantly, without buying or maintaining hardware, with SLA and in many regions. You lose control over the hypervisor, pay for every hour the VM is running, and are locked into that provider's network and images.

Before you start¶

Phase 1 — Network completed (static IPs in br0).
br0 operational on all 3 nodes.
≥ 20 GB free on each node (see storage requirements).

Storage requirements¶

Default HomeLab: driver dir (directory on the OS filesystem, no dedicated disk). Variables in group_vars/incus_cluster/vars.yml.

Aspect	Value / requirement
Driver	`dir` — no dedicated partition or loop
Pool per node	`local` (each member has its local pool)
Pool path	`/var/lib/incus/storage-pools/local`
Filesystem	ext4 or xfs; not NFS/CIFS as backend
Minimum free space	20 GB per node; 50 GB+ if you will create VMs or many instances
Metadata	dqlite and daemon config share disk with the pool

Per node¶

Node	Disk	Implication
`invincible`	Kingston 224 GB SSD	Leader + UI; largest pool of the x86 nodes
`oliver`	TECLAST 120 GB SSD	Quorum; local pool mandatory even if it receives no instances; avoid filling disk with CAPN images
`deborah`	eMMC 233 GB (NVMe 250 GB not detected today)	If eMMC tight: `incus_storage_path: /srv/incus/storage-pools/local` in host_vars

Limitations of `dir`¶

No pool-level copy-on-write (instance snapshots and export tarball do work).
VMs (virtual machines) slower than with zfs/btrfs/lvm.
No thin provisioning or native compression.
Each member maintains its own local pool; no distributed storage like Ceph.

Deep dive: Incus Cluster — Storage

Run¶

Step 1 — Install Incus package on all 3 nodes¶

Recommended (Ansible)Manual alternative

cd ansible
ansible-playbook -i inventory.ini playbook-bootstrap.yml

On each node:

curl https://pkgs.zabbly.com/get/incus-stable | sudo bash -x

Step 2 — Bootstrap, join, groups, scheduler, and UI¶

Recommended (Ansible)Manual alternative

A single playbook: bootstrap on invincible, join on oliver and deborah, cluster groups, manual scheduler, and UI.

cd ansible
ansible-playbook -i inventory.ini playbook-incus-cluster.yml

Optional tags: --tags bootstrap, --tags join, --tags groups, --tags ui.

On invincible (bootstrap):

sudo incus admin init
# clustering=yes, no te unes (bootstrap), storage backend dir

On invincible, generate tokens:

incus cluster add oliver
incus cluster add deborah

On oliver and deborah: incus admin init → join with token.

Cluster groups and scheduler:

incus cluster group create x86-nodes
incus cluster group create arm64-nodes
incus cluster group assign invincible x86-nodes,default
incus cluster group assign oliver x86-nodes,default
incus cluster group assign deborah arm64-nodes,default
incus cluster set oliver scheduler.instance manual

UI on invincible:

sudo apt install incus-ui-canonical
sudo incus config set core.https_address=:8443

Network profile (instances on LAN)¶

Mandatory manual step — neither the Ansible playbook nor incus admin init do this automatically, in neither of the two paths above. Without this, Incus instances remain on the default private NAT (Network Address Translation) network, not on your LAN (local network).

On invincible (profiles are shared across the whole cluster, no need to repeat on oliver/deborah):

incus profile device add default eth0 nic nictype=bridged parent=br0

Verify:

# tiene que mostrar eth0 → parent br0
incus profile show default

Admin UI¶

Ansible installs incus-ui-canonical and core.https_address on invincible (play ui of the playbook). Initial access via client certificate; SSO (single sign-on) (Dex) in Phase 4.

Add incus.homelab.local in /etc/hosts (full block in HomeLab Summary).
Open https://incus.homelab.local:8443
Accept the self-signed certificate
Generate/import a client certificate

OIDC with Dex

SSO login (GitHub via Dex) is configured in Phase 4 — Incus UI OIDC (OpenID Connect). Only the devops team from maceosoft enters (Dex filter) and, with Incus default configuration, those OIDC users have full access.

Deep dive: Incus Cluster — UI

Backup and snapshots (optional)¶

Not configured in bootstrap. The procedure is in Incus Backups; the alternatives, in Backup and snapshots.

Options¶

Decision	Default HomeLab	Alternative
Storage backend	`dir` (Ansible)	`btrfs`, `zfs`, `lvm` via `incus_storage_driver` in host_vars
oliver	`scheduler.instance manual`	—
Pool on deborah	Default path on eMMC	`incus_storage_path` on NVMe (`/srv/incus/...`)

Change driver in existing cluster

An already initialized cluster cannot be re-initialized with a preseed without manual migration. Choose the driver before the first playbook-incus-cluster.yml.

Trade-off analysis

Ansible (playbook-incus-cluster.yml)

Idempotent preseed, tokens, and UI in one step

Requires Ansible and SSH to all 3 nodes

Manual (incus admin init)

No playbook; useful for debugging a node

Single-use tokens; easy to drift versions

Storage dir

Zero friction, 20 GB free is enough

Slow VMs; no pool CoW

zfs / btrfs

Snapshots and better I/O

Dedicated disk/loop; cannot change after bootstrap

Pool on eMMC (deborah)

No NVMe mount needed

Risk of filling eMMC with images

Pool on NVMe (/srv/incus)

More space and I/O

Extra config in host_vars

Verify¶

incus cluster list
# 3 miembros ONLINE
incus cluster group list
incus storage list
# eth0 → parent br0
incus profile show default
curl -kI https://incus.homelab.local:8443

If it fails¶

Symptom	Check
Join rejected	Single-use token; stable IP in `br0`
Broken quorum	Incus versions aligned on all 3 nodes
Insufficient space	`df -h`; minimum 20 GB (`incus_storage_min_free_gb`)
Playbook not idempotent	Node already clustered; check `incus cluster list`

Next¶

→ Phase 3 — K3s

URLs and global /etc/hosts: HomeLab Summary (after Phase 4).