Skip to content

Phase 6 — CAPN (optional)

Objective

Provision workload clusters kubeadm on Incus instances via Cluster API and CAPN (Cluster API Provider for Incus).

What You Learn

CAPI (Cluster API) declares clusters as Kubernetes resources; CAPN translates LXCMachine into Incus instances — ephemeral clusters that do not touch the management K3s (lightweight Kubernetes distribution).

Learn this phase

In simple terms: Cluster API is a 3D printer for clusters: instead of building each cluster by hand, you write a description and the system fabricates it. CAPN is the "accessory" that uses Incus as the factory: each node of the new cluster is an Incus instance.

Key concepts

  • Management cluster and workload cluster: the former manages; the latter is the one that is created and used.
  • Provider: the plugin that knows how to create machines on a specific platform (here, Incus).
  • Declarative resources: a cluster is described as a Kubernetes object (for example LXCMachine).
  • Ephemeral clusters: created for a test and destroyed afterward.

Practical challenge (safe): generate the description of a cluster and read it, without applying it. First export the variables that appear in the "Run" section of this phase; then:

clusterctl generate cluster demo -i incus \
  --kubernetes-version v1.36.2 \
  --control-plane-machine-count 1 \
  --worker-machine-count 2 > /tmp/demo.yaml
# solo lectura: no ejecutes kubectl apply
less /tmp/demo.yaml
Why create clusters from a manifest and not by hand?

They repeat the same way every time, they stay in Git and can be destroyed and recreated without effort.

What is the difference between a management cluster and a workload cluster?

The management cluster creates and monitors the others; workload clusters run the applications.

Questions? Use the Learn with AI button next to each title, or the page Learn.

Stack for this phase

sequenceDiagram
  participant Op as Operador
  participant K3s as K3s_management
  participant CAPI as Cluster_API
  participant CAPN as CAPN_controller
  participant Incus as Incus_API
  participant WL as Workload_cluster
  Op->>K3s: clusterctl_init
  Op->>K3s: apply_demo_yaml
  CAPI->>CAPN: reconcile_LXCMachine
  CAPN->>Incus: crear_LXC
  Incus->>WL: kubeadm_listo
  Op->>WL: kubectl_workload
  • Cluster API — Declarative API; Phase 6.
  • CAPN — Incus provider; Phase 6.
  • clusterctl — Bootstrap CAPI in the management cluster.
  • Incus — Instance runtime; Phase 2.
  • Sealed Secrets — Encrypted Incus credentials in Git.

Further reading: CAPN · Secrets

This decision in other environments

Cluster API with CAPN creates kubeadm test clusters as Incus instances from a manifest in Git, and deletes them when done. It serves to learn the same model that large platforms use, without paying for cloud.

  • Benefits: clusters are declared as code: they are created the same way every time, versioned and audited in Git, updated node by node with a version change, and replaced instead of being fixed by hand. With the same API, clusters are managed in the data center and across multiple clouds.
  • Fleet scale: with dozens or hundreds of clusters, inventory, policies, and centralized updates are needed; a fleet console is often added, such as SUSE Rancher Prime or Red Hat Advanced Cluster Management.
  • The management cluster is critical: if it is lost, the clusters keep running but no one can update or scale them; in production, it is given high availability and backup, and it is moved with clusterctl move.
  • Support: CAPN is a community project of Linux Containers; cloud providers have more users and vendor support.
  • When the same applies as in a HomeLab: private cloud on Incus, edge sites, and test environments that are created and destroyed often, where a managed cloud cluster is not available or does not make sense.
Cloud Equivalent service
GCP Provider CAPG; fleet with GKE fleet management
AWS Provider CAPA; outside AWS, EKS Anywhere or EKS Hybrid Nodes
Azure Provider CAPZ; fleet with Azure Arc-enabled Kubernetes
Oracle Cloud (OCI) Provider CAPOCI

Replacing CAPN with a cloud provider gives you managed machines, networks, and load balancers, with SLAs and near-unlimited scale. You lose control of the hardware, pay for every node that is running, and the manifests use cloud-specific resources, so they do not move as-is to another cloud.

Before You Start

The Phase 5 — vCluster is optional: it is not required for this phase.

Control plane load balancer

Each CAPN cluster creates its own load balancer for the API (LOAD_BALANCER='lxc: {}', a container inside Incus). It is independent of the Incus network load balancers, which only exist in the OVN network ovn-lan: see Load Balancing and OVN.

Run

Install clusterctl and init CAPN

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.10.10/clusterctl-linux-amd64 -o clusterctl
chmod +x clusterctl && sudo mv clusterctl /usr/local/bin/
mkdir -p ~/.cluster-api
curl -o ~/.cluster-api/clusterctl.yaml https://capn.linuxcontainers.org/static/v0.1/clusterctl.yaml
clusterctl init -i incus

Generate and apply demo cluster

See full commands in CAPN — further reading.

kubectl apply -f gitops/argocd/apps/capn-demo.yaml

Manual sync in ArgoCD (Application capn-demo).

With the Argo CD CLI:

argocd login argocd.homelab.local --grpc-web
argocd app sync capn-demo

Options

Decision Default HomeLab
Sync Argo CD Manual (ephemeral, avoids accidental deletions)
Target workers @arm64-nodes (deborah) / CP @x86-nodes
Trade-off analysis
Manual sync
Ephemeral clusters do not delete themselves
Apply + Sync in UI each time
Workers on @arm64-nodes
Leverages 31 GB of deborah
Less margin if deborah is already a CP K3s
CP on @x86-nodes
Separates CP workload / management
More RAM on limited x86 nodes
CAPN vs vCluster
Real VMs/Incus; full kubeadm
Much more RAM than vCluster in pods

Verify

kubectl get clusters
kubectl get machines

If it fails

Symptom Check
The Machine does not advance Secret Incus, Incus IP in lxc-secret
Image pull images.linuxcontainers.org

Next

→ HomeLab Summary · Tech Stack