Skip to content

feat(cluster): let nodes on one LAN reach each other directly over the host network - #79

Draft
Mearman wants to merge 6 commits into
feat/pin-cluster-versionsfrom
feat/cluster-host-network
Draft

Mearman wants to merge 6 commits into
feat/pin-cluster-versionsfrom
feat/cluster-host-network

Conversation

@Mearman

@Mearman Mearman commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

Stacked on #80 (version pins), which it needs: the Compose renders here set both pinned versions, and the role's checks run after the version checks.

Lets cluster nodes that share a LAN reach each other over it directly, with the mesh's relays as the fallback, without changing anything k3s uses. The role adds a network mode and the checks around it; moving a node onto a Docker VM with its own LAN address is a documented runbook, not automation.

Today every node's k3s container sits on Compose's bridge network behind the Docker host's NAT, and on macOS behind the Docker VM's too. So the Tailscale client inside it (k3s --vpn-auth) has no address to offer its peers, and every pair of nodes is relayed, even on one LAN.

Changes

  • github_runner_cluster_network_mode (bridge, the default, or host). host merges docker-compose.host-network.yml last, putting the k3s and k3s-agent services on the Docker host's network with no published ports. The node's Tailscale client then offers the host's own addresses. Node IP, etcd peer URLs, TLS SANs and the mesh identity are unchanged. In host mode the role checks that the API server answers on the host's loopback before using the kubeconfig there.
  • Checks before the role changes anything, all read-only and run under --check too:
    • the mode must be bridge or host;
    • host mode needs github_runner_cluster_lan_cidr, and the Docker host must have an address in it, read from a throwaway container on the host network;
    • host mode is refused when the Docker host's network already has a Tailscale device that is not the node's own: the machine's own client on a Linux host with no VM, or a second node on the same Docker host;
    • when the running node is on the other kind of network, Compose is about to recreate it. The role reads the cluster through each server of the inventory in turn (this one first), each read bounded by github_runner_cluster_cluster_read_timeout as a whole and as kubectl's --request-timeout, etcd's included, using the first that answers, and refuses unless the other servers whose node is Ready and whose API server reports /readyz/etcd ok (read through each server's container on its host) still form a majority, and no unfinished ARC runner pod is on the node. Whatever cannot be told refuses, saying why: no server that answers (each named with its reason), no server in the inventory at all (an agent bootstrap.sh joins to an outside cluster), an unreachable Docker daemon, servers without their role labels, a missing or unreachable server, a server the inventory does not hold. Only one node may move per run, since each host's checks see the others as healthy before any is recreated.
  • No Colima automation. The role creates and changes no VM: a Colima profile's network mode is fixed when it is created, so an existing user-mode VM cannot be converted.
  • Docs. The cluster role README's Networking section covers both modes, the checks, what host mode exposes (API server and kubelet on the host's LAN address; other containers on the Docker host gain a route into the mesh), why host mode needs a Docker host of its own, and a runbook for moving a node to a bridged Colima VM: etcd health and the leader read with a throwaway etcdctl, moving leadership off the node, an etcd snapshot, cordon and drain, carrying the node password, the bridged interface (and that it can change on a later restart, seen with a test profile only), COLIMA_HOME, --activate=false and sizing, volumes created with Compose's labels, moving the engine first on the bridge network and then the network through the role, so the guard runs on the real move, identity checks after each (InternalIP, etcd member IDs and peer URLs, mesh device, no node-password event, the kubeconfig rewritten through the new VM's mount), starting the profile at login (what brew services covers and what it does not), and the rollback limits.
  • Entrypoint comment corrected to what k3s's StartVPN does: tailscale up ... --reset on every start.
  • No fixed tailscaled UDP port: the client offers its real local endpoint whatever port it chose, and the mesh scenario reaches every peer directly on random ports.

Tests

  • tests/unit/test_cluster_network.py renders the Compose project in both modes.
  • tests/unit/test_cluster_guard.py covers the guard: quorum for three and five servers, etcd unhealthy or unreachable, servers without labels, servers missing from the node list or from the inventory, runner pods, and LAN address matching.
  • tests/unit/test_host_network_role.py runs tasks/host_network.yml against a stand-in docker: the mode check, the LAN and Tailscale device checks, a healthy move (including under --check), refusals for a not-Ready server, an unhealthy etcd, unlabelled servers, a runner pod and two nodes moving at once, a read through another server when this one cannot list the cluster, a refusal naming each server when none can, an agent reading through the servers of a group with no bootstrap host, an agent whose servers are outside the inventory, an unreachable daemon, and the skips for a node already on its network, stopped or missing. It also checks from tasks/main.yml that the loopback API wait runs only on the host network and that the checks come before any change.
  • tests/mesh/run.sh host_network (Linux only) runs each node in a docker:dind daemon of its own, standing in for a VM with a LAN address: the usual cluster checks, every node reaching every other directly at its daemon's address, no published port, and a second run restarting nothing. Then the role must refuse to move two running nodes to the bridge network in one run without restarting anything, and move one node to the bridge network and back through its guard against the real cluster, the node keeping its mesh address and the cluster its three etcd members.
  • Each check was mutation-tested by breaking the code it covers, including the cluster read under --check, set -e in it, the loopback API wait's condition and the mode check.

Not tested here

These need a bridged Colima VM on a real host: the runbook end to end, Lima forwarding the API server's port to the host's loopback, and the address probe on a real bridged VM. The role does not read which server leads etcd, so leadership is moved by hand in the runbook.

@Mearman

Mearman commented Oct 13, 2026

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
🔒 Security Review ✅ Completed 2026-10-13T14:36:01.358768Z a6266fa Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: fedaf7ae4c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@Mearman Mearman changed the title feat(cluster): let nodes on one LAN reach each other directly feat(cluster): let nodes on one LAN reach each other directly over the host network Oct 13, 2026
@Mearman

Mearman commented Oct 13, 2026

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: a6266fa094

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@Mearman
Mearman force-pushed the feat/cluster-host-network branch from a6266fa to a2bf347 Compare October 13, 2026 16:34
@Mearman
Mearman changed the base branch from main to feat/pin-cluster-versions October 13, 2026 16:34
@Mearman

Mearman commented Oct 13, 2026

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@Mearman
Mearman force-pushed the feat/cluster-host-network branch from a2bf347 to 302048d Compare October 13, 2026 16:43
@Mearman
Mearman force-pushed the feat/pin-cluster-versions branch from ca5df0f to b728aec Compare October 13, 2026 18:30
@Mearman
Mearman force-pushed the feat/cluster-host-network branch from 302048d to 445a987 Compare October 13, 2026 18:46
@Mearman

Mearman commented Oct 13, 2026

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

This security review couldn’t start because the paying account has exhausted its Codex allowance and has insufficient available credits. The payer can check their balance and add credits in the Codex usage dashboard, then try again. If you do not manage the paying account, contact this repository's admins.

@Mearman
Mearman force-pushed the feat/pin-cluster-versions branch from b728aec to c2b9c25 Compare October 13, 2026 20:28
@Mearman
Mearman force-pushed the feat/cluster-host-network branch from 445a987 to dfdfe6d Compare October 13, 2026 20:41
@Mearman
Mearman force-pushed the feat/pin-cluster-versions branch from c2b9c25 to 3c9fe72 Compare October 13, 2026 22:09
…e up

The entrypoint said k3s's StartVPN checks the backend state and skips its own tailscale up once
the daemon is running. It does not: on every start it runs tailscale up with --reset, adding only
--login-server and the --vpn-auth extraArgs. The identity survives, but every other preference is
reset, so a Tailscale setting a node needs has to be a tailscaled flag or an extraArgs entry.
A node container on Compose's bridge network sits behind the Docker host's NAT, so its Tailscale
client can offer its peers no address they can reach and every pair of nodes is relayed, even on
one LAN. github_runner_cluster_network_mode host merges docker-compose.host-network.yml last,
which puts the server and agent services on the host's network and empties the port list; the
client then offers the host's own addresses and nodes that share a network connect directly,
falling back to a relay otherwise. bridge stays the default. Nothing k3s uses changes. In host
mode the role checks that the API server answers on the host's loopback before it uses the
kubeconfig there, since no port is published.
…ir own

A host_network scenario in the mesh integration test gives each node a Docker daemon of its own,
a docker:dind container at a fixed address on the shared test network standing in for a VM with
an address on the LAN, and runs the role with github_runner_cluster_network_mode host against a
Headscale server the test starts. Besides the usual cluster checks, every node's Tailscale client
must reach every other directly at that daemon's address, the node containers must publish no
port, and a second run must restart nothing. The workflow runs it on pull requests.
… node

Before the role changes anything on a host it checks the node's network mode,
and with github_runner_cluster_network_mode host that the Docker host has an
address on github_runner_cluster_lan_cidr, read from a throwaway container on
the host network, and no Tailscale device of another client there: the host's
own on a Linux machine with no VM, or a second node on the same Docker host.

Moving a running node between the bridge and the host network recreates its
container, so the role first reads the cluster through each server of the
inventory in turn, this one first, and refuses unless the other servers whose
node is Ready and whose API server reports etcd healthy still form a majority,
and no ARC runner pod is on the node. Whatever cannot be told refuses the move,
saying why: no server that answers, servers without their role labels, a missing
or unreachable server, one the inventory does not hold, or no server in the
inventory at all. Only one node may move per run. Every check only reads.
@Mearman
Mearman force-pushed the feat/cluster-host-network branch from dfdfe6d to f549691 Compare October 13, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant