Route all pod egress via Cilium Egress Gateway; restrict node1 SSH to Servers - #246
Merged
Conversation
node1 is multi-homed (Servers/K3s/Storage VLANs); the kernel's own routing table sends everything - including pod traffic - out via Servers regardless of Cilium's own network config, so nothing from the cluster could ever reach rb5009's Management API even though the firewall already allowed it on the K3s-facing chain. A CiliumEgressGatewayPolicy SNATs all pod egress through a kidibox.net/egress-gateway-labeled node's k3s interface instead, independent of the host's own default-route choice. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
sshd previously listened on every interface node1 has (Servers, K3s, Storage), reachable from any VLAN. Bind it to node1's Servers address only, matching how deploy/nixos-anywhere-install already reach it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
kid
force-pushed
the
cilium-egress-gateway
branch
from
August 20, 2026 17:19
e6b227b to
7e2aa0e
Compare
kid
enabled auto-merge
August 20, 2026 17:21
4 tasks
kid
added a commit
that referenced
this pull request
Aug 20, 2026
## Summary Follow-up to #246: node1 got a fresh `nixos-anywhere` reinstall after two problems surfaced. - **SSH lockout**: binding `sshd` to node1's Servers IP (`services.openssh.listenAddresses`, from #246) locked out SSH entirely after a reboot — the address is DHCP-assigned, not guaranteed present when sshd starts. Reverted to plain `enable = true;` for now; the interface/firewall-scoped version of this restriction is a separate follow-up. - **k3s/etcd outage**: node1 is multi-homed (Servers/K3s/Storage VLANs) and k3s's own `--node-ip` auto-detection isn't stable across reboots. After the reboot, k3s picked a different address than etcd's on-disk membership record expected (`this server is not a member of the etcd cluster`), and the control plane got stuck retrying forever. Pinned `--node-ip` explicitly to each host's own `<hostname>-k3s` device address (the K3s VLAN) so it's deterministic — same root-cause class as the pod-egress routing issue in #246, just hitting k3s's own control-plane binding this time. Verified against node1's current live state (post-reinstall, currently healthy on `10.0.40.10` — the K3s VLAN address) — the pin matches what's already running, so applying it is a same-address restart, not an IP change. ## Test plan - [x] `nix build .#nixosConfigurations.node1.config.system.build.toplevel` — builds cleanly, confirmed `--node-ip=10.0.40.10` in the built k3s unit - [x] `nix flake check --print-build-logs` — all checks pass - [ ] `deploy node1` and confirm k3s restarts cleanly with no etcd membership error - [ ] Confirm SSH access is restored from any VLAN (matching pre-#246 behavior) until the interface-scoped restriction is designed properly 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
node1is multi-homed (Servers/K3s/Storage VLANs). The kernel's own routing table sends all outbound traffic — including pod traffic — via the Servers interface regardless of the pod's originating network, so nothing from the cluster ever reached rb5009's Management API even though the firewall already allowed it on the K3s-facing chain (input-K3s). Confirmed directly on bothnode1(ip route get 10.99.0.1, plaincurlfrom the host) and on rb5009 (/ip firewall connection printshowed literally nothing arriving from node1).CiliumEgressGatewayPolicy(modules/kubernetes/cilium/egress-gateway.nix) that SNATs all pod egress traffic, any destination, through akidibox.net/egress-gateway: "true"-labeled node'sk3sinterface — independent of the host's own default-route choice. Deliberately cluster-wide rather than Management-CIDR-specific, so K3s becomes the one deterministic egress path for pod traffic and any future consumer (e.g. mikrotik-exporter) needs no policy change.applications.cilium.yamls), not nixidy's typed CRD generator — confirmed Cilium's chart ships nocrds/directory (its CRDs are installed by the runningcilium-operator), sogenerators.fromChartCRDModulecan't introspect them. Matches the existing pattern already used inmodules/kubernetes/cilium/bgp.nixfor the same reason.modules/hosts/node1.nixgets thekidibox.net/egress-gateway=truek3s node label (generic, not hostname-pinned, so a second node just needs the same flag) and, in a separate commit, restrictssshdto node1's Servers address only (previously reachable from every VLAN node1 touches).Test plan
nix run .#write-manifests— renderedmanifests/prd/cilium/CiliumEgressGatewayPolicy-egress-gateway.yamlnix flake check --print-build-logs— all checks passnix build .#nixosConfigurations.node1.config.system.build.toplevel— builds cleanly; confirmed--node-label=kidibox.net/egress-gateway=truein the built k3s unit andListenAddress 10.0.10.10:22in the built sshd configdeploy node1(applies the node label + SSH restriction) and let ArgoCD sync (applies the Cilium Helm value + policy) — label should land before/alongside the policy synckubectl get ciliumegressgatewaypolicy egress-gateway -o yaml— check.status10.99.0.1:443and goesReadyssh kid@10.0.10.10still works🤖 Generated with Claude Code
https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR