Skip to content

Kubernetes secrets, third alternative: peak-scale/sops-operator - #240

Merged
kid merged 11 commits into
mainfrom
sops-operator-secrets
Aug 18, 2026
Merged

Kubernetes secrets, third alternative: peak-scale/sops-operator#240
kid merged 11 commits into
mainfrom
sops-operator-secrets

Conversation

@kid

@kid kid commented Aug 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Summary

Kubernetes secrets as SopsSecret custom resources, committed to git as
real ciphertext, decrypted in-cluster by
peak-scale/sops-operator
using a dedicated per-cluster age keypair.

Stacked on #238 (host SSH key bootstrap): the cluster's age key is
decrypted using each cluster-member host's own SSH host key as the
sops/age recipient, so a live host needs that identity already committed
before it can join a cluster.

Design

  • One age keypair per cluster (not per host), generated locally via
    provision-cluster-key, sops-encrypted, and committed under
    secrets/clusters/<cluster>/.
  • The cluster's age key is delivered via
    sops-nix: decrypted straight from
    committed ciphertext on every activation, using the host's own persisted
    SSH key as the decryption identity, so a rotated key reaches a live host
    on the next deploy — no reinstall required. (Only the host's own SSH
    identity is legitimately install-time-only and stays on
    nixos-anywhere.)
  • .sops.yaml (generated from Nix, write-sops-config) has rules scoped
    per host, per cluster, and per cluster-manifest path. Human recipients
    are sourced from den.users.registry, not hardcoded. The rule governing
    rendered SopsSecret manifests sets encrypted_regex/mac_only_encrypted
    per sops-operator's own documented requirements, so only
    data/stringData get encrypted — apiVersion/kind/metadata stay
    cleartext, which kubectl/the operator need to route the object at all.
  • Secret values: a committed, sops-encrypted, pure-data JSON file at
    secrets/clusters/<cluster>/<namespace>/<name>.sops.json — no
    Kubernetes shape leaks into secrets/, and grouping related fields
    (e.g. username/password) is just more keys in that file. Edited
    directly with sops (sops -e -i to create, sops <file> to edit),
    no custom provisioning tool. _secrets-lib.nix renders only the empty
    SopsSecret shape via Nix; write-manifests builds the plain sandboxed
    manifest tree, then for each rendered SopsSecret-*.yaml looks up its
    value file by the file's own metadata, decrypts it for real, splices it
    into stringData, and re-encrypts the file in place.
    manifests/prd/** stays fully generated and safe to delete/regenerate —
    nothing under it is ever hand-edited.
  • A standalone systemd oneshot (k3s-sops-operator-seed) syncs the
    decrypted cluster key into a Kubernetes Secret on every activation
    (kubectl apply, not create-if-absent), so a key rotation propagates.
  • Prototype: a Cloudflare DNS API token SopsSecret under cert-manager,
    proving the chain end to end.

Verified

  • sops-nix's decryption path exercised for real: decrypted node1's own
    host key locally, converted it to an age identity with ssh-to-age,
    and used only that identity — no human key involved — to decrypt the
    cluster's committed sops-age key.
  • The full secret-value chain exercised for real: sops --decrypt on the
    rendered SopsSecret-cloudflare-dns-api-token.yaml recovers the real
    committed value; apiVersion/kind/metadata are cleartext and only
    stringData is ENC[...]; a second write-manifests run reproduces
    the same result from the same committed source.
  • nixosConfigurations.node1 evaluates cleanly with sops-nix wired in.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR

@kid
kid force-pushed the sops-operator-secrets branch 2 times, most recently from 2d59496 to f6ef125 Compare August 17, 2026 18:31
@kid
kid marked this pull request as ready for review August 18, 2026 15:09
@kid
kid force-pushed the sops-operator-secrets branch from 5684997 to 1c7ae7c Compare August 18, 2026 15:15
Base automatically changed from host-ssh-key-bootstrap to main August 18, 2026 15:16
kid and others added 11 commits August 18, 2026 18:04
@kid @claude
Add provision-cluster-key, cluster sops recipients, and bootstrap wiring
66660ec 
provision-cluster-key <cluster>: generates an age keypair locally, commits
the sops-encrypted private half and plaintext public half under
secrets/clusters/<cluster>/ (mirrors provision-host-key's shape). Ran it
for prd — a real committed key, not a placeholder.

write-sops-config now also derives one path-scoped rule per cluster with
a committed key, alongside the existing per-host rules.

nixos-anywhere-install now also stages a cluster's decrypted sops-age key
at persist/etc/sops-operator/age-key when the target host is a k3s node
for a cluster with one committed, via the same --extra-files mechanism as
the host's own SSH key.

New k3s-sops-operator aspect (included on node1): persists that key
through impermanence, and a standalone systemd oneshot (not joined to the
six-wave k3s-bootstrap chain) that idempotently creates and labels the
sops-age-prd k8s Secret sops-operator reads as its decryption key.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Add sops-operator app aspect + cluster-scoped SopsProvider
3b22887 
Deploys peak-scale/sops-operator via a chart fetched directly from its
GitHub repo (not in nixhelm's catalog), pinned to v0.10.1. One
cluster-scoped SopsProvider (empty matchLabels on both keys and sops --
matches everything, no Capsule/tenant scoping needed for a single-tenant
homelab cluster).

Verified: nix run .#write-manifests renders correctly, including a real
SopsProvider-prd.yaml with the exact expected shape.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Restructure write-manifests to use the real nixidy switch CLI
e34b2c9 
Replaces the sandboxed environmentPackage+rsync build with an invocation
of nixidy's own switch command, so objectTransforms.postProcess actually
runs (needed for modules/kubernetes/sops-operator/default.nix's
sops --encrypt rule on SopsSecret objects — postProcess deliberately
never runs inside a Nix build sandbox, only via the real CLI).

checks.manifests keeps comparing against environmentPackage (unchanged
mechanism) but excludes SopsSecret-*.yaml from the diff, since those are
expected to differ: committed = real ciphertext (produced by a human
running write-manifests), environmentPackage = never-postprocessed
plaintext (sandboxed build can't run postProcess or hold real sops keys).

Verified for real: ran the actual `nix run .#write-manifests` end to end
(no SopsSecret exists yet, so this only proves the non-secret path, but
proves it against the real packaged command, not just a manual CLI test).
Also separately confirmed with a manual `nixidy switch .#prd` in a
disposable git worktree that the mechanism itself works before wiring it
into files.nix — this caught a real mistake (the flake attribute is the
cluster's own name, "prd", not the internal "prd-prd" env identifier
nixidy also exposes).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Add SopsSecret authoring convention + fix cluster-key sops recipients
b473a0f 
SopsSecret's CRD schema requires a top-level sops block that only
exists after encryption, so it can't be authored as a typed nixidy
resource (fromChartCRDModule enforces the full schema at eval time,
before that block exists). _secrets-lib.nix instead renders it as raw
YAML from a local, gitignored plaintext value file, re-encrypted by
nixidy's postProcess the moment write-manifests runs.

That postProcess command piped through /dev/stdin with no filename,
so sops had nothing to match .sops.yaml's path_regex rules against and
silently fell back to the humans-only catch-all -- sops-operator,
holding only the cluster's private key, could never have decrypted
its own SopsSecrets. Fixed by passing --filename-override "$TARGET_PATH",
and by adding a creation_rule scoped to each cluster's own rendered
manifest path. Verified end to end: a real write-manifests run now
encrypts to humans + the prd cluster's key, and sops --decrypt round-trips.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Drop code-comment references to the other draft secrets-management PRs
9623eef 
sops-operator stands on its own; comparing it against sibling designs
belongs in PR descriptions, not comments that outlive the comparison.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Deliver the cluster sops-age key via sops-nix, not nixos-anywhere --e…
090acc8 
…xtra-files

nixos-anywhere only runs at install time, so a rotated or newly-committed
cluster key would never reach an already-live host without a full
reinstall -- the wrong lifecycle for an ongoing application secret. Host
identity (the SSH key itself) is still legitimately install-time-only and
stays on nixos-anywhere; the cluster key it protects now goes through
sops-nix instead, decrypted straight from committed ciphertext on every
activation using the host's own persisted SSH key, so `deploy` alone picks
up a rotated key.

sops-config.nix's clusterRule now also grants each cluster-member host's
own pubkey as a recipient on the cluster's committed key file, and the
existing prd file was updated via `sops updatekeys`. Verified for real:
decrypted node1's own host key locally, converted it to an age identity
with ssh-to-age, and used only that identity (no human key involved) to
decrypt the cluster's sops-age key -- the exact path sops-nix takes on the
host itself.

The k3s-sops-operator-seed unit now `kubectl apply`s instead of
create-if-absent, so a rotated key actually re-syncs the cluster's
Kubernetes Secret instead of being silently ignored after the first run.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Rework secret values: committed under secrets/, merged into manifests…
2a2e4b7 
…/ at write-manifests time

Three real problems with the previous design, from direct review:

- The value lived in a gitignored local-only file -- nothing committed,
  nothing recoverable if that file was lost. "Stored in git, encrypted"
  was the actual requirement.
- Nix-eval-time builtins.readFile can never see a decrypted value anyway:
  nixidy's build (and any nested `nix build`) only sees git-tracked/staged
  files, so a gitignored scratch copy is invisible to it regardless of
  where it lives. This was a latent bug in the very first version of this
  design -- it was never actually exercised with a real, non-empty value
  until now.
- sops-operator's own docs (usage.md#generate-sops-configuration) require
  `encrypted_regex: ^(data|stringData)$` and `mac_only_encrypted: true` on
  the creation_rule -- otherwise apiVersion/kind/metadata get encrypted
  too, which breaks kubectl/the operator routing the object at all before
  it's even decrypted. Confirmed broken by inspecting a real committed
  file: apiVersion/kind/metadata.name were all ENC[...].

New design: a secret's value is a committed, sops-encrypted, pure-data
JSON file under secrets/clusters/<cluster>/<namespace>/<name>.sops.json --
no Kubernetes shape leaks into secrets/, and grouping fields (e.g.
username/password) is just more keys in that same object. Edited directly
with sops (`sops -e -i` to create, `sops <file>` to edit later), no custom
provisioning tool. `_secrets-lib.nix` renders only the empty SopsSecret
shape via Nix (namespace/name, stringData = {}); `write-manifests` builds
the plain sandboxed environmentPackage as before, then for each rendered
SopsSecret-*.yaml looks up its value file by the file's own metadata,
decrypts it for real (this step already has genuine sops access, unlike
Nix eval), splices it into stringData, and re-encrypts the whole file in
place. manifests/ stays fully generated and safe to delete/regenerate --
nothing under it is ever hand-edited.

Verified end to end: sops --decrypt on the resulting
manifests/prd/cert-manager/SopsSecret-cloudflare-dns-api-token.yaml
recovers the real committed value, apiVersion/kind/metadata are cleartext
and only stringData is ENC[...], and a second write-manifests run
reproduces the same result from the same committed source.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Name committed cluster key ciphertext .sops, not .sops.binary
9321494 
Same rename as the host key (host-ssh-key-bootstrap) -- the .binary
suffix was redundant on top of .sops. Also fixes a stale comment in
provision-cluster-key.nix still describing the pre-sops-nix
nixos-anywhere --extra-files delivery mechanism, and tightens the
secrets/clusters/ .gitignore rule to un-ignore bare .sops files too
(previously only .sops.* with a further suffix matched).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Move SopsSecret shape-building onto den.clusters.prd.methods, drop _s…
7da7d3f 
…ecrets-lib.nix

_secrets-lib.nix was a standalone import at each app-aspect call site.
Relocated its one function onto the cluster entity instead, under a new
generic `methods` namespace (den.schema.cluster) that any aspect can
contribute a named callable to -- not a single hardcoded field, so the
generic schema doesn't need to know about sops-operator specifically.
modules/kubernetes/sops-operator/default.nix sets
den.clusters.prd.methods.mkSopsSecret; app aspects add `cluster` to their
k8s-manifests arg pattern (den auto-injects the full entity by declared
arg name -- confirmed working for an app aspect, not just nixidy's own
core module) and call cluster.methods.mkSopsSecret {...} directly.

Also tried switching this to a typed nixidy CRD resource instead of
builtins.toJSON, per a real request -- confirmed infeasible, not just in
theory: adding "SopsSecret" to the fromChartCRDModule kindFilter breaks
the entire environment build even with zero resources.sopsSecrets
instances declared, because the CRD's `sops` block requires
lastmodified/mac with no default and nixidy's schema-walk forces
evaluation of every declared kind's defaults regardless of usage.
nixidy's own docs independently steer away from typed resources for
sops-encrypted manifests for a related reason (round-tripping ciphertext
through kube.fromYAML/toYAML can reformat ENC[...] values). Kept
builtins.toJSON, per direct confirmation after presenting this finding.

Verified end to end: `cluster.methods.mkSopsSecret` resolves and renders
from an app aspect, and sops --decrypt on the resulting
SopsSecret-cloudflare-dns-api-token.yaml still recovers the real
committed value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Trim comment verbosity in the last commit
db73dd9 
Comments should be 1-2 lines, only for facts not inferable from the code
or git history -- not a narration of what changed or how it was found.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid @claude
Fix stale _secrets-lib.nix reference in .gitignore comment
e4c9351 
Left over from before that file was deleted; also trims it per the
1-2-line comment convention.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
@kid
kid force-pushed the sops-operator-secrets branch from fc07975 to e4c9351 Compare August 18, 2026 16:05
@kid
kid merged commit 62b4936 into main Aug 18, 2026
1 check passed
@kid
kid deleted the sops-operator-secrets branch August 18, 2026 16:33
@kid kid mentioned this pull request Aug 18, 2026
Merged
3 tasks
kid added a commit that referenced this pull request Aug 18, 2026
@kid
Sweep all app manifests for Namespace/CRD resources in k3s bootstrap (#…
b229e61 
…241)

## Summary
- `k3s-bootstrap-cert-manager` failed on node1: cert-manager's manifests
include a `SopsSecret` custom resource (#240), but its CRD only exists
under `manifests/prd/sops-operator/`, which wave 1
(`k3s-bootstrap-crds`) never scanned.
- Waves 0 and 1 used a hand-maintained list of app directories that
predated sops-operator, and had the same gap for `miroir`'s
CRDs/namespace.
- Point both waves at the whole `manifests/prd/` tree instead, so a new
app's Namespace/CRD resources get picked up automatically.

## Test plan
- [x] `nix flake check --print-build-logs` passes
- [x] Built `k3s-bootstrap-namespaces`/`k3s-bootstrap-crds` ExecStart
scripts locally and confirmed they now `find` across `manifests/prd/`
(picking up sops-operator's and miroir's CRDs/namespaces)
- [ ] `deploy node1` and confirm `k3s-bootstrap-cert-manager`,
`k3s-bootstrap-coredns`, `k3s-bootstrap-argocd` all succeed

https://claude.ai/code/session_01CezjyaVpC3FVPTUECp7cMR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@kid