netaminity

Netaminity is a Kubernetes operator for TCP tunnels built around netaminity-agent, a ekzhang/bore fork.

What is this?

Kubernetes networking is usually easy when every pod can reach every other pod. But sometimes you need to connect pods across network boundaries. The most common solution is to join all nodes into a single network using VPN, but in some cases you cannot or do not want to do that. That's where Netaminity comes in.

You might run one part of a cluster on cloud nodes and another part on nodes inside a private network. You might also run multiple clusters where one cluster can expose a reachable endpoint, while the other can only make outbound connections. Netaminity gives you Kubernetes-native resources for these cases without wiring bespoke sidecars, scripts, or manually managed tunnel processes.

What does "netaminity" mean?

idk.

How it works?

There are two main patterns.

One Cluster, Split Networks

Use Tunnel when the endpoints are in the same Kubernetes cluster but some nodes are not directly reachable from other nodes. For example, proxy pods can run on cloud nodes while target pods run on private-network nodes. The target side opens outbound connections to the proxy rendezvous Service, and users connect through the shared Tunnel Service. The proxy never needs to initiate a network connection to the target.

This is ideal when one operator instance can manage both sides.

What happens step by step:

1. You label nodes according to where they can run, for example environment=cloud and environment=private.
2. You create one Tunnel and provide separate nodeSelector values for its proxy and target pods.
3. Netaminity generates a shared Secret for the tunnel.
4. Netaminity creates indexed Proxy and Target resources, such as private-api-0 and private-api-1.
5. Each generated Proxy creates a proxy pod and a rendezvous Service. The proxy pods are scheduled on cloud nodes.
6. Each generated Target creates a target pod on a private node. The target pod opens outbound connections to its matching Proxy rendezvous endpoint.
7. Netaminity creates one shared consumer Service named private-api. This Service selects all generated proxy pods.
8. A consumer connects to private-api:30080. The proxy carries the application bytes bidirectionally over a data connection initiated by the target, and the target opens the final connection to private-api.private.svc:8080.

flowchart TB
    Consumer[Consumer] -->|TCP :30080| PublicService[Service/private-api]

    subgraph Cluster[One Kubernetes cluster]
        subgraph Cloud[Cloud nodes]
            PublicService --> Proxy0[Proxy pod private-api-0]
            PublicService --> Proxy1[Proxy pod private-api-1]
            Rendezvous0[Rendezvous Service :31035] --> Proxy0
            Rendezvous1[Rendezvous Service :31036] --> Proxy1
        end

        subgraph Private[Private-network nodes]
            Target0[Target pod private-api-0]
            Target1[Target pod private-api-1]
            Backend[private-api.private.svc:8080]
            Target0 -->|TCP| Backend
            Target1 -->|TCP| Backend
        end

        Target0 -. opens outbound connections .-> Rendezvous0
        Target1 -. opens outbound connections .-> Rendezvous1
        Proxy0 <== application bytes over target-initiated connection ==> Target0
        Proxy1 <== application bytes over target-initiated connection ==> Target1
    end

Loading

First, label the nodes. Use labels that match your own infrastructure; the following names are only examples:

kubectl label node <cloud-node> netaminity.exeteres.net/environment=cloud
kubectl label node <private-node> netaminity.exeteres.net/environment=private

Then create the Tunnel:

apiVersion: netaminity.exeteres.net/v1
kind: Tunnel
metadata:
  name: private-api
spec:
  replicas: 2
  service:
    name: private-api
    port: 30080
  podTemplate:
    resources:
      requests:
        cpu: 10m
        memory: 32Mi
  proxy:
    replicas: 2
    podTemplate:
      nodeSelector:
        netaminity.exeteres.net/environment: cloud
    proxyService:
      type: NodePort
      nodePort: 31035
  target:
    endpoint: private-api.private.svc:8080
    podTemplate:
      nodeSelector:
        netaminity.exeteres.net/environment: private

In this example:

• The Tunnel creates two independent Proxy/Target pairs because spec.replicas is 2.
• Each generated Proxy runs two interchangeable proxy pods because spec.proxy.replicas is 2.
• Proxy pods are scheduled onto cloud nodes.
• Target pods are scheduled onto private-network nodes.
• The first proxy rendezvous Service uses NodePort 31035; the second uses 31036.
• Consumers connect to Service/private-api on port 30080.
• The Tunnel reconciler creates and distributes the shared Secret to its generated Proxy and Target resources.

When all Target-to-Proxy tunnel connections stay inside the cluster, use proxy.proxyService.type: ClusterIP and omit nodePort.

The proxy rendezvous Service accepts the target's outbound connections. Proxied application bytes flow bidirectionally over those target-initiated connections; no inbound connection from a proxy pod to a target pod is required.

Tunnel has two independent scaling dimensions. spec.replicas controls the number of indexed Proxy/Target pairs, while spec.proxy.replicas controls the number of interchangeable proxy pods within every generated pair. Each pair still has one target pod. Use multiple pairs for parallel tunnel capacity and per-pair Proxy replicas for proxy-side failover.

(!) Warning: All traffic between the proxy and target pods is intentionally unencrypted. You should use TLS or other encryption if the network between them is not trusted.

Multiple Clusters

Use Proxy and Target directly when the two endpoints live in different Kubernetes clusters. In this model, each cluster has its own operator instance, and you deliver the shared Secret yourself to both clusters.

What happens step by step:

1. You install Netaminity in both clusters.
2. You generate a strong shared secret outside the operator.
3. You create a Kubernetes Secret with the same value in both clusters. Netaminity does not copy Secrets across clusters.
4. In the reachable cluster, you create a Proxy. It creates the consumer Service, proxy rendezvous Service, and proxy pod.
5. You wait for Proxy.status.proxyEndpoint to contain an endpoint reachable from the other cluster.
6. In the private cluster, you create a Target using that value as spec.proxyEndpoint.
7. The target pod opens outbound connections from the private cluster to the proxy rendezvous endpoint.
8. Consumers connect to the Proxy consumer Service. Traffic travels bidirectionally over the target-initiated connections and exits from the target pod toward the private endpoint.

flowchart TB
    Consumer[Consumer] -->|TCP :30080| ConsumerService[Proxy consumer Service]

    subgraph CloudCluster[Cloud cluster]
        ConsumerService --> Proxy[Proxy pod]
        Rendezvous[LoadBalancer rendezvous Service :7835] --> Proxy
        SecretA[Secret/private-api-tunnel] -. authentication .-> Proxy
    end

    subgraph PrivateCluster[Private cluster]
        Target[Target pod]
        Backend[private-api.private.svc:8080]
        SecretB[Secret/private-api-tunnel] -. authentication .-> Target
        Target -->|TCP| Backend
    end

    Target -. opens outbound connections .-> Rendezvous
    Proxy <== application bytes over target-initiated connection ==> Target
    SharedSecret[Same secret value delivered by user] -.-> SecretA
    SharedSecret -.-> SecretB

Loading

Install the operator in both clusters and select each context when applying resources. Generate the shared value once:

SHARED_SECRET="$(openssl rand -base64 32)"

Create the Secret in the cloud cluster:

kubectl --context <cloud-cluster> create secret generic private-api-tunnel \
  --from-literal=secret="${SHARED_SECRET}"

Create the same Secret in the private cluster:

kubectl --context <private-cluster> create secret generic private-api-tunnel \
  --from-literal=secret="${SHARED_SECRET}"

The cluster that receives inbound traffic runs Proxy:

apiVersion: netaminity.exeteres.net/v1
kind: Proxy
metadata:
  name: private-api
spec:
  replicas: 2
  secretRef:
    name: private-api-tunnel
  service:
    name: private-api
    port: 30080
  proxyService:
    type: LoadBalancer

Apply it to the cloud cluster, then wait for it to become ready and read the resolved endpoint:

kubectl --context <cloud-cluster> apply -f proxy.yaml
kubectl --context <cloud-cluster> wait proxy/private-api \
  --for=condition=Ready=True \
  --timeout=5m
kubectl --context <cloud-cluster> get proxy/private-api \
  -o jsonpath='{.status.proxyEndpoint}{"\n"}'

The cluster that can reach the private endpoint runs Target:

apiVersion: netaminity.exeteres.net/v1
kind: Target
metadata:
  name: private-api
spec:
  secretRef:
    name: private-api-tunnel
  endpoint: private-api.private.svc:8080
  proxyEndpoint: proxy.example.net:7835

Set proxyEndpoint to the value returned by the Proxy status, then apply the Target in the private cluster:

kubectl --context <private-cluster> apply -f target.yaml
kubectl --context <private-cluster> wait target/private-api \
  --for=condition=Ready=True \
  --timeout=5m

proxyEndpoint must point at the reachable proxy rendezvous endpoint reported by the Proxy status. The Target opens outbound connections to this endpoint. For LoadBalancer, Netaminity reports the LoadBalancer ingress on port 7835. For NodePort, configure the operator with NETAMINITY_PROXY_HOST or Helm config.proxyHost.

Proxy.spec.replicas defaults to 1. With a larger value, the operator keeps exactly one live proxy pod in the rendezvous Service's managed EndpointSlice. It retains that active endpoint across reconciliations and switches to a standby when Kubernetes reports that the selected pod is terminating, its agent container stops or restarts, or its node becomes non-ready. This active/standby routing does not depend on preserving the Target source IP through an external load balancer. Only the pod carrying the healthy Target session becomes ready and receives consumer traffic. Standby pods remain live but unready without restarting. If the active pod is lost, the Target reconnects through the same rendezvous Service and the selected standby can become ready. A Proxy resource is Ready when at least one proxy pod is available.

Install

Install the published Helm chart from GHCR:

helm install netaminity oci://ghcr.io/exeteres/charts/netaminity \
  --version 0.3.2 \
  --namespace netaminity-system \
  --create-namespace

For local development, install from the checked-out chart directory:

helm install netaminity ./charts/netaminity \
  --namespace netaminity-system \
  --create-namespace

If you use NodePort proxy rendezvous Services, set a public node hostname or IP:

helm upgrade --install netaminity oci://ghcr.io/exeteres/charts/netaminity \
  --version 0.3.2 \
  --namespace netaminity-system \
  --create-namespace \
  --set config.proxyHost=<public-node-hostname-or-ip>

config.proxyHost is required when using NodePort proxy rendezvous Services. It is not needed for ClusterIP or LoadBalancer rendezvous Services.

Default images:

ghcr.io/exeteres/netaminity:0.3.2
ghcr.io/exeteres/netaminity-agent:0.6.0-na.4

Override them with Helm values under operator.image.* and agent.image.*.

Published artifacts:

• Operator image: ghcr.io/exeteres/netaminity
• Helm chart: oci://ghcr.io/exeteres/charts/netaminity
• Agent image: ghcr.io/exeteres/netaminity-agent

Scheduling

Proxy and Target pods support scheduling options through podTemplate:

podTemplate:
  # Enabled by default; prefer placing matching workload replicas on different nodes.
  distributeByNodes: true
  # Useful for Targets that must reach services bound to the node network.
  hostNetwork: false
  nodeSelector:
    kubernetes.io/os: linux
  tolerations:
    - key: dedicated
      operator: Equal
      value: netaminity
      effect: NoSchedule
  affinity: {}
  topologySpreadConstraints: []
  priorityClassName: ""
  runtimeClassName: ""
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      cpu: 100m
      memory: 64Mi
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containerSecurityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL

For Tunnel, use spec.podTemplate for shared proxy/target settings. Use spec.proxy.podTemplate or spec.target.podTemplate to override fields for one side.

distributeByNodes defaults to true and adds preferred pod anti-affinity on kubernetes.io/hostname. This is a preference rather than a requirement, so pods can still schedule when there are fewer eligible nodes than replicas. Set it to false to disable the automatic rule; explicit affinity remains supported and is preserved when distribution is enabled.

Set hostNetwork: true on a Target pod template when the target must reach node-local services. Netaminity uses ClusterFirstWithHostNet, so the Target can still resolve an in-cluster Proxy rendezvous Service. Every agent binds its health server to port 8080; host-networked Netaminity pods scheduled on the same node can therefore conflict on that port. Use node selection, anti-affinity, or another scheduling constraint to prevent such colocation.

Status

All resources expose a standard Ready condition.

Tunnel also reports:

• readyProxyReplicas
• readyTargetReplicas
• readyReplicas
• replicasStatus

Development

make manifests generate
make test
make docker-build IMG=ghcr.io/exeteres/netaminity:0.3.2
helm lint charts/netaminity
helm template netaminity charts/netaminity

Internal reconciliation behavior is documented in docs/INTERNAL.md.

Release

Releases are tag-driven. A pushed git tag like v0.1.0 runs the release workflow, which publishes image and chart versions without the v prefix:

• ghcr.io/exeteres/netaminity:0.3.2
• oci://ghcr.io/exeteres/charts/netaminity:0.3.2

Prepare a release commit and annotated tag with:

scripts/release.sh 0.1.0

Push the generated commit and tag:

git push origin HEAD
git push origin v0.1.0

Or prepare and push in one command:

scripts/release.sh --push 0.1.0

The release script updates the Helm chart version, app version, default operator image tag, Kustomize image tag, and README image references. It also runs generation, unit tests, e2e compile checks, and Helm validation before committing and tagging.

The release workflow requires the git tag version, after stripping the v prefix, to match both version and appVersion in charts/netaminity/Chart.yaml. For example, git tag v0.1.0 publishes image tag 0.1.0 and chart version 0.1.0.