Cross-namespace support in PackageVariant

[kispaljr]

Hivemind! I want to pick your brains regarding cross-namespace support in PackageVariant.

First, let me describe a Nephio-like use case: I have a so-called DeploymentStrategy CRD, whose controller will create multiple PackageVariants that clone blueprints to "for-deployment" repositories, and in the process it injects the WorkloadCluster object that describes the configuration and state of the target cluster into the downstream package. Due to lack of cross-namespace support in PackageVariant I have to put everything into the same namespace: the DeploymentStrategy object, the PackageVariants, the upstream and downstream Repository objects and the whole Nephio "inventory" that contains entries (KRM objects) that I occasionally want to inject into the deployment packages.

IMHO it would be reasonable to separate the Repositories and WorkloadCluster objects (and more) into a common nephio-inventory namespace that is common to multiple deployments/tenants. At the same time I want to keep the DeploymentStrategy and the PackageVariants it owns in a tenant/deployment-specific namespace.

What do you think?

@johnbelamaric wrote this in a previous thread:

But the answer is, yes, because permissions are managed by namespace, at least for now we don't try to do cross-namespace access requests. Those are complicated.

You can register the same repo multiple times, yes, that should work. But it's not super scalable right now. I want to see that addressed as part of #450.

I agree that this is a valid argument. I would still argue that it would worth the effort to enable cross-namespace support for at least the objects that PVs inject into downstream packages. One solution that comes to mind is to extend the PV spec with a serviceaccount field, and only allow access to resources from other namespaces that is allowed for that particular serviceaccount. I understand that this adds quite a bit of complexity to the PV controller implementation.

This issue is closely related to #507, but I extended the topic to the injected in-cluster KRM resources, as well.

[tliron]

In my opinion, we should raise the question as to whether any of the Porch resources need to be namespaced objects. IMO, they are management objects that do not clearly belong in namespaces and could happily live outside of them. The only advantage provided by namespaces is some kind of RBAC isolation (multi tenant management clusters? do such things even exist?), but I wonder if any of these edge cases that can't be solved equally well with ClusterRoles.

(In my TKO experiment the entire KRM interface is non-namespaced.)

[kispaljr]

Namespaces also help avoiding naming conflicts in a clean way. If I could I would even turn CRDs to namespaced resources :) , because that would allow e.g. different versions of the same Redis operator to live peacefully side-by-side in separate namespaces.

[kispaljr]

Still, I can see your point, if we talk about Repositories and PackageRevisions, but I would argue to keep PackageVariants as namespaced resources, exactly to avoid naming conflicts between tenants/topology deployments.

[johnbelamaric]

One solution that comes to mind is to extend the PV spec with a serviceaccount field, and only allow access to resources from other namespaces that is allowed for that particular serviceaccount.

Perhaps, but how do you make sure the PV author has permission to add that SA?

Gateway API solves this with ReferenceGrant objects, which effectively enable the admin to define import/export rules for resources between namespaces. See https://gateway-api.sigs.k8s.io/api-types/referencegrant/. There has been some discussion of making this part of the standard K8s API, but I don't know the status of that.

[henderiw]

I believe a SA approach is the way forward as I also brings in identity capabilities which is a big missing items today.

[kispaljr]

Perhaps, but how do you make sure the PV author has permission to add that SA?

Good question :) I am not a security expert, so I naively thought that if the PV accepts only SAs that are in the same namespace, then it is enough security for our purposes. :)

[johnbelamaric]

That's not unreasonable. Looking at https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-subjects it seems you would need to create a RoleBinding in the non-PV namespace that references the SA in the PV namespace. You would probable want to standardize on either a specific name for these types of SAs or at least a format; if you did that it may be possible to use gatekeeper or something to further restrict the creation of those SAs to based on other policies (instead of just anyone that can create an SA in the namespace).

[kispaljr]

Yes, exactly.
I very much like your idea of standardizing on the acceptable SAs in order to make Gatekeeper/Kyverno usage more efficient.
But, even without a policy engine, it is not just anyone who can create an SA in the namespace, but anyone who can create a Role and a RoleBinding or a ClusterRole and a ClusterRoleBinding that allows access to the external resources for the SA. I have no idea if it is enough, that's why I am glad to hear others' opinion. Probably we should involve SIG Security, as well.

[henderiw]

In the approach I will present on Wednesday I changes the logic of the packageVariant a bit. So the PackageVariant provides context for the pipeline, but all the injection and fancy logic becomes part of the pipeline. So this discussion would move to the pipeline and the access right the pipeline processor has for a particular package. I believe this is a more clean separation of the duties at hand.