Mixed data from different hosts

[gforcada]

The scenario is a bit complicated, I will try my best to make it as simple as possible 🤞🏾

I have a component Pkgs that provides itself under a few names:

class Pkgs(Component):
    features = ()

    packages = []

    def configure(self):
        if 'zope' in self.features:
            self.provide('pkgs:zope')
            self.packages.extend(['dep1', 'dep2']

        if 'db' in self.features:
            self.provide('pkgs:zope')
            self.packages.extend(['dep3', 'dep5']

       ...
       self += File('/etc/local/nixos/system-packages.nix')

And system-packages.nix is very simple:

{ pkgs, ...}:
{
  environment.systemPackages = [
    {% for pkg in component.packages %}
    pkgs.{{pkg}}
    {% endfor %}
  ];
}

The environment that triggered the problem is like this:

[host:db]
components =
    pkgs:db
    pkgs:python
    ...

# workers
[host:worker1]
components =
    pkgs:python
    pkgs:zope

[host:worker2]
components =
    pkgs:python
    pkgs:zope
    ...

i.e. a database server that needs db and python features from pkgs component and two workers that also need python but rather than db they need zope.

Turns out that when deploying (with batou==2.7.0) the zope dependencies are also installed in the database server, and even more, the second worker gets all packages (from db but twice the worker dependencies).

The fix that works for me is:

    def configure(self):
        tmp_list = []

        if 'zope' in self.features:
            self.provide('pkgs:zope')
            tmp_list.extend(['dep1', 'dep2']

        ...

        self.packages = sorted(set(tmp_list))

With this, each server gets the expected packages to be installed. No duplicates and specially no packages needed from by other servers.

On the components that need this Pkgs component, all of them use lines like self.require_one('pkgs:python', self.host)

Which I thought (the self.host part) ensured that only the pkgs:python configured for that very same host would be used.

I hope I managed to explain it properly 😅 otherwise please mention it and I will try to explain as best as I can.

[gforcada]

Actually that work on an environment, but the same code on a different environment was not working, so I'm even more confused now 😖

What's the best practice in batou to centralize system dependencies from different components to create a single list of what needs to be installed?

At the end what we need is a way to generate:

{ pkgs, ...}:
{
  environment.systemPackages = [
    {% for pkg in component.packages %}
    pkgs.{{pkg}}
    {% endfor %}
  ];
}

Where this component.packages is not the system dependencies of a single component, but from multiple ones, and some of them depending on the same dependency (i.e. python313).

Is it possible within NixOS to generate multiple such files ⤴️ ? that could simplify the problem (maybe)