Rework of the JuliaPackage easyblock

[Crivella]

Summary of changes

• Adds compile_cache check to sanity checks to ensure that the compiled version of the package can actually be found/re-used
• Allow specifying test-only dependencies with the is_test_dependency EC param
• Allow turning on Julia's verbose output with the julia_debug EC param
• Allow reusing packages from a dependency (fix JuliaPackage can't really work with depedencies #4123)
  • The environments of the dependencies are merged into the one of the package being installed
• Use Pkg.instantiate() to install all extensions instead of installing them one by one with Pkg.develop
  • Remove need for deps to be ordered in the EC files
  • Makes use of Julia's capability to install extension in parallel (still respecting the max_parallel option)
• Allows running package tests in a dedicated step (setting runtest to True for every extension that needs testing)
  • The test are both skippable (--skip-test-step) and ignorable (--ignore-test-failure)
  • Tests are ran in offline mode by default unless test_online is set to true
  • If both offline and online test are to be performed, the offline ones go first as they should not modify the test environment that we are setting up
• Switch to using an EB specific environment variable to keep track of the LOAD/DEPOT PATHS and use them to correctly set them for Julia in a site-specific startup script

Maybe TODO

• Improve Lua/Tcl syntax in module footer
• Modify Julia itself similar to https://github.com/easybuilders/JSC/blob/6285a066b6debed28211730afb8ffc5eb2430c2f/Custom_EasyBlocks/julia.py to allow the interpreter to pick up the DEPOT/LOAD path directly from the new env variables
• Test how reusable deps really are. The original approach aimed at copying all packages from the deps into the current package and re-install everything. Here we actually re-use them but the results seems to be dependent on the load order (see discussion in JuliaPackage can't really work with depedencies #4123) so if the dep order is switched w.r.t. the one at build time slow re-compilation might happen
  • PoC in {ai}[system/system] Flux v0.16.10 w/ Julia 1.12.2 easybuild-easyconfigs#26214 at-least on package used at the same time should work, more investigation might be needed on loading multiple packages that re-compiles the same deps at the same time

Notes

This PRs includes the changes from:

• Fix for running --sanity-check-only for a JuliaBundle #4122
• JuliaPackage: Allow running tests in a dedicated test_step #4124

Fixes:

• JuliaPackage can't really work with depedencies #4123

AI usage

• GitHub CoPilot FIM auto-completion is active in my IDE and has been used in this PR to sporadically auto-complete code/docstrings

[Crivella]

Moved the final setting of LOAD_PATH and DEPOT_PATH to Julia's site-specific startup script.
This resolves:

• having to write custom module code
• the reversing setenv problem
• Lua-vs-Tcl syntax

[Crivella]

Concerning the points of "re-usability" of deps I am unsure what is the best way.
It is possible to do so, but if we wanted to do things properly, every bundle should never define more than 1 package (or 1 package + its optional subpackages if those are never used independently).

The reason being that if 2 packages define the same extension, the compile cache will could work or be broken depending on the load order of the packages from what i've seen (this is why i check for conflicts when rebuilding the manifest).
I think this mainly happens if one package could use the other as an optional dependency, eg: I am working on a package that depends on JLD2 and Flux as deps.
Flux also has JLD2 as an optional dependency. With this approach if JLD2 is not built into Flux (what i am doing right now) if JLD2 is loaded first everything works, but if not i got into an infinite compilation loop which exited after some max-try limit and ended up working in what i assume is an interpreted-only mode

The old attempted route of just using the dependencies just as a source of information for the packages to install (but build a completely new and isolated environment) also has downsides:

• Bundle should still be "1-package" only or anything using it as a dependency will also bring in potentially un-needed extra packages
• I think weird think could still happen also in this scenario with compile caches depending on load order

EDIT: added a PR for what i have concerning the aforementioned packages

• {ai}[system/system] Flux v0.16.10 w/ Julia 1.12.2 easybuild-easyconfigs#26214

Pinging also @lexming since he was the original author of the easyblock to get feedback on these points and the new EB in general

[Flamefire]

every bundle should never define more than 1 package

Why would we need a bundle then? Isn't this effectively saying: Use JuliaPackage?

[Crivella]

Why would we need a bundle then? Isn't this effectively saying: Use JuliaPackage?

There are some packages that have optional features that are enabled by installing a subpackage. In that case i expected them to always be bundable without problems.

Indeed i think the proper approach would be to only have Package, even in Python we sometime encounter trouble when extensions of one package shadow one of another which depending on version might or might not work

[Flamefire]

Indeed i think the proper approach would be to only have Package, even in Python we sometime encounter trouble when extensions of one package shadow one of another which depending on version might or might not work

I mentioned something like that in Slack recently as it seems like PythonBundle is/should be preferred but IMO it makes it too easy to add additional packages that might silently conflict with others. The suggestion was to enhance the CI checks to catch that.
So might be worth to discuss this in a larger group

[ocaisa]

@lexming Your feedback on this would be very valuable

[Flamefire]

It seems odd that it is set outside the if, but only used in one branch.
Besides that this will likely be fully ignored. Maybe you meant to add this as a sanity check command only?
And why isn't it done for non-extensions, i.e. plain JuliaPackages?

[Crivella]

I do not understand what you mean, in the case of a Bundle the sanity_check_step is handling setting it's own exts_filter in case it is an extension or just checking for the expected dir if it is the main bundle.

Do not remember if i tested it for simple packages, but with this, in the same spirit as python we should always use bundles

[Flamefire]

You set exts_filter outside the if but only use it inside. So why not set it inside, closer to where it is used?

For non-extensions I'm quite sure exts_filter won't be used so you need the custom_cmds above. See comment there

in the same spirit as python we should always use bundles

I still think this is counter-productive as bundles for a single package are awkward to write and invite adding more packages that would better be suited for own easyconfigs.
I'm referring to both syntax by having to put everything in a dict with lots of extra quoting and the restrictions on what extensions can do, e.g. single-source only

[Crivella]

You set exts_filter outside the if but only use it inside. So why not set it inside, closer to where it is used?

Can agree on this (same point as the other thread and solved)

[Crivella]

I still think this is counter-productive as bundles for a single package are awkward to write and invite adding more packages that would better be suited for own easyconfigs.

I mean that is a decision that was taken more broadly, do not think this is the right venue to try and change this

[Flamefire]

Will still keep using lists, i think it is more readable than concatenating strings with the risk of missing spaces and such

What about using custom_commands for that? See my earlier comment on possibly using exts_filter from the extension instead of the parent for filtering. Then it would become an issue and I'd say it is cleaner anyway to separate the "does it exist" from the "does it work" check.

I mean that is a decision that was taken more broadly, do not think this is the right venue to try and change this

Not sure where the right venue is but I'd like to bring that up before applying it to other Bundle-Package-changes than already is

[Crivella]

Was not even that aware that exts_filter was called so because of the --skip (even then still bad name), but for what i can see it is in general used to define the sanity check to be performed on an extension.

What about using custom_commands for that?

I think twice you refered to a wrong quote so i am not sure i am getting what you mean.

I think the problem is that we do not want to run the commands themselves on the Bundle.

We either split the sanity check for bundles in JuliaBundle and for Package only in JuliaPackage or i do not see this working. That is not just trivial refactoring...

[Flamefire]

but for what i can see it is in general used to define the sanity check to be performed on an extension.

I guess it was an after-thought to use it as part of the sanity check too, which made sense when the definition is/was(?) basically: Check if this package is installed

I think twice you refered to a wrong quote so i am not sure i am getting what you mean.

What I meant: Have exts_filter be the simple existence check (julia -e ...), which (I think) it was supposed to be, and move the other command(s) that checks for correctness to custom_commands. This way you don't have to care about how separating and joining commands and it is cleaner, to me at least.

I think the problem is that we do not want to run the commands themselves on the Bundle.

If I see this correctly for extensions you run _COMPILECACHE_CHECK once, otherwise for each Julia package in all dependencies.
That's what you do right now: The bundle runs it through custom_commands already, the package and extension through exts_filter
If all you change is moving exts_filter.append(_COMPILECACHE_CHECK % {'ext_name': self.name, 'grep_loc': self.name}) to custom_commands instead and always add that, the behavior will still be the same, won't it?

I would even set cfg[exts_filter] in the constructor, so it can be changed by derived classes and later be used instead of the parent/bundle easyconfig for --skip (if we do that change at some point)

[Crivella]

I have a feeling that for this to work properly

https://github.com/easybuilders/easybuild-framework/blob/a464f4fa70b8df3bb4f54976d31f0aae199923cb/easybuild/framework/extensioneasyblock.py#L191
should really be a elif not self.is_extension.

Not sure why we would run the Extension sanity check on the bundle.
This combined with a

        if not self.is_extension:
            self.sanity_check_load_module()

at the start of JuliaPackage sanity check would solve the issue.

Not sure though if the first part of the fix would have unforseen consequences, the way code for bundles is written is way too entangled for me to predict (and at the very least i would not block this PR for this since it can be added as a feature later).

[Flamefire]

I now think this approach might not be the best: The major issue is that a JuliaBundle needs to be a Julia package or things fail at least in the sanity check. I'm not fully sure if this is really inherited but IIRC Bundles cannot have sources themselves, so that doesn't make sense to treat it as a package.
Hence I think the better approach is to make JuliaBundle inherit only from Bundle similar to the other bundles like PythonBundle.
To avoid duplicated work I imagine that it is possible to do it once in the Bundle and disable it in the extensions.

Because unless we enforce that JuliaPackages are only ever part of a JuliaBundle then relying on self.is_extension to decide what to do is very fragile and may break at least standalone installations or when part of other easyconfigs.
E.g. I recently had a similar use case: Install some software with CMakeMake and then the Python wrappers via exts_list. I imagine this to be a use case for Julia too

[Crivella]

I now think this approach might not be the best:

So the main reason is that this was how the package was structured in the beginning so when i started making changes i tried to keep the original structure.

If the only problem is using exts_filters to enable --skip that is a non issue since we have to delete the installdir before we start an installation (the old way of installing them all 1by1 is tens of times slower than doing them all at once).
Also the mechanism of installing them all at once would become much more difficult to handle since that would need to be implemented twice (once for packages trough deps and once for bundles).

E.g. I recently had a similar use case: Install some software with CMakeMake and then the Python wrappers via exts_list. I imagine this to be a use case for Julia too
Probably not impossible but much less likely for a Julia package to use external libraries (usually they compile everything within Julia)

I'd be more in favor of hiding/disallowing to make use of JuliaPackage directly than do double work on the easyblocks especially since it is policy in EB to prefer bundles over single packages.
Probably considering how little is happening in JuliaBundle itself we could just move that extra part in JuliaPackage and rename it to JuliaBundle

[Flamefire]

How does this not cause an error in Julia <1.5? Shouldn't you at least only error when online == True?
Also download_pkg_deps doesn't avoid this error: If set to True the error will still happen when passed to this function. And pkgs_to_test_online also triggers this error unconditionally and without need.
And maybe also mention test_online

[Flamefire]

if the only problem is using exts_filters to enable --skip

It is not only that. Also 'dirs': [os.path.join('packages', self.name)], won't be checked for standalone packages anymore. Maybe there are other related changes I missed

I'd be more in favor of hiding/disallowing to make use of JuliaPackage directly than do double work on the easyblocks

It could be a shared method. Or if the main point is doing a single "install cmd" on the last extension that could be something done by the Bundle: Go over the extensions and on the last consecutive JuliaPackage issue an install command on it, e.g. by passing an appropriate parameter to a method of that extensions easyblock

especially since it is policy in EB to prefer bundles over single packages.

Besides that I wanted to discuss that I would not go so far as assuming that every JuliaPackage is part of a JuliaBundle and assuming things will be done there. As Julia allows wrapping C libraries I do expect the use case of having them as extensions to other easyblocks.
Maybe this at least needs to check its assumptions: That the parent is a JuliaBundle and that all extensions are JuliaPackages. Otherwise e.g. the use of is_last_extension will be unreliable.
That would exclude installing Julia wrappers together with their main library in the same module but at least avoids silently wrong behavior.