Skip to content

Add pip-installable wheels for Linux and macOS - #151

Open
HugoFara wants to merge 3 commits into
forefireAPI:masterfrom
HugoFara:feat/python-wheels
Open

Add pip-installable wheels for Linux and macOS#151
HugoFara wants to merge 3 commits into
forefireAPI:masterfrom
HugoFara:feat/python-wheels

Conversation

@HugoFara

@HugoFara HugoFara commented Jul 28, 2026

Copy link
Copy Markdown

Closes #149.

Makes pip install forefire work. Wheels for Linux and macOS ship the pyforefire module, the forefire command-line interpreter, and a vendored NetCDF. No NetCDF install, no NETCDF_HOME, no prior CMake build.

pip install forefire
forefire -v
python -c "import pyforefire; print(pyforefire.ForeFire())"

The PyPI distribution is named forefire, as the issue asked. The import name stays pyforefire, so existing scripts keep working.

The three obstacles from @antonio-leblanc's reply

1. setup.py assumed a pre-built libforefireL.

Replaced with scikit-build-core, which drives the existing CMakeLists.txt. The wheel build now compiles the C++ core itself. Packaging moved to the repository root, because an sdist rooted at bindings/python/ cannot reach ../../src. bindings/python/setup.py and bindings/python/pyproject.toml are deleted.

2. NetCDF vendoring.

On Linux, tools/devops/install-netcdf-manylinux.sh installs NetCDF C and HDF5 from EPEL inside the manylinux_2_28 container. It then builds netcdf-cxx4 from source with a pinned checksum, since no RHEL-family package provides the C++4 API. auditwheel copies the result into the wheel. macOS uses Homebrew plus delocate.

3. The MPI axis (#102).

Wheel builds set FOREFIRE_ENABLE_MPI=OFF explicitly. A published wheel can no longer depend on whatever happened to be installed on the build machine. Users who need coupling build from source with FOREFIRE_ENABLE_MPI=ON. This is now documented rather than implicit.

CMake changes

The build is option-driven. Defaults preserve the previous behaviour for ordinary source builds:

Option Source build Wheel
FOREFIRE_ENABLE_MPI ON OFF
FOREFIRE_NATIVE_ARCH ON OFF
FOREFIRE_BUILD_PYTHON OFF ON
FOREFIRE_STATIC_CORE OFF ON
FOREFIRE_BUILD_TOOLS ON OFF
FOREFIRE_CHECK_LFS ON OFF

Also changed:

  • NetCDF is located with find_path and find_library, replacing bare link_libraries(netcdf netcdf_c++4). This is what makes the macOS wheel possible. Homebrew names the library libnetcdf-cxx4, not libnetcdf_c++4, and /opt/homebrew is not on the default search path on Apple Silicon. A missing NetCDF now fails at configure time with install instructions, instead of failing at link time.
  • The version is parsed from src/include/Version.h. project(VERSION), CPack and the wheel version now track the file tools/increment-version.sh edits. project() previously said 1.0 while the code said v2.3.0.
  • Four duplicated if(NETCDF_STATIC_LIBS)/if(MPI_FOUND) link blocks collapse into one forefire_netcdf interface target.
  • Writes into the source tree (bin/, lib/), the LFS check and CPack are skipped under SKBUILD only. Native builds are unaffected.

Two portability bugs fixed along the way

  • -march=native was unconditional. Any binary built on one machine and run on an older CPU dies with SIGILL. That includes the published Docker image, so the Dockerfile now passes -DFOREFIRE_NATIVE_ARCH=OFF. Native builds keep the flag on by default.
  • The Dockerfile built its wheel from ./bindings/python, which is no longer a project. It now builds from the root and takes the forefire binary from that single wheel, rather than copying a second one.

CI

New .github/workflows/wheels.yml, running on pull requests and pushes to master:

  • wheels via cibuildwheel on ubuntu-latest (x86_64), ubuntu-24.04-arm (aarch64, native runner, no QEMU), macos-14 (arm64) and macos-13 (x86_64), for CPython 3.9 to 3.13.
  • sdist, compiled and smoke-tested against Ubuntu's NetCDF packages. This is the path a Windows or musl user takes.
  • publish to PyPI on GitHub release, via trusted publishing.

Every wheel is smoke-tested before upload. tests/python/test_wheel.py imports the module, runs an isotropic 1000 s simulation, and asserts the front grew. forefire -v then checks the shipped interpreter.

Verification

Built in a real manylinux_2_28_x86_64 container with podman, then installed and tested on a Fedora 44 host with no NetCDF installed at all. That second step is the meaningful one: the test machine is not the build machine, and
glibc differs (2.43 against 2.28).

Check Result
Wheel size 8.8 MB, full NetCDF/HDF5 stack in forefire.libs/
ldd on the extension Only librt and libresolv from the system
Smoke test 122 fire nodes
forefire -v from any directory v2.3.0
Repository example tests/runff/run.ff Exit 0, valid GeoJSON MultiPolygon

That last row runs a full Rothermel simulation over the 4 MB tests/runff/data.nc, so the vendored HDF5 and NetCDF are genuinely exercised rather than merely loaded.

The plain cmake -S . -B build path is unchanged: shared lib/libforefireL.so, bin/forefire and bin/ANN_test, -march=native still on, LFS check still running. The native binary and the wheel produce byte-identical output on run.ff. The sdist is 402 KB and contains CMakeLists.txt, which .gitignore had been silently excluding.

Two bugs were caught by that local run rather than by review, which is why the smoke test now asserts on them:

  1. Linking the core as a static archive dropped every self-registering propagation model. addLayer("propagation", "Iso", ...) printed model Iso is not recognized, and the next call segfaulted. Fixed with an OBJECT library. tests/python/test_wheel.py now fails loudly instead of crashing.
  2. The static core needed Threads::Threads linked explicitly. The shared library had been providing it implicitly.

macOS wheels are not verified locally, I may do it later on. Both the Linux and macOS jobs run forefire -v against the installed wheel. If delocate does not relocate the interpreter binary the way auditwheel does, CI fails rather than publishing something broken. Worth watching on the first run.

Before this can publish

The forefire project on PyPI needs to exist, I'll leave it to you. Until then the workflow builds and tests wheels, and the publish job never runs. It is gated on github.event_name == 'release'.

Packaging choices worth a look

The root pyproject.toml replaces a much smaller bindings/python/pyproject.toml. Some values in it are forced by the toolchain, others are policy. The policy ones:

Choice Was Now Reason
Minimum Python >=3.8 >=3.9 3.8 is end of life, and cibuildwheel 4 no longer builds it. Keeping 3.8 means pinning an older cibuildwheel.
Version source hardcoded 2025.2 parsed from src/include/Version.h, currently 2.3.0 Stops the two drifting apart. It also moves the published series from CalVer to SemVer.
pybind11 runtime dependency build dependency Only needed to compile. Nothing imports it at runtime.
License metadata { file = "LICENSE" } "GPL-3.0-or-later" PEP 639 SPDX. Matches the LICENSE text, which is GPLv3 with the "any later version" clause. Change it if you mean GPL-3.0-only.
OS classifiers OS Independent Linux and MacOS X The wheels are POSIX only.
Homepage the GitHub repo forefire.univ-corse.fr Documentation and Issues links added alongside it.
sdist contents not applicable tests/, docs/, paper/ excluded, smoke test kept Keeps the sdist at 402 KB rather than carrying roughly 60 MB of LFS fixtures. An sdist therefore cannot run the full suite.
Wheel targets not applicable PyPy, musllinux, i686 and win32 skipped No usable NetCDF C++4 on any of them.
macOS floor not applicable 14.0 on arm64, 13.0 on x86_64 Must match the macOS version the runner's Homebrew bottles were built for. Apple Silicon wheels therefore require macOS 14.

Two suggestions rather than changes:

  • No Development Status classifier is set. 5 - Production/Stable would fit v2.3.0 and the JOSS paper, if you want the maturity advertised.
  • cibuildwheel will build CPython 3.14, but the classifiers stop at 3.13. Either add the 3.14 classifier or skip 3.14 in the skip list.

Notes for reviewers

  • Windows is out of scope. There is no reasonable prebuilt netcdf_c++4 for it, and the sources use <sys/time.h> and popen, so it needs a porting effort rather than a packaging one. Windows users fall back to the sdist.
  • bindings/python/README.md is rewritten around pip install. It is what PyPI renders as the project page.
  • MACOSX_DEPLOYMENT_TARGET lives in the workflow matrix, not pyproject.toml, because the correct value is a property of the runner image rather than of the project. It will need revisiting when these runners are replaced.

HugoFara added 2 commits July 28, 2026 17:38
`pip install forefire` now works, providing both the pyforefire module and
the forefire command-line interpreter with NetCDF vendored in. Closes forefireAPI#149.

Packaging moves to the repository root and is driven by scikit-build-core,
which runs the existing CMake build instead of assuming a pre-built
libforefireL: an sdist rooted at bindings/python cannot reach ../../src.
bindings/python/{setup.py,pyproject.toml} are removed.

CMakeLists.txt becomes option-driven, keeping today's behaviour as the
default for source builds while wheels get portable settings:

  FOREFIRE_ENABLE_MPI, FOREFIRE_NATIVE_ARCH, FOREFIRE_BUILD_PYTHON,
  FOREFIRE_STATIC_CORE, FOREFIRE_BUILD_TOOLS, FOREFIRE_CHECK_LFS

Wheels disable MPI so a published artifact never depends on what happened
to be installed on the builder (see forefireAPI#102), and disable -march=native, which
was previously unconditional and made any redistributed binary — including
the published Docker image — crash with SIGILL on older CPUs.

NetCDF is now located with find_path/find_library behind a single
forefire_netcdf interface target rather than bare link_libraries. This is
what makes a macOS wheel possible: Homebrew names the library
libnetcdf-cxx4, not libnetcdf_c++4, and /opt/homebrew is off the default
search path. Missing NetCDF now fails at configure time with install hints.

The core is an OBJECT library, not STATIC, when linked directly into the
artifacts: propagation and flux models register themselves from static
initialisers that nothing references by name, and a static archive lets the
linker discard them, leaving the model table empty at runtime.

Other changes:
- version is parsed from src/include/Version.h, so project(VERSION), CPack
  and the wheel all track the file tools/increment-version.sh edits
- .gitignore no longer ignores CMakeLists.txt, which had been dropping the
  build file from the sdist
- Dockerfile builds its wheel from the root and installs the interpreter
  from that one wheel instead of copying a second binary
- tests/python/test_wheel.py smoke-tests an installed wheel; every wheel is
  tested before upload
Project maturity is a maintainer's call, not something packaging should
assert. Left unset rather than guessed.
@antonio-leblanc

Copy link
Copy Markdown
Collaborator

Thanks for this @HugoFara — went through it with Claude helping me dig into the CMake/packaging side (I'm honestly not a C++ build expert, so wanted a second pair of eyes on that part specifically). Overall take: solid work — the OBJECT library split, the NetCDF detection covering the Homebrew naming difference, single-source-of-truth versioning off Version.h — all checks out.

One thing before merging: Wheels (arm64 on macos-14) is currently failing in CI. Not a code bug — it's a version mismatch. The Homebrew NetCDF/HDF5/zstd/libaec libs on the macos-14 runner are built targeting a macOS 14.0 minimum, but the wheel gets tagged macosx_11_0_arm64, so delocate-wheel correctly refuses to repair it. Full log: https://github.com/forefireAPI/forefire/actions/runs/30405503029/job/91247326062

Simplest fix is probably setting MACOSX_DEPLOYMENT_TARGET=14.0 for that cibuildwheel job (or statically linking NetCDF for that platform). Shouldn't be a big lift.

I'm traveling this week with limited bandwidth, back around Aug 10th. Happy to hop on a call then to close this out — or if you get a chance to fix the macOS target in the meantime, I'll do a final pass and merge once CI's green. Either works for me.

The macos-14 arm64 wheel failed to repair: every Homebrew bottle on that
runner (netcdf, hdf5, netcdf-cxx4, zstd, libaec, libsz) has a minimum
target of 14.0, so delocate refused to relocate them into a wheel tagged
for 13.0.

The right value follows the runner image rather than the project, so it
moves from pyproject.toml into the workflow matrix: 14.0 on macos-14,
13.0 on macos-13. Apple Silicon wheels now require macOS 14.
@HugoFara

Copy link
Copy Markdown
Author

Thanks for the review, and for digging into the CMake side.

Fixed in 347a720. One small correction on the diagnosis: MACOSX_DEPLOYMENT_TARGET=13.0 was already being applied. Delocate's error says "Library dependencies do not satisfy target MacOS version 13.0", so it saw the 13.0 setting. The actual problem is that every Homebrew bottle on the macos-14 runner is built for 14.0:

libzstd, libaec, libsz, libnetcdf, libhdf5, libnetcdf-cxx4, libhdf5_hl
  -> each "has a minimum target of 14.0"

So 13.0 was not too low in general, it was too low for that particular runner image. I tried to keep a single source-of-truth number in pyproject.toml, but that will not work for the runner.

Two things though:

  • Apple Silicon wheels now require macOS 14: arm64 users on macOS 11 to 13 fall back to the sdist. If you would rather keep them, the alternative is building NetCDF from source on macOS with a lower target, the way install-netcdf-manylinux.sh already does on Linux. That is a heavier macOS job, so I did not do it unilaterally. Happy to if you prefer the coverage.
  • The x86_64 macOS path is still unproven: In the run you linked, Wheels (x86_64 on macos-13) never actually started, it was still queued when arm64 failed. 13.0 matches that runner's bottles so I expect it to pass, but we will have to see with the runner (unless you can test it).

No rush to merge, enjoy your trip! If you are okay with this PR, we can discuss it through about its next adjustments. I want to keep it minimal for readability, but there are more edits I'd like to make (old NetCDF + unnecessary dependencies in the wheel), so it'll be a good occasion to have a chat about its specific aspects. Let me know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python Wheels

2 participants