Skip to content

Redistributed Submesh and MeshHierarchy - #5270

Open
pbrubeck wants to merge 15 commits into
pbrubeck/mg-redistfrom
pbrubeck/mesh-redistribution
Open

Redistributed Submesh and MeshHierarchy#5270
pbrubeck wants to merge 15 commits into
pbrubeck/mg-redistfrom
pbrubeck/mesh-redistribution

Conversation

@pbrubeck

@pbrubeck pbrubeck commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Mesh redistribution is now a Submesh of the mesh it was redistributed from, rather than an ad-hoc redist attribute bolted onto the mesh.

  • Submesh(..., redistribute=True) repartitions instead of inheriting the parent's parallel distribution. It works both for subdomain_id=None (the whole mesh, which is what a redistributed MeshHierarchy needs) and for a proper subdomain. The child records a submesh_point_sf pushing the parent's plex points onto its own, so submesh_parent/submesh_ancestors make every consumer parent-aware.
  • Entity orientations are preserved. The redistributed mesh inherits the parent's universal vertex numbering, and its cell orientations on quadrilaterals/hexahedra, so cell closures are ordered identically on both sides. Reference values therefore remain valid across the redistribution and RT/BDM/N1curl/RTCF/RTCE transfer exactly, instead of picking up sign and permutation flips.
  • RedistributedMeshTransfer and firedrake/redist.py are gone. Data moves between the two DOF layouts through Function.assign, which composes the point SF with the two function spaces' PETSc sections into a section SF. This is the first step towards unifying non-matching DOF layouts — RestrictedFunctionSpace and same-distribution Submesh — on one mechanism; those two still go through the cell_node_map permutation for now.
  • Assembling or interpolating a redistributed mesh alongside its parent raises NotImplementedError, since entity maps are only defined when the two share a distribution.

API

rmesh = Submesh(mesh, redistribute=True)            # whole mesh, repartitioned
rmesh.submesh_parent is mesh                        # True; mesh.has_empty_rank was why
Function(V_rmesh).assign(u)                         # parent -> redistributed
Function(V_mesh).assign(u_r, allow_missing_dofs=True)   # and back
assemble(inner(u, u) * dx(domain=rmesh))            # NotImplementedError

sub = Submesh(mesh, subdomain_id=1, redistribute=True)  # a proper subdomain too

mh = MeshHierarchy(base, 3)                         # redistribute=True by default
prolong(uc, uf)                                     # transfers on the parent, then assigns

Testing

Redistribution round-trips exactly (≤1e-16) for tri/quad/tet/hex × CG/DG/RT/BDM/N1curl/RTCF/RTCE on 1–3 ranks, including curved (P2) and periodic (DG1) coordinate fields. tests/firedrake/submesh and tests/firedrake/multigrid pass at 1, 2, 3 and 4 ranks.

RedistributedMeshTransfer, which this PR removes, was borrowed from @wence-'s unmerged branch main...wence/feature/redist-mg

🤖 Generated with Claude Code

Reintroduce firedrake/redist.py and the RedistributedMeshTransfer-based
support for redistributing refined/adaptively-refined meshes to avoid
empty MPI ranks, threaded through MeshHierarchy, AdaptiveMeshHierarchy,
and refine_marked_elements.

Split out of pbrubeck/mg-redist (#5215) into its own branch/PR, based
on top of that branch with the redistribution support removed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@pbrubeck
pbrubeck requested a review from connorjward July 21, 2026 17:22
@pbrubeck
pbrubeck marked this pull request as ready for review July 22, 2026 07:17
Comment thread firedrake/mg/mesh.py
Comment thread tests/firedrake/multigrid/test_redist_mesh.py
Comment thread tests/firedrake/multigrid/test_redist_mesh.py
Comment thread tests/firedrake/multigrid/test_redist_mesh.py
@connorjward

Copy link
Copy Markdown
Contributor

Is this ready for review? You seem to still be working on it.

@pbrubeck

Copy link
Copy Markdown
Contributor Author

Is this ready for review? You seem to still be working on it.

It is ready you can review it, but ideally we should merge #5213 and #5215 first.

I'm adding the review suggestions from the meeting, and I have a genuine question: should we introduce a new redistribute kwarg or should we pass it through the distribution_parameters?

Comment thread firedrake/redist.py Outdated
from pyop2.mpi import MPI


class RedistributedMeshTransfer:

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we move this to Function.assign()? Maybe we can do it more generalically for Submesh, RestrictedFunctionSpace, and RestributedMesh

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I think so.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now can redistribute a Mesh/Submesh. I am only adding support for assign across the submesh and its parent, which is what redistributed multigrid needs.

pbrubeck and others added 9 commits July 27, 2026 10:53
…oject/firedrake into pbrubeck/mesh-redistribution
This reverts commit 3582bb0.
This reverts commit 3821989.
Redistribution is now a submesh whose parent is the mesh it came from,
rather than an ad-hoc `redist` attribute on the mesh.

`Submesh` gains a `redistribute` kwarg, valid for `subdomain_id` None
(the whole mesh, as needed by MeshHierarchy) and for a proper subdomain,
and the resulting topology carries a `submesh_point_sf` mapping the
parent's points onto its own.  Entity orientations are preserved across
the redistribution by inheriting the parent's universal vertex numbering
(and, for quadrilaterals, its cell orientations), so reference values
remain valid and RT/BDM/N1curl/RTCF/RTCE transfer exactly.

`RedistributedMeshTransfer` (and `firedrake/redist.py`) go away in favour
of `Function.assign`, which composes the point SF with the two function
spaces' sections to move data between the differing DOF layouts.
Assembling or interpolating across meshes that do not share a parallel
distribution raises `NotImplementedError`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Pablo Brubeck <brubeck@protonmail.com>
@pbrubeck pbrubeck changed the title Mesh redistribution support (split from #5215) Submesh(mesh, redistribute=True) Jul 28, 2026
@pbrubeck pbrubeck changed the title Submesh(mesh, redistribute=True) Redistributed Submesh and MeshHierarchy Jul 28, 2026
pbrubeck and others added 2 commits July 28, 2026 16:39
The helper mapped a redistributed mesh back to the one carrying the cell
maps through the removed `redist` attribute, so it silently returned the
redistributed mesh instead.  Use `mg.utils.transfer_mesh`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread firedrake/mg/mesh.py

if redistribute and fmesh.has_empty_rank:
fmesh = firedrake.Submesh(fmesh, redistribute=True)
cdm = _unoverlapped_dm(fmesh.topology_dm)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should only call _unoverlapped_dm on the base mesh

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants