Redistributed Submesh and MeshHierarchy - #5270
Conversation
Reintroduce firedrake/redist.py and the RedistributedMeshTransfer-based support for redistributing refined/adaptively-refined meshes to avoid empty MPI ranks, threaded through MeshHierarchy, AdaptiveMeshHierarchy, and refine_marked_elements. Split out of pbrubeck/mg-redist (#5215) into its own branch/PR, based on top of that branch with the redistribution support removed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
Is this ready for review? You seem to still be working on it. |
It is ready you can review it, but ideally we should merge #5213 and #5215 first. I'm adding the review suggestions from the meeting, and I have a genuine question: should we introduce a new |
| from pyop2.mpi import MPI | ||
|
|
||
|
|
||
| class RedistributedMeshTransfer: |
There was a problem hiding this comment.
Should we move this to Function.assign()? Maybe we can do it more generalically for Submesh, RestrictedFunctionSpace, and RestributedMesh
There was a problem hiding this comment.
We now can redistribute a Mesh/Submesh. I am only adding support for assign across the submesh and its parent, which is what redistributed multigrid needs.
…oject/firedrake into pbrubeck/mesh-redistribution
This reverts commit 3582bb0.
This reverts commit 3821989.
Redistribution is now a submesh whose parent is the mesh it came from, rather than an ad-hoc `redist` attribute on the mesh. `Submesh` gains a `redistribute` kwarg, valid for `subdomain_id` None (the whole mesh, as needed by MeshHierarchy) and for a proper subdomain, and the resulting topology carries a `submesh_point_sf` mapping the parent's points onto its own. Entity orientations are preserved across the redistribution by inheriting the parent's universal vertex numbering (and, for quadrilaterals, its cell orientations), so reference values remain valid and RT/BDM/N1curl/RTCF/RTCE transfer exactly. `RedistributedMeshTransfer` (and `firedrake/redist.py`) go away in favour of `Function.assign`, which composes the point SF with the two function spaces' sections to move data between the differing DOF layouts. Assembling or interpolating across meshes that do not share a parallel distribution raises `NotImplementedError`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Pablo Brubeck <brubeck@protonmail.com>
The helper mapped a redistributed mesh back to the one carrying the cell maps through the removed `redist` attribute, so it silently returned the redistributed mesh instead. Use `mg.utils.transfer_mesh`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
||
| if redistribute and fmesh.has_empty_rank: | ||
| fmesh = firedrake.Submesh(fmesh, redistribute=True) | ||
| cdm = _unoverlapped_dm(fmesh.topology_dm) |
There was a problem hiding this comment.
we should only call _unoverlapped_dm on the base mesh
Summary
Mesh redistribution is now a
Submeshof the mesh it was redistributed from, rather than an ad-hocredistattribute bolted onto the mesh.Submesh(..., redistribute=True)repartitions instead of inheriting the parent's parallel distribution. It works both forsubdomain_id=None(the whole mesh, which is what a redistributedMeshHierarchyneeds) and for a proper subdomain. The child records asubmesh_point_sfpushing the parent's plex points onto its own, sosubmesh_parent/submesh_ancestorsmake every consumer parent-aware.RedistributedMeshTransferandfiredrake/redist.pyare gone. Data moves between the two DOF layouts throughFunction.assign, which composes the point SF with the two function spaces' PETSc sections into a section SF. This is the first step towards unifying non-matching DOF layouts —RestrictedFunctionSpaceand same-distributionSubmesh— on one mechanism; those two still go through thecell_node_mappermutation for now.NotImplementedError, since entity maps are only defined when the two share a distribution.API
Testing
Redistribution round-trips exactly (≤1e-16) for tri/quad/tet/hex × CG/DG/RT/BDM/N1curl/RTCF/RTCE on 1–3 ranks, including curved (P2) and periodic (DG1) coordinate fields.
tests/firedrake/submeshandtests/firedrake/multigridpass at 1, 2, 3 and 4 ranks.RedistributedMeshTransfer, which this PR removes, was borrowed from @wence-'s unmerged branch main...wence/feature/redist-mg🤖 Generated with Claude Code