ESGF Publisher Workflow

pubflow is a lightweight workflow manager for publishing ESGF datasets from mapfiles. It provides:

• A persistent DuckDB database for tracking campaigns, datasets, files, publication attempts, and archival state.

• A Typer-based command-line interface for managing publication workflows.

• Configurable batch publication, retries, and dry-run support.

• Multiple ESGF publisher configuration profiles for different deployment environments.

• CSV and Grist status export capabilities.

• A decoupled archival workflow for computing centres where the publisher lacks direct write access to the final storage location.

The workflow is designed to keep orchestration concerns separate from the underlying esgpublish application and from the computing centres hosting the published data.

---

Overview

The workflow is divided into registration, publication, status export, and archival stages:

                         Publisher VM

                              |

             +----------------+----------------+

             |                                 |

             v                                 v

        Register                         Publish

             |                                 |

             v                                 v

           DuckDB <---------------- Publication status

             |

       +-----+------+

       |            |

       v            v

     Export       Grist

                    |

                    v

              Dashboard

Archival is deliberately decoupled:

Publisher VM

    |

    v

Generate archive tasks

    |

    v

archive_tasks.csv

    |

    | portable transfer

    v

Computing Centre

    |

    v

bin/archive.py

    |

    v

archive_results.csv

Note: The publisher does not need write access to the final archive filesystem.

---

Installation

The project is a Python application that exposes the pubflow command.

After installing the package in the appropriate environment:

pubflow --help

Main Commands

| Command | Description |

|---|---|

| pubflow register | Register datasets and files from mapfiles |

| pubflow publish | Publish datasets |

| pubflow validate | Validate registered datasets |

| pubflow export | Export database state to CSV |

| pubflow grist | Synchronize workflow status with Grist |

| pubflow archive | Generate portable archive tasks |

Note: Running pubflow without a command displays the version.

---

Configuration

Pubflow uses several configuration files, each with a deliberately separate responsibility.

config/

├── campaigns.yml

└── publisher.yml

~/.esg/

├── esg.yaml.EASTINT

├── esg.yaml.WESTINT

├── esg.yaml.EAST

└── esg.yaml.WEST

Campaign Configuration

Campaigns are defined in:

config/campaigns.yml

A campaign contains the project/activity metadata and the locations required by the workflow:

campaigns:

  tipmip-cnrm:

    project: CMIP6Plus

    activity: TIPMIP

    institution: CNRM-CERFACS

    mapfile_root: /modfs/esgf/topublish/CNRM-CERFACS/.mapfiles

    archive:

      enabled: true

      root: /mnt/scality/WCRP/CMIP6Plus/TIPMIP/CNRM-CERFACS

      depth: experiment_id

Campaigns therefore retain both:

• project

• activity

These fields are also exported to Grist and are used by the dashboard to filter campaigns.

---

Publisher Configuration

Publisher execution is configured separately in:

config/publisher.yml

For example:

publisher:

  executable: esgpublish

  arguments:

    - --no-xarray

  batch:

    size: 50

  execution:

    dry_run: false

  logging:

    directory: /home/esguser/esgf-publisher-workflow/logs

  retry:

    enabled: true

    max_attempts: 3

  mapfile_path_mappings:

    - from: /ccc/work/cont003/cmip6/cmip6

      to: /mnt/tgcc/

esg:

  config:

    active: EAST-int

    profiles:

      EAST-int:

        path: /home/esguser/.esg/esg.yaml.EASTINT

      WEST-int:

        path: /home/esguser/.esg/esg.yaml.WESTINT

      EAST-prod:

        path: /home/esguser/.esg/esg.yaml.EAST

      WEST-prod:

        path: /home/esguser/.esg/esg.yaml.WEST

The publisher configuration intentionally contains only the generic esgpublish execution settings.

The actual ESG publisher configuration is selected through an ESG configuration profile.

When a dataset is published, Pubflow effectively constructs:

esgpublish \

    --no-xarray \

    --config /home/esguser/.esg/esg.yaml.EASTINT \

    --map <mapfile>

This keeps the publisher command configuration independent from the campaign definitions.

ESG Configuration Profiles

The profile system allows the same Pubflow installation to target different ESG publisher environments.

For example:

EAST-int

WEST-int

EAST-prod

WEST-prod

The active profile is selected in publisher.yml:

esg:

  config:

    active: EAST-int

Before publication, Pubflow verifies that:

1. The selected profile exists.

2. The configured ESG configuration file exists.

An invalid profile or missing configuration therefore fails early rather than producing a less useful publisher error.

Note: Pubflow does not currently interpret or modify the contents of the esg.yaml files. They remain configuration files owned by the ESG publisher environment.

---

Authentication and EGI Check-in Tokens

Authentication is deliberately handled by esgpublish.

The ESG publisher stores its authentication token in the location configured by the selected ESG configuration, for example:

~/.esgf-publisher.json

When the token expires, esgpublish can initiate the EGI Check-in authentication flow and display the URL that must be visited to renew authentication.

Pubflow does not currently attempt to manage or renew these tokens.

This is intentional:

• esgpublish remains responsible for authentication.

• Pubflow does not need to understand the token format.

• Authentication behaviour remains consistent with standalone esgpublish.

• Manual renewal remains possible when required.

Operational note: A publication process may pause while waiting for manual authentication. For long-running campaigns, running Pubflow inside tmux or another persistent terminal session is recommended.

---

Database

Pubflow uses DuckDB to maintain persistent workflow state.

The database tracks:

• Campaigns

• Datasets

• Files

• Publication attempts

• Archive status

The schema is defined in:

db/schema.sql

Initialization

Initialize the database with:

python bin/init_db.py

Load campaign definitions with:

python workflow/campaign.py

---

Registration

Registration scans a campaign's mapfile directory and registers datasets and their files in DuckDB.

pubflow register tipmip-cnrm

Example output:

Found 2446 mapfiles

Registered CMIP6Plus.... (12 files)

Registered CMIP6Plus.... (8 files)

...

Completed: 2445 succeeded, 1 failed

Note: Registration does not publish anything.

Registration is therefore safe to run before a publication campaign begins.

---

Publishing

Publish datasets for a campaign:

pubflow publish tipmip-cnrm

Controlled testing

Use a limit when testing:

pubflow publish tipmip-cnrm --limit 10

Publication is performed in batches. The batch size is configured in publisher.yml:

publisher:

  batch:

    size: 50

The executor tracks each publication attempt and updates the corresponding dataset state in DuckDB.

Retries

Retries can be configured in publisher.yml:

publisher:

  retry:

    enabled: true

    max_attempts: 3

Dry Run

The publisher execution mode can be controlled through:

publisher:

  execution:

    dry_run: false

---

Publication Logging

Each publication run receives a unique run identifier and an associated log file.

Run information includes:

• Campaign

• Run ID

• Start time

• End time

• Batch information

• Mapfile path mappings

• Dataset successes

• Dataset failures

• Exit codes

• Error messages

• Publication summary

The configured log directory is:

publisher:

  logging:

    directory: /home/esguser/esgf-publisher-workflow/logs

Publication failures are also persisted in DuckDB and exported to Grist.

---

Validation

Validate registered datasets without triggering publication:

pubflow validate tipmip-cnrm

Validation does not invoke the publication workflow.

---

Export

Export the current database state to CSV:

pubflow export tipmip-cnrm

CSV export is useful for inspecting workflow state outside the Pubflow environment or for downstream processing.

---

Grist Integration

Pubflow can synchronize campaign, dataset, and failure information with Grist:

pubflow grist sync

The Grist document contains three main tables:

Campaigns

Datasets

Failures

Campaigns

The campaign table contains:

| Field | Description |

|---|---|

| campaign | Campaign name |

| project | Project identifier |

| activity | Activity identifier |

| total | Total datasets |

| published | Successfully published datasets |

| failed | Failed datasets |

| pending | Datasets still pending |

The dashboard can therefore filter campaigns by:

• Project

• Activity

This allows campaigns belonging to different projects or activities to be monitored from the same Grist document.

Datasets

| Field | Description |

|---|---|

| dataset_id | Dataset identifier |

| campaign | Associated campaign |

| publication_status | Current publication status |

| last_attempt_status | Status of the last publication attempt |

| finished_at | Timestamp of the last attempt |

| log_file | Associated log file |

Typical publication statuses are:

SUCCESS

FAILED

PENDING

Failures

| Field | Description |

|---|---|

| dataset_id | Dataset identifier |

| campaign | Associated campaign |

| run_id | Publication run identifier |

| started_at | Start timestamp |

| finished_at | Finish timestamp |

| status | Publication status |

| exit_code | Publisher exit code |

| log_file | Associated log file |

| error_message | Error information |

Note: Grist credentials are supplied via environment variables and are not stored in the repository.

Grist Dashboard

The Grist document contains a custom dashboard widget providing:

• Overall dataset KPIs

• Published/failed/pending counts

• Campaign-level publication progress

• Project filtering

• Activity filtering

• Campaign progress bars

The dashboard is intended as an operational view of the DuckDB workflow state rather than as a replacement for the database.

---

Archival Workflow

Archival is separated from publication.

The publisher VM may lack write access to the final storage location. Instead, Pubflow generates a portable list of archive operations.

Eligibility

A dataset becomes eligible for archival when:

publication_status = SUCCESS

archive_status = PENDING

Generate Archive Tasks

pubflow archive tipmip-cnrm

Use a limit for testing:

pubflow archive tipmip-cnrm --limit 10

The command generates a CSV containing:

| Column | Description |

|---|---|

| dataset_id | Dataset identifier |

| mapfile | Path to the mapfile |

| archive_path | Destination archive path |

Example:

dataset_id,mapfile,archive_path

CMIP6Plus.TIPMIP.CNRM-CERFACS.CNRM-ESM2-1.esm-piControl.r1i1p2f2.AERmon.cdnc.gr.v20231218,/modfs/esgf/topublish/CNRM-CERFACS/.mapfiles/....map,/mnt/scality/WCRP/CMIP6Plus/TIPMIP/CNRM-CERFACS/CNRM-ESM2-1/esm-piControl/.mapfiles/....map

Note: The archive path is dynamically generated from the dataset's DRS and the campaign's configured archive depth.

---

Archive Depth

The archive.depth setting determines how deep the archive destination should be created within the project's DRS hierarchy.

For example:

archive:

  depth: experiment_id

can produce:

/mnt/scality/WCRP/CMIP6Plus/TIPMIP/CNRM-CERFACS/

    CNRM-ESM2-1/

        esm-piControl/

            .mapfiles/

The DRS is interpreted using esgvoc.

The workflow does not hard-code project-specific DRS fields, allowing different projects to use their own vocabulary and DRS definitions.

---

DRS Handling

The workflow uses esgvoc for DRS interpretation.

For example, for the cmip6plus project, ESGVOC provides DRS components including:

• mip_era

• activity_id

• institution_id

• source_id

• experiment_id

• member_id

• table_id

• variable_id

• grid_label

• version

Note: Pubflow does not contain a CMIP6Plus-specific hard-coded parse_drs() implementation. This allows different projects to use their own vocabulary and DRS definitions.

---

Standalone Archive Executor

The generated archive task CSV is designed to be portable.

It can be transferred to a computing centre and executed independently:

python bin/archive.py archive_tasks.csv

Requirements

The archive executor does not require:

• DuckDB

• Grist credentials

• Publisher credentials

• Access to the publisher VM

• The Pubflow workflow database

It reads the task CSV and copies each mapfile to its specified archive destination.

Results CSV

Request a separate results CSV with:

python bin/archive.py archive_tasks.csv \

    --results archive_results.csv

Result format:

| Column | Description |

|---|---|

| dataset_id | Dataset identifier |

| mapfile | Path to the mapfile |

| archive_path | Destination archive path |

| status | Status of the operation |

| error_message | Error message, if any |

Example:

TEST.DATASET,/source/test.map,/archive/.mapfiles/test.map,SUCCESS,

The current executor uses Python's shutil.copy2() and creates the destination directory when necessary.

---

Repository Structure

esgf-publisher-workflow/

│

├── config/

│   ├── campaigns.yml

│   └── publisher.yml

│

├── db/

│   └── schema.sql

│

├── logs/

│

├── pubflow/

│   └── cli.py

│

├── workflow/

│   ├── archive.py

│   ├── campaign.py

│   ├── config.py

│   ├── database.py

│   ├── exporter.py

│   ├── executor.py

│   ├── grist.py

│   └── registry.py

│

├── bin/

│   ├── init_db.py

│   ├── publisher.py

│   └── archive.py

│

└── pyproject.toml

Note: The exact contents may evolve as the workflow develops.

---

Environment Variables

External credentials and connection information are supplied via environment variables.

Grist

export GRIST_BASE_URL=...

export GRIST_API_KEY=...

export GRIST_DOC_ID=...

These variables should not be committed to the repository.

For permanent local configuration, they can be added to the appropriate shell configuration, such as:

~/.bashrc

---

Development Philosophy

Pubflow intentionally separates responsibilities between the workflow manager, ESG publisher, and computing centre.

Pubflow

Pubflow is responsible for:

• Campaign configuration

• Dataset registration

• Publication orchestration

• Batch management

• Retry handling

• Publication state tracking

• Logging

• Status export

• Grist synchronisation

• Archive task generation

• ESG publisher profile selection

esgpublish

The ESG publisher remains responsible for:

• Dataset extraction

• ESG metadata generation

• ESG publication

• STAC interaction

• EGI Check-in authentication

• Token management

Computing Centre

The computing centre is responsible for:

• Executing archive tasks

• Writing mapfiles to the final archive location

• Returning archive results

This separation keeps Pubflow lightweight and avoids duplicating functionality already provided by the ESG publisher.

---

Current Status

Implemented Features

• Campaign configuration

• Project/activity metadata

• DuckDB database

• Dataset registration

• Publication workflow

• Batch publication

• Publication retries

• Dry-run configuration

• Publication logging

• Validation workflow

• CSV export

• Grist synchronisation

• Grist operational dashboard

• Project/activity filtering in Grist

• pubflow CLI

• ESGVOC-based DRS parsing

• Configurable archive depth

• Portable archive task generation

• Standalone archive executor

• Archive result CSV

• Multiple ESG publisher configuration profiles

• Runtime --config selection for esgpublish

• Publisher-side mapfile path mappings

---

Planned Improvements

Potential future improvements include:

• Safe handling of existing archive destination files

• Resumable archive operations

• Explicit archive conflict detection

• Import of archive results into the workflow database

• More detailed Grist campaign/run visualisations

• Additional operational monitoring for long-running publication processes

• Optional inspection of selected ESG publisher configuration values

Authentication/token management is not currently planned for Pubflow, as this functionality is already handled by esgpublish.

---

Typical Workflow

A standard publication campaign follows these steps:

# Initialize the database

python bin/init_db.py

# Load campaign definitions

python workflow/campaign.py

# Register datasets

pubflow register tipmip-cnrm

# Optionally validate datasets

pubflow validate tipmip-cnrm

# Test a small publication batch

pubflow publish tipmip-cnrm --limit 10

# Publish the campaign

pubflow publish tipmip-cnrm

# Export status

pubflow export tipmip-cnrm

# Synchronize status to Grist

pubflow grist sync

# Generate archive tasks

pubflow archive tipmip-cnrm

# Transfer archive_tasks.csv to the computing centre

# Execute archive tasks there

python bin/archive.py archive_tasks.csv \

    --results archive_results.csv

For long-running publication campaigns, running Pubflow inside a persistent terminal session such as tmux is recommended.