Fix/reinforce lines voltage issues all segments - #738
Open
joda9 wants to merge 25 commits into
Open
Conversation
Release 0.2.1
- engine() without arguments now auto-detects: egon-data database if a configuration file exists (EGON_DATA_CONFIG or ~/.ssh/egon-data.configuration.yaml), otherwise OEP; choice is logged - configuration file without ssh-tunnel section connects directly instead of raising KeyError - engine_from_settings(): strict source vocabulary (egon-data/oep/ empty), unknown values raise instead of silently falling back - pass ssh_pkey as str — sshtunnel silently ignores pathlib.Path and falls back to default keys, failing gateway auth - SSH keepalive + pool_pre_ping/pool_recycle keep pooled connections usable across long multi-grid runs - pin paramiko < 4.0 (DSSKey removed in 4.x breaks sshtunnel) - conftest: pin test engine to OEP so auto-detection cannot redirect the suite, replace hardcoded home path with default_config_path()
- Main.jl: use the solution whenever the solver proves optimality or reports a feasible primal point; computing an IIS on a feasible model raises Gurobi error 10015 and killed the run. The SOC tightness check now runs on every used primal point (a loose relaxation matters for suboptimal solutions too); termination status is passed through to opf_results.status - pm_optimize: split a non-contiguous time index (e.g. from automatic timestep selection) into contiguous intervals and optimize each separately — storage/heat state does not carry across a gap. Results are merged over the full index, per-interval solve reports land in opf_results.interval_results, infeasible intervals raise after feasible results are stored - from_powermodels: write results only onto the current interval's time steps instead of the whole frame - align overlying-grid SOC series and EV flexibility bands to the analysis time index (year shift + reindex) via new tools.align_series_to_timeindex
egon_etrago_bus/egon_etrago_link live in schema "grid" in a local egon-data database; only the OEP resolves them via the alias mapping keyed on "supply". Heat pump import against a local database failed with UndefinedTable before.
YAML/JSON-driven task pipelines with a single entry point run_edisgo(config): registered tasks with requires/provides metadata, a static pre-run validator, extends-based config merging, and three bundled presets (worst_case, flex_opf, overlying_grid_opf). - flexibilities: top-level carrier list drives both import_flex and the optimize task's flexible selection - timestep selection split into set_timeindex (manual, early) and select_critical_timesteps (auto scoring, late) - database source resolved via engine_from_settings (auto-detect, oep, egon-data) - overlying-grid data via run_edisgo(..., overlying_grid_data=...) from eTraGo or CSVs, incl. new OverlyingGrid attributes - ORM mapping fix for egon-data tables/views without primary key
Cherry-picked from edisgo_run_edisgo (b83c884), which built this on top of dozens of commits this branch doesn't have. Adds spatial_reduce / spatial_restore tasks bracketing optimize, so the OPF can run on a spatially-reduced grid while reinforcement still runs on the full topology, plus the underlying spatial_complexity_reduction algorithm and its tests. Dropped edisgo.EDisGo.run_pipeline() from the same squashed source commit since it calls edisgo.run._run_pipeline_on, which doesn't exist on this branch's simpler run_edisgo() entry point, and nothing in the spatial feature depends on it.
Cherry-picked from edisgo_run_edisgo (dc1249b). Supersedes the task-local reduce_timeseries_data_to_given_timeindex workaround added in the previous commit: get_flexibility_bands now year-aligns and trims its own returned/stored bands to edisgo_obj.timeseries.timeindex, so any caller (not just this pipeline) gets correct bands. Renamed the ported spatial-reduction preset from uc6_spatial_reduction.yaml to spatial_reduction_opf.yaml, and its example runner from run_example_06.py to examples/run_spatial_reduction_example.py, to follow this branch's <name>[_opf].yaml / examples/ convention (worst_case, flex_opf, overlying_grid_opf) instead of edisgo_run_edisgo's ucN_ numbering. Adapted the preset's pipeline to this branch's timestep-selection API: it previously used edisgo_run_edisgo's separate uc5 feature (a dual-mode select_timesteps task switched via position:/mode:), which this branch doesn't have. Hardwired to the manual case the preset already shipped with, via this branch's set_timeindex task (parameters passed as step params, not via a top-level timeseries_selection block); the post_grid auto-selection step was dropped since it would have no-opped here anyway.
get_database_alias_dictionaries built a fresh edisgo.io.db.engine() with no arguments to read the data.edut_00 name-mapping table. With no arguments that call is in auto-detect mode: in any environment that has an egon-data configuration file (e.g. ~/.ssh/egon-data.configuration.yaml) it resolves to the local egon-data database and opens an SSH tunnel, instead of the OEP the caller is actually importing from. data.edut_00 only exists on the OEP, so the lookup then failed with NoSuchTableError: edut_00 against the local database — breaking every *_oedb code path (all OEDB tests) even when a valid OEP engine/token was passed in. The stray auto-detected tunnel is also what produced the endless paramiko "keepalive@lag.net" logging errors, since it is never closed. Thread the engine that import_tables_from_oep already holds (guaranteed to be the OEP engine inside the "openenergyplatform" branch) through _ensure_db_mappings_loaded to get_database_alias_dictionaries, so the alias dictionary is read from the same OEP connection as the mapped tables. Engine() auto-detect remains only as the fallback when no engine is passed.
…handling Two fixes surfaced by running the temporally + spatially reduced OPF (overlying_grid_opf_spatial) through the full eGo/eTraGo chain: - powermodels_io.from_powermodels: the HV-requirement relative-error calculation (opf_version 3/4) compared the per-interval result frame (indexed by the interval's timeindex) against hv_flex_dict built over the full reduced index, raising a shape broadcast error (e.g. 168 vs 336) for any multi-interval run. Align the requirement series to the result frame's index (.loc[df2.index]); a no-op for a single full run. - Main.jl: broaden has_solution to also accept NEARLY_FEASIBLE_POINT (numerically loose but valid primal points), and wrap compute_conflict! (IIS) in a try/catch so a feasible-but-misreported model no longer hard- crashes with "Gurobi Error 10015: Cannot compute IIS on a feasible model".
Combined "full" use case: overlying_grid_opf (egon time series, overlying- grid requirements, automatic critical-timestep selection = temporal reduction, powermodels OPF) PLUS the spatial_reduce / spatial_restore bracket around optimize, so the OPF runs on a spatially reduced grid while reinforcement runs on the full topology. Both reductions are togglable.
ssh_tunnel() started an SSHTunnelForwarder but only returned its local port — the forwarder was never referenced again, so the tunnel and its keepalive daemon thread leaked for the whole life of the process and were never stopped. Register an atexit hook so every tunnel is torn down cleanly at process exit (the tunnel still stays open during the run, as the engine is cached and reused across tasks/grids). Also raise the paramiko.transport logger to WARNING when the tunnel is created: paramiko logs every keepalive request at DEBUG from that daemon thread, which — before this cleanup — could fire after pytest/interpreter had closed the log streams, flooding the end of a test run with harmless "ValueError: I/O operation on closed file" logging errors.
- spatial_reduction_opf.yaml: replace the hardcoded /home/gurobi overlying
path with a placeholder, and fix the database source ("local" is not a
valid pipeline source — use "egon-data") so the shipped preset works.
- run_spatial_reduction_example.py: use placeholder paths instead of a
developer's local ding0/overlying directories.
- drop CONTEXT.md and docs_notes/ (internal design/grilling notes, one of
which leaked a local repo path) — not intended for publication.
Feat/pipeline runner
edisgo/opf/eDisGo_OPF.jl was missing from every built artifact, so pm_optimize() could never work in a non-editable installation. It fails at powermodels_opf.py:415, which launches julia with an absolute path derived from __file__ -- fine in a source checkout, a missing file in site-packages. This has been broken since at least v0.2.1, so the OPF has never run from a released eDisGo. It matters now because eGo pulls eDisGo as "edisgo @ git+https://github.com/openego/eDisGo.git@dev", which pip builds into a real wheel. package_data patterns are resolved relative to the package they are listed under, and edisgo/opf is a package of its own (unlike edisgo/config, edisgo/equipment and edisgo/run/presets, which are plain data directories inside the edisgo package). The Julia tree therefore needs its own "edisgo.opf" key; it was unreachable from the existing "edisgo" entry. Nobody noticed because everything installs editable during development and the test workflow installs from the checkout, never from a built artifact. So add a Packaging workflow that builds both artifacts and checks them: - .github/scripts/check_packaged_data.py asserts that the Julia sources, the config files, the equipment tables and the run presets are all present in the wheel and the sdist. Runnable locally before a release. It also warns about the egg-info trap: a stale SOURCES.txt keeps packing files from earlier builds, which makes a broken sdist look complete. - the workflow additionally installs the wheel into a fresh venv and resolves Main.jl the way powermodels_opf.py does at runtime, asserting the directory is writable too, since Main.jl runs Pkg.instantiate() in place. Verified: without the package_data key the check fails on both artifacts (0 of 16 Julia files); with it the wheel and sdist carry all 16, and the installed layout resolves Main.jl.
The OEDB tests hit the live OEP, which intermittently returns 503/504
(oedialect ConnectionException) or refuses the connection outright
(requests ConnectionError). These transient outages fail otherwise-green
CI runs.
- add pytest-rerunfailures to the Linux basic + coverage install steps.
It is declared in setup.py's dev extras, which the Linux jobs do not
install; the Windows job gets it through eDisGo_env_dev.yml, which pins
"-e .[dev]".
- retry only network failures: --reruns 2 --reruns-delay 15
--only-rerun "Connection(Error|Exception)|Timeout", so genuine
assertion/logic failures still fail immediately. Only two tests are
marked runonlinux, so the OEP-hitting tests run on Windows as well and
want the same treatment.
The Windows invocation stays on a single line. That step runs in
PowerShell (the job sets no `defaults: shell:`), where "\" is not a line
continuation, so a wrapped command aborts before pytest even starts:
ParserError: Missing expression after unary operator '--'.
…0.3.0 The changelog stopped at the documentation restructuring and covered none of the work merged since: the whole edisgo.run pipeline framework (#731), the spatial complexity reduction integration (#706), the OPF rework towards feasible non-optimal solutions and per-interval solving, the reworked database access, the get_flexibility_bands time-index scoping and the SPDX license headers (#607). Also sets the release date.
- setup.py: version 0.3.0 (was 0.3.0dev, which normalised to 0.3.0.dev0), and fill in the metadata PyPI has been missing entirely: python_requires (>=3.10,<3.13 -- numpy is pinned to 1.26.4, which has no 3.13 wheels, and the test matrix covers 3.10 to 3.12), trove classifiers and project_urls. - authors: Jonas Danke and Moritz Schlösser are maintaining eDisGo now, so they lead the author field and are named as maintainers; the previous contributors stay listed. Maike Held was missing from it entirely despite being the second-largest contributor. - AUTHORS.md and CITATION.cff are new. Neither existed, so there was no complete credit list anywhere outside the git history, and GitHub had nothing to render for "Cite this repository". - .mailmap consolidates the identities people committed under: git counted 43 distinct authors for 19 actual people. Two commits were authored with Jonas' name but Kilian's address, so that entry matches on both fields rather than reassigning all of Kilian's commits. - doc/conf.py: extend the stale copyright year. Verified: twine check passes on both artifacts, the wheel METADATA carries the new author/maintainer/classifiers/Requires-Python, and CITATION.cff parses with all required CFF 1.2.0 fields present.
Co-authored-by: Moritz Schloesser <moritz.schloesser@rl-institut.de> Co-authored-by: joda9 <66819219+joda9@users.noreply.github.com>
The three RLI thesis PDFs had gone missing from the website and were restored from backup under year-based upload paths (2021/09, 2022/09, 2023/09). The old 2024/09 paths now return 404, failing the docs linkcheck CI.
* docs(whatsnew): document the run pipeline, OPF and database work for 0.3.0 The changelog stopped at the documentation restructuring and covered none of the work merged since: the whole edisgo.run pipeline framework (#731), the spatial complexity reduction integration (#706), the OPF rework towards feasible non-optimal solutions and per-interval solving, the reworked database access, the get_flexibility_bands time-index scoping and the SPDX license headers (#607). Also sets the release date. * chore(release): 0.3.0 metadata, authors and citation info - setup.py: version 0.3.0 (was 0.3.0dev, which normalised to 0.3.0.dev0), and fill in the metadata PyPI has been missing entirely: python_requires (>=3.10,<3.13 -- numpy is pinned to 1.26.4, which has no 3.13 wheels, and the test matrix covers 3.10 to 3.12), trove classifiers and project_urls. - authors: Jonas Danke and Moritz Schlösser are maintaining eDisGo now, so they lead the author field and are named as maintainers; the previous contributors stay listed. Maike Held was missing from it entirely despite being the second-largest contributor. - AUTHORS.md and CITATION.cff are new. Neither existed, so there was no complete credit list anywhere outside the git history, and GitHub had nothing to render for "Cite this repository". - .mailmap consolidates the identities people committed under: git counted 43 distinct authors for 19 actual people. Two commits were authored with Jonas' name but Kilian's address, so that entry matches on both fields rather than reassigning all of Kilian's commits. - doc/conf.py: extend the stale copyright year. Verified: twine check passes on both artifacts, the wheel METADATA carries the new author/maintainer/classifiers/Requires-Python, and CITATION.cff parses with all required CFF 1.2.0 fields present. * Update eDisGo_OPF.jl Manifest for Julia 1.12 compatibility StaticArrays 1.5.12 (pinned in the old manifest) fails to precompile on Julia >=1.11 due to Base's AbstractTriangular gaining a type parameter. Pkg.update() resolved newer, compatible versions of StaticArrays and its dependents (JuMP, PowerModels, MathOptInterface, Gurobi.jl, etc.). * Bump Julia to 1.12 in CI, compat and install docs The updated eDisGo_OPF.jl Manifest was resolved with Julia 1.12 and pins stdlibs that do not exist in earlier versions (e.g. JuliaSyntaxHighlighting, StyledStrings, LinearAlgebra 1.12). Main.jl runs Pkg.instantiate() on that checked-in manifest, so the CI job pinned to Julia 1.6 could not resolve it and test_pm_optimize failed. Keeping both 1.6 and 1.12 working is not possible with a single checked-in manifest, so the declared Julia version is raised everywhere it appears.
Release v0.3.0
The date stamped in the 0.3.0 metadata commit was a placeholder from before the dev -> master merge landed. Set both CITATION.cff and the changelog to the day the release is actually published, so the tagged tree and the Zenodo citation metadata agree with the PyPI upload date.
config_opf_julia.cfg still pointed at julia-1.1.0/bin while the bundled eDisGo_OPF.jl package targets Julia 1.12, and the configuration reference includes the file verbatim, so the stale path was published in the docs right next to an installation page asking for 1.12. julia_bin has not been read since pm_optimize() started resolving julia from the system PATH, so the reference no longer claims the file points to the binary in use, and the file itself says what it is. Keeping the section rather than dropping the file, so config directories carrying it stay loadable.
reinforce_lines_voltage_issues() determines the node at two thirds of the feeder length (node_2_3) and normally disconnects the line there and reconnects it to the station. If node_2_3 turns out to be the feeder representative - the bus directly connected to the station - no line can be disconnected and the measure falls back to reinforcing lines instead. In that fallback only the single line between the station and the representative was reinforced, not the remaining lines on the path to the critical node. Since the voltage deviation at the critical node is generally dominated by the segments further away from the station, the measure frequently did not change the reported voltage issue at all. Grid reinforcement then repeated the same ineffective measure in every iteration and aborted with MaximumIterationError, typically reporting a residual deviation in the per mille range. LVGrid_5 of the test grid shows this: for a voltage issue at Bus_BranchTee_LVGrid_5_2, node_2_3 is moved back to the representative Bus_BranchTee_LVGrid_5_1. Only Line_50000003 was reinforced, which is 0.56 m long, while Line_50000002 - 30 m of NAYY 4x1x35 causing nearly all of the voltage drop - was left untouched. All lines on the path from the station to the critical node are now reinforced. Feeders consisting of a single line are unaffected. Short feeders are affected most, which is why the error shows up predominantly in grids that went through spatial complexity reduction, where node_2_3 frequently coincides with the representative. Test expectations for the number of equipment changes and grid expansion costs are updated accordingly, as more lines are reinforced per measure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please include a summary of the change and which issue is fixed.
Fixes # (issue)
Type of change
Please delete options that are not relevant.
Checklist:
pre-commithooks