Skip to content

fix: do not cap <IntRangeExpr> expansion at the list-form limit #327

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
leongdl merged 1 commit into from
Aug 14, 2026
Merged

fix: do not cap <IntRangeExpr> expansion at the list-form limit #327

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 12 additions & 29 deletions src/openjd/model/v2023_09/_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -980,9 +980,14 @@ def validate_let_field(value: Any, info: ValidationInfo, *, simple_action: bool
return value


# §3.4: the maximum number of values a task parameter's range may take on.
# Not raised by FEATURE_BUNDLE_1 in 2023-09 (matches openjd-rs's
# EffectiveLimits.max_task_param_range_len).
# §3.4: the maximum number of elements in a task parameter's *list*-form range
# — `<IntRangeList>` (§3.4.1.1), `<FloatRangeList>` (§3.4.1.2) and
# `<StringRangeList>` (§3.4.1.3). Not raised by FEATURE_BUNDLE_1 in 2023-09.
#
# Do not apply this to an `<IntRangeExpr>` expansion. §3.4.1.1.1 constrains that
# form only by "no two ranges may overlap" and states its purpose is to express
# frame ranges succinctly, so capping the expansion rejects the form's primary
# use case and pre-empts the host service's own task-count limits.

@github-actions github-actions Bot Aug 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the expansion cap leaves an unbounded materialization on the CHUNK[INT] + range-expression path. _MAX_TASK_PARAM_RANGE_LEN was the only thing bounding it.

_step_param_space_iter.py:291-294 expands a CHUNK[INT] range into a real Python list, and then set(...) of it (lines 333, 968, 1049):

if isinstance(parameter.range, list):
    parameter_range: list[int] = [int(v) for v in parameter.range]
else:
    parameter_range = list[int](parameter.range)   # IntRangeExpr -> full expansion

So a template with

- name: Frames
  type: "CHUNK[INT]"
  range: "1-100000000"
  chunks: { defaultTaskCount: 100 }

now parses cleanly (IntRangeExpr.from_str is O(1) — it stores range objects), and any consumer that later builds a StepParameterSpaceIterator allocates ~100M ints in a list plus a set — multiple GB — before it can reject anything. Previously _check_range_expr_len rejected this at parse time. For a library that validates untrusted job templates, that is a memory-exhaustion vector rather than a validation error.

Note the plain INT range-expression path is fine — RangeExpressionIdentifierNode keeps the IntRangeExpr and iterates lazily, and len() is O(1). The problem is specific to the chunk path (and to the range_set/_range_set construction).

If uncapping the expression form is the intended spec reading, the chunk path probably needs to either stop materializing (chunk boundaries over an IntRangeExpr are computable without expanding it, since __getitem__ is O(log n)) or carry its own explicit, documented bound so the failure is a ValidationError and not an OOM.

@seant-aws seant-aws Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec confirms CHUNK[INT] can absolutely use <IntRangeExpr>

_MAX_TASK_PARAM_RANGE_LEN = 1024


Expand Down Expand Up @@ -1289,26 +1294,6 @@ class RangeExpressionTaskParameterDefinition(OpenJDModel_v2023_09):
# has a value when type is CHUNK[INT], which is only possible from the TASK_CHUNKING extension
chunks: Optional[TaskChunksDefinition] = None

@field_validator("range")
@classmethod
def _validate_range_len(cls, value: Any) -> Any:
# §3.4: a range expression that arrives via format-string resolution
# (e.g. `range: "{{RawParam.Frames}}"` with a RANGE_EXPR parameter) is
# only parsed at instantiation, so the expansion cap must be enforced
# here too — matching openjd-rs's resolve-time checks in create_job.
if isinstance(value, IntRangeExpr):
_check_range_expr_len(value)
return value


def _check_range_expr_len(parsed_range: IntRangeExpr) -> None:
"""§3.4: a range expression may expand to at most 1024 values."""
if len(parsed_range) > _MAX_TASK_PARAM_RANGE_LEN:
raise ValueError(
f"range expression expands to {len(parsed_range)} elements "
f"(max {_MAX_TASK_PARAM_RANGE_LEN})."
)


def _range_task_param_target(model: Any, typed_values: dict) -> Type[OpenJDModel]:
"""``create_as`` target-model selector shared by the INT and CHUNK[INT]
Expand Down Expand Up @@ -1412,21 +1397,19 @@ def _native_element_type_name(elem: Any) -> str:
def _validate_int_range_elements(value: Any) -> Any:
"""Shared ``range`` post-validator for the INT and CHUNK[INT]
task-parameter definitions: a literal range-expression string must parse
and may expand to at most 1024 values (§3.4); a list-form range is
length-capped. Ranges containing format expressions defer to the
RangeExpressionTaskParameterDefinition model once they are resolved.
against the ``<IntRangeExpr>`` grammar; a list-form range is length-capped
(§3.4). The expansion of a range expression is deliberately not capped —
see ``_MAX_TASK_PARAM_RANGE_LEN``.

@github-actions github-actions Bot Aug 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-implementation parity note. The two comments this PR deletes both asserted that the cap matched openjd-rs (EffectiveLimits.max_task_param_range_len, and the resolve-time checks in create_job/ranges.rs). Dropping the check on the Python side without a corresponding change in openjd-rs means a template with range: "1-5000" now validates here but would still be rejected at create_job, so callers using this library as a pre-submission validator get a late, surprising failure instead of an early one.

Worth confirming the openjd-rs side is being changed in step (or that the earlier comments were simply wrong about what it enforces) — otherwise the two implementations disagree on which templates are valid.

@seant-aws seant-aws Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rs changes in progress

"""
if isinstance(value, FormatString):
# If there are no format expressions, we can validate the range expression.
# otherwise we defer to the RangeExressionTaskParameter model when
# they've all been evaluated
if len(value.expressions) == 0:
try:
parsed_range = IntRangeExpr.from_str(value)
IntRangeExpr.from_str(value)
except Exception as e:
raise ValueError(str(e))
# §3.4: the range may take on at most 1024 values.
_check_range_expr_len(parsed_range)
else:
validate_task_param_range_list_len(value)
return value
Expand Down
64 changes: 64 additions & 0 deletions test/openjd/model_v0/v2023_09/test_parameter_space.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
FloatTaskParameterDefinition,
IntTaskParameterDefinition,
PathTaskParameterDefinition,
RangeExpressionTaskParameterDefinition,
RangeListTaskParameterDefinition,
StepParameterSpaceDefinition,
StringTaskParameterDefinition,
)
Expand Down Expand Up @@ -329,6 +331,10 @@ class TestRangeExpressionTaskParameterDefinition:
},
id="format string with multiple",
),
pytest.param(
{"name": "foo", "type": "INT", "range": "1-5000"},
id="expansion past the list-form cap",
),
),
)
def test_parse_success(self, data: dict[str, str]) -> None:
Expand Down Expand Up @@ -375,6 +381,64 @@ def test_parse_fails(self, data: dict[str, Any]) -> None:
assert len(excinfo.value.errors()) > 0


class TestTaskParameterRangeLength:
"""§3.4 caps the number of elements in the *list* forms of a task parameter's
range. §3.4.1.1.1 `<IntRangeExpr>` carries no element cap, so an expression's
expansion must not be capped — the form exists to express frame ranges, which
routinely run to thousands of values.
"""

@pytest.mark.parametrize(
"range_expr,expected_len",
(
pytest.param("1-1024", 1024, id="at the list-form cap"),
pytest.param("1-1025", 1025, id="one past the list-form cap"),
pytest.param("1-5000", 5000, id="ordinary frame range"),
pytest.param("1-100000:2", 50000, id="large range with a step"),
),
)
def test_range_expression_expansion_is_not_capped(
self, range_expr: str, expected_len: int
) -> None:
# WHEN the template-layer definition parses a literal range expression
_parse_model(
model=IntTaskParameterDefinition,
obj={"name": "foo", "type": "INT", "range": range_expr},
)

# AND the instantiation target parses the same expression
instantiated = _parse_model(
model=RangeExpressionTaskParameterDefinition,
obj={"type": "INT", "range": range_expr},
)

# THEN neither rejects it, and the range expands in full
assert len(instantiated.range) == expected_len

@pytest.mark.parametrize(
"model,obj",
(
pytest.param(
IntTaskParameterDefinition,
{"name": "foo", "type": "INT", "range": [1] * 1025},
id="template layer",
),
pytest.param(
RangeListTaskParameterDefinition,
{"type": "INT", "range": [1] * 1025},
id="instantiation layer",
),
),
)
def test_list_form_range_is_still_capped(self, model: Any, obj: dict[str, Any]) -> None:
# WHEN a list-form range one element past the §3.4 cap is parsed
with pytest.raises(ValidationError) as excinfo:
_parse_model(model=model, obj=obj)

# THEN it is rejected
assert len(excinfo.value.errors()) > 0


class TestStepParameterSpaceDefinition:
@pytest.mark.parametrize(
"data",
Expand Down
Loading