Skip to content

Fix #959: count reports the number of Data Points - #967

Merged
albertohernandez1995 merged 22 commits into
1.9.Xfrom
cr-959
Aug 13, 2026
Merged

Fix #959: count reports the number of Data Points#967
albertohernandez1995 merged 22 commits into
1.9.Xfrom
cr-959

Conversation

@albertohernandez1995

@albertohernandez1995 albertohernandez1995 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Applies the meeting decision
on count, and closes #996 with it: the eight cases reported there were checked against the same
decision, as its comment
asked.

What count does now

  • A Data Set evaluation applies the operator to each Measure and gives that Measure back counted:
    COUNT(Me_1) AS Me_1. A Data Set holding a single Measure returns it renamed to int_var.
  • The Data Set form and the Component form report the same numbers:
    count(DS_1 group by Id_1) and DS_1[aggr n := count(Me_1) group by Id_1] agree.
  • A group that counts no value reports 0, not null. The DuckDb transpiler dropped the two
    NULLIF(..., 0) wrappers that turned an empty count into null; the pandas engine fills the
    groups the aggregation leaves out, after the having filter reads them.
  • count() with no operand still reports the number of Data Points: it is now given no Measures
    to count, which also fixes it reporting the first Measure's non-null count on the pandas engine,
    and reporting nothing at all when the Data Set is not grouped.

The cases of #996

# Case Reported Now Settled by
1 count(DS_1) 0 Me_int 9, Me_num 10, Me_str 8 this PR
2 count(DS_1 group by Id_1) NA one count per Measure, 0 for an empty group this PR
3 [aggr n := count(Me_int) group by Id_2] NA 3, 3, 3, 0 this PR
4 [aggr n := count() group by Id_2] NA on pandas 3, 3, 3, 1 this PR
5 count(Me_int over (partition by Id_2)) 1 on pandas 0 #990
6 count(Me_num over (order by Id_2, Id_1)) 1 on pandas 10 on every row #1002
7 having + viral attribute KeyError on pandas 3, 3, 3 #990
8 count(Me_str over (partition by Id_2)) TypeError on pandas 3, 2, 2, 1 #990

Two of them answer differently from what that issue expected, on purpose: count over a Data Set
no longer reports the number of Data Points (cases 1 and 2), and an omitted window clause covers
the whole partition (case 6, #1002). Checking the first of those turned up one more bug, fixed
here: DS_1 [ aggr n := count() ] with no group clause raised
Invalid Input Error: Need a DataFrame with at least one column on the pandas engine while
DuckDb answered 10, because an ungrouped count() is left with no column at all.

Checklist

  • Code quality checks pass (ruff format, ruff check, mypy)
  • Tests pass (pytest, both duckdb and pandas backends)
  • Documentation updated (if applicable)

Impact / Risk

  • Breaking. count over a Data Set no longer reports the number of Data Points:
    • a Data Set with several Measures returns one counted Measure per Measure instead of a single
      int_var, so [rename int_var to ...] and [keep int_var] stop resolving;
    • a Data Set with a single Measure returns that Measure's non-null values, a smaller number
      wherever the operand holds nulls.
  • Grouped counts return 0 where they used to return null.
  • Scripts that mean "number of Data Points" migrate to DS [ aggr int_var := count ( ) group ... ]
    or count ( DS#<identifier> group ... ) — an identifier is never null, so both keep the previous
    numbers and the int_var name. Seven test scripts were migrated that way, including the AnaMart
    and AnaVal projects.
  • 65 reference files updated to the new shape, and total_records_per_month in GL_466_1 and
    aggr.numDPCouYear in DEMO1 show the effect on real scripts.
  • Needs a release note.

Notes

GH_959_1 covers a Data Set of one Measure, GH_959_2 a Data Set of two, and GH_996_1 runs the
nine cases of #996 over the Data Set of that issue.

GH_980_1 pins the analytic count over a partition that holds only a null, which reports 0. That
was already fixed by #990 and is not changed here: the fixture is what makes the count rework
answerable for it, since the two engines disagreed on it before (1 on pandas, 0 on DuckDb) and
the Data Set is the one attached to that issue.


Closes #959
Closes #996
Closes #980

@albertohernandez1995
albertohernandez1995 requested review from a team and javihern98 August 4, 2026 08:24
@albertohernandez1995
albertohernandez1995 enabled auto-merge (squash) August 4, 2026 11:28
@javihern98 javihern98 linked an issue Aug 4, 2026 that may be closed by this pull request
2 tasks
albertohernandez1995 and others added 10 commits August 5, 2026 08:51
count in a Data Set evaluation applies to each Measure and gives back that same
Measure, counted; a lone Measure is renamed to int_var. The Data Set form and the
Component form now report the same numbers, and a group that counts no value
reports 0 instead of null. count() with no operand still reports the number of
Data Points.
Every case in the issue behaves as decided for count, on both engines, so the
fixture records them: a Data Set operand counts each Measure and reports 0 for a
group holding no value, count() with no operand counts the Data Points, an
analytic count reports 0 for a partition of nulls and covers the whole partition
where the window clause is omitted, and a having condition keeps the viral
attributes.
Every case in the issue behaves as decided for count, on both engines, so the
fixture records them: a Data Set operand counts each Measure and reports 0 for a
group holding no value, count() with no operand counts the Data Points, an
analytic count reports 0 for a partition of nulls and covers the whole partition
where the window clause is omitted, and a having condition keeps the viral
attributes.
@albertohernandez1995 albertohernandez1995 self-assigned this Aug 12, 2026
@javihern98 javihern98 linked an issue Aug 12, 2026 that may be closed by this pull request
2 tasks

@javihern98 javihern98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks! 😊

@albertohernandez1995
albertohernandez1995 merged commit 8f830df into 1.9.X Aug 13, 2026
17 checks passed
@albertohernandez1995
albertohernandez1995 deleted the cr-959 branch August 13, 2026 09:56
@javihern98 javihern98 mentioned this pull request Aug 13, 2026
2 tasks
@javihern98 javihern98 linked an issue Aug 13, 2026 that may be closed by this pull request
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Several count errors Count difference (DuckDb) Count skips Data Points that have a null Measure

2 participants