feat: Generalize metrics and add nullability delta metric

[Moritz Potthoff (MoritzPotthoffQC)]

Motivation

I like the concept of metrics to give the user as much information about their data at first glance. I think it would be nice to extend it even further by generalizing metrics to non-numeric columns. The specific use-case I have for this is to explain how the share of null values in a column changed.

Changes

• Generalize metrics to non-numeric columns: Keep existing behavior for passing callables (numeric only), but additionally support metrics with custom column selector
• Add a nullability change metric with format <old> -> <new> (<delta>). This is consistent with the sample values we show. feat: Improve existing numeric metrics for new output format #44 also applies this to existing metrics in follow-up.
• Update docs and CLI accordingly

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 100.00%. Comparing base (ea78b69) to head (eb1c427).

Additional details and impacted files

@@            Coverage Diff            @@
##              main       #43   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           11        11           
  Lines         1018      1030   +12     
=========================================
+ Hits          1018      1030   +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR extends Diffly’s per-column summary metrics to support non-numeric columns via an explicit Metric(fn, selector=...) wrapper, and introduces a built-in metric for nullability changes so users can quickly see how null fractions evolve between left/right frames.

Changes:

• Allow summary(metrics=...) / testing helpers to accept either a bare metric callable (numeric-only) or a Metric with a Polars selector (any column types).
• Add the null_fraction_change preset and expose it as ΔNull% in DEFAULT_METRICS and the CLI.
• Update summary formatting + docs, and add tests + generated summary fixtures for the new metric behavior.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
diffly/metrics.py	Adds Metric wrapper type, null-fraction change preset, and updates DEFAULT_METRICS to include selector-based metrics.
diffly/comparison.py	Accepts Metric objects in summary(metrics=...) and resolves bare callables as numeric-only.
diffly/summary.py	Allows metrics to render string results in column summaries.
diffly/testing.py	Broadens metrics typing/docs to allow Metric values in assertion helpers.
diffly/cli.py	Updates CLI help text and passes selector-based preset metrics through to summary().
docs/api/metrics.rst	Documents selector-based metrics and adds Metric + null_fraction_change to the API docs.
tests/test_metrics.py	Adds unit tests for null_fraction_change output (including non-numeric columns).
tests/cli/test_cli.py	Adds CLI coverage for selecting the ΔNull% metric preset.
tests/summary/fixtures/metrics_null_fraction/test_metrics_null_fraction.py	New fixture generator test for summaries including ΔNull% and a selector-based custom metric.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_True_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_True_sample_pk_True.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_True_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_True_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_False_perfect_False_top_False_slim_False_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.
tests/summary/fixtures/metrics_null_fraction/gen/pretty_True_perfect_True_top_True_slim_True_sample_rows_False_sample_pk_False.txt	Generated summary fixture variant for ΔNull%.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 2 comments.