AnnData obs/var string to categoricals by ajkswamy · Pull Request #1170 · scverse/spatialdata

ajkswamy · 2026-07-31T16:11:49Z

Issue:

#1152

Summary:

Writing AnnData table to zarr file now uses different methods depending on zarr format being written and anndata version installed
A new custom Deprecation warning is thrown when writing any element to zarr format v2.
A new hatch env config to help run all tests for combination of anndata/pandas versions

How to test:

Use this notebook, modified from the one provided by LazDaria in Missing values in AnnData .uns DataFrame become string "nan" after SpatialData Zarr round trip #1152 to cover all cases there
Use the hatch env added here and call hatch run test-anndata-pandas: test {args}. Check config before running, every script is run multiple times based on the matrix defined.

Related PRs

Support sharding through config and raster_write_kwargs #1106 (comment)

… tables to zarr v2

…pd.NA/np.nan

…data/pandas

codecov · 2026-07-31T16:14:39Z

Codecov Report

❌ Patch coverage is 86.66667% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.40%. Comparing base (eb4fb3d) to head (08b811c).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/spatialdata/_io/exceptions.py	60.00%	4 Missing ⚠️
src/spatialdata/_io/io_table.py	90.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1170      +/-   ##
==========================================
- Coverage   92.44%   92.40%   -0.04%     
==========================================
  Files          51       52       +1     
  Lines        7820     7858      +38     
==========================================
+ Hits         7229     7261      +32     
- Misses        591      597       +6

Files with missing lines	Coverage Δ
src/spatialdata/_core/spatialdata.py	`93.86% <ø> (ø)`
src/spatialdata/_io/io_points.py	`100.00% <100.00%> (ø)`
src/spatialdata/_io/io_raster.py	`89.35% <100.00%> (+0.30%)`	⬆️
src/spatialdata/_io/io_shapes.py	`96.34% <100.00%> (+0.18%)`	⬆️
src/spatialdata/_io/io_table.py	`89.28% <90.47%> (-1.20%)`	⬇️
src/spatialdata/_io/exceptions.py	`60.00% <60.00%> (ø)`

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

LucaMarconato · 2026-08-03T15:52:18Z

+    assert element_format in TablesFormats.values(), FormatVersionUnknownError(
+        element_type="table", version_encountered=element_format
+    )


This assertion gets silenced when running code with python -O. An if raise is preferable. Please change.

also here there is no raise, so even in the assertion is evaluated we would get an AssertionError, not FormatVersioUnknownError

Agree, done

LucaMarconato · 2026-08-03T15:56:38Z

+        # Write the table to the path of the table group
+        table.write_zarr(store=resolved_store, consolidate_metadata=False)
+
+        table_group = group[name]


do we need to redefine table_group? If yes please explain in a comment why. If not please remove.

you are right, I have removed them

LucaMarconato · 2026-08-03T16:01:52Z

+
+        table_group = group[name]
+    else:
+        table.strings_to_categoricals()


Here there is a problem: table.strings_to_categoricals() changes the dtype of the string column to a categorical. An in-place operation from a write operation is an unexpected side effect.

We need to find a different solution. I suggest to add an extra assert to test_sdata_with_nan_in_obs() to verify that the dtype of the obs columns of the input object does not change after write, and to run this against all the combinations of pandas and anndata via the hatch config you created.

Yes, right, I added the extra assert statements to test_sdata_with_nan_in_obs() and indeed there is a side effect caused by strings_to_categoricals().

anndata.AnnData.write_zarr has an argument convert_strings_to_categorical which is set to True by default. Hence the default round trip for nullable stings columns works by default with anndata, but with this side effect. Here, I have added an argument to io_table.write_table along with all functions in the call chain from Spatialdata.write that allows users the same options to convert strings in AnnData objects to categorical before writing. The main difference that we set the default value to be false, i.e., we do not convert by default and users need to explicitly set this to be able to do round trip with nullable string/object columns with pandas 2.0.

LucaMarconato · 2026-08-03T16:04:22Z

+    "prek>=0.4.11",
+    "ruff>=0.16.0",


Not something to do now, but after moving to the scverse cookiecutter these dev packages may change. We'll figure this out later.

LucaMarconato · 2026-08-03T16:08:54Z

+    {value="anndata>=0.12,<0.13", if = ["0.12-2"]},
+    {value="pandas>=2.3,<3", if = ["0.12-2"]},


why isn't this included in the tests anndata>=0.12,<0.13 with pandas~=3.0? I would either include or explain why it's not in a comment.

Also instead of the comment # every option where the if-condition is True gets included

I would consider adding these comments, to make the 3 cases immediate. But up to you.

# case 1 {value="anndata~=0.13", if = ["0.13-2"]}, {value="pandas>=2.3,<3", if = ["0.13-2"]}, # case 2 {value="anndata~=0.13", if = ["0.13-3"]}, {value="pandas~=3.0", if = ["0.13-3"]}, # case 3 {value="anndata>=0.12,<0.13", if = ["0.12-2"]}, {value="pandas>=2.3,<3", if = ["0.12-2"]}, # case 4: anndata xxx pandas yyy not added because zzz (or add missing case)

I've added a comment why the case of anndata>=0.12,<0.13 with pandas~=3.0 .

I have simplified the conditionals defining the dependency versions. For now, I prefer to keep them organized per dependency and version. We can modify them if needed later.

LucaMarconato · 2026-08-03T16:13:57Z

+        element_type="table", version_encountered=element_format
+    )
+
+    if element_format.zarr_format == 3 and version("anndata") >= "0.13":


The second boolean condition performs a string comparison that is evaluated lexicografically, not parsing the version numbers:

version("anndata") >= "0.2.0" Out[5]: False version("anndata") Out[6]: '0.12.7'

To fix you can wrap each string with packaging.version.Version()

Right, done

LucaMarconato · 2026-08-03T16:45:23Z

One last thing, please rename the PR to a more descriptive text since upon release the automatically-generated release notes will use the PR title.

ajkswamy added 8 commits July 27, 2026 18:26

chore: added prek to dev deps

1c1606c

feat: writing table to zarr using AnnData.write_zarr for table version 2

2672c01

chore: added ruff to dev deps

a88accb

fix: using internal resolve store function + categorical when writing…

26a928f

… tables to zarr v2

fix: test no longer expectes 'nan' after table round trip instead of …

247cdc6

…pd.NA/np.nan

feat: added hatch env configs for testing version combinations of ann…

75dcba9

…data/pandas

feat: update hatch config for testing pandas/anndata versions

129bf9e

feat: added DeprecationWarnings when writing to zarr v2

5f4afca

ajkswamy requested review from LucaMarconato, Tomaz-Vieira and melonora July 31, 2026 16:11

ajkswamy self-assigned this Jul 31, 2026

ajkswamy marked this pull request as ready for review July 31, 2026 16:12

ajkswamy requested a review from ilan-gold August 3, 2026 06:44

LucaMarconato reviewed Aug 3, 2026

View reviewed changes

ajkswamy changed the title ~~Iss1152~~ AnnData obs/var string to categoricals Aug 10, 2026

ajkswamy added 7 commits August 10, 2026 14:26

fix: if-raise instead of assert

9d5c7dc

refac + doc: pyproject.toml

20d9c55

fix: made version comparison of anndata more robust

1a3c318

fix: doc string

74ab61f

fix: made converting table strings to categorical optional + non-default

2d742ad

feat: documentation strings

5ab0148

feat: revised tests to work with new write parameter

ef2901c

fix: removed unnecessary redeclaration of table group

08b811c

ajkswamy requested a review from LucaMarconato August 12, 2026 18:51

		{value="anndata>=0.12,<0.13", if = ["0.12-2"]},
		{value="pandas>=2.3,<3", if = ["0.12-2"]},

Uh oh!

Conversation

ajkswamy commented Jul 31, 2026

Issue:

Summary:

How to test:

Related PRs

Uh oh!

codecov Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucaMarconato Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LucaMarconato commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov Bot commented Jul 31, 2026 •

edited

Loading

LucaMarconato Aug 3, 2026 •

edited

Loading