Why is "Metadata,DUP" almost 5x bigger now?

[JohnRobson]

I bought a new HDD (same model and size) to back up my 1-year-old current disk. I decided to format it and RSync all the data, but the new disk "Metadata,DUP" is almost 5x bigger (222GB vs 50GB). Why? Is there some change in the BTRFS that makes this huge difference?

I ran "btrfs filesystem balance start --full-balance" twice, which did not decrease the Metadata, keeping the same size. I did not perform a scrub, but I think this won't change the metadata size.

The OLD Disk was formatted +- 1 year ago and has +- 40 snapshots (more data): $ mkfs.btrfs --data single --metadata dup --nodiscard --features no-holes,free-space-tree --csum crc32c --nodesize 16k /dev/sdXy

$ sudo btrfs filesystem usage -T /dev/sdd2
Device size: 15.37TiB
Device allocated: 14.09TiB
Device unallocated: 1.28TiB
Device missing: 0.00B
Device slack: 3.50KiB
Used: 14.08TiB
Free (estimated): 1.29TiB (min: 660.29GiB)
Free (statfs, df): 1.29TiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single DUP DUP Unallocated Total Slack
-- --------- -------- -------- -------- ----------- -------- -------
1 /dev/sdd2 14.04TiB 50.00GiB 16.00MiB 1.28TiB 15.37TiB 3.50KiB
-- --------- -------- -------- -------- ----------- -------- -------
Total 14.04TiB 25.00GiB 8.00MiB 1.28TiB 15.37TiB 3.50KiB
Used 14.04TiB 24.58GiB 1.48MiB

The NEW Disk was formatted now and I performed just 1 snapshot: $ mkfs.btrfs --data single --metadata dup --nodiscard --features no-holes,free-space-tree --csum blake2b --nodesize 16k /dev/sdXy

$ btrfs --version
btrfs-progs v6.16
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=libgcrypt

Device size: 15.37TiB
Device allocated: 12.90TiB
Device unallocated: 2.47TiB
Device missing: 0.00B
Device slack: 3.50KiB
Used: 12.90TiB
Free (estimated): 2.47TiB (min: 1.24TiB)
Free (statfs, df): 2.47TiB
Data ratio: 1.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no
Data Metadata System
Id Path single DUP DUP Unallocated Total Slack
-- --------- -------- --------- -------- ----------- -------- -------
1 /dev/sdd2 12.68TiB 222.00GiB 16.00MiB 2.47TiB 15.37TiB 3.50KiB
-- --------- -------- --------- -------- ----------- -------- -------
Total 12.68TiB 111.00GiB 8.00MiB 2.47TiB 15.37TiB 3.50KiB
Used 12.68TiB 110.55GiB 1.36MiB

The nodesize is the same 16k, and only the checksum algorithm is different (but they use the same 32 bytes per node, this won't change the size). I also tested the nodesize 32k and the "Metadata,DUP" increased from 222GB to 234GiB. Both were mounted with "compress-force=zstd:5"

The OLD disk has More data because of the 40 snapshots, and even with more data, the Metatada is "only" 50GB compared to 222+GB from the new disk. Some changes in BTRFS code during this 1-year created this huge difference? Or does having +-40 snapshots decreases the Metadata size?

Only the metadata is bigger, I did an "ncdu" and all folders have the same size on both disks. In fact, the old HDD contains even more data that wasn’t copied (the .snapshots folder), yet its metadata usage is only about 50 GB, compared with 222+ GB on the new HDD. I suspect that a change in the mkfs.btrfs code or defaults is responsible for this metadata bloat.

[Arcitec]

Thanks for reporting this. Will be interesting to hear if it's a bug!

I came here from the related Reddit conversation. Adding it since it provides some more context:

https://www.reddit.com/r/btrfs/comments/1naaxqi/

[Augusto7743]

That is because I format using mix data and metadata. Use less disk space, but can create fragmentation.
Between an bit of fragmentation and high usage of metadata my choice is the mix data and metadata.

[voidpointertonull]

Were there any lessons learned from this?

As I can see in Reddit comments, a test of comparing crc32c and blake2b showed that using blake2b results in significantly inflated metadata size. Unfortunately the test environment and methodology weren't described.

Mostly wondering if there's a problem with the large size of blake2b checksums, or the problem appears as soon as there's a deviation from the default crc32c.
I'm mostly curious because I use xxhash, and at some point (maybe after a kernel update) writing started appearing to be more I/O intense, even including deleting files (on HDDs) stalling I/O for a long time for other processes, even though it should be mostly just metadata updating. May not be related, but a lot more metadata needing to be updated could explain what I'm seeing.

[kakra]

There may be multiple issues playing into that:

• If you transferred not only current data but also snapshots with rsync, rsync won't keep deduplication within snapshots, it will write duplicate files from the snapshot as new duplicated data. rsync doesn't know about btrfs-specifics and how file contents are shared in snapshots.
• If you transfer data with rsync that already has snapshots on the target, rsync won't keep partially shared file blocks because by default it will re-create each file fully to a temporary file, then rename it over the original file. But rsync has an operating mode to make it more CoW-friendly (with some caveats).

If you want to keep data shared as best as possible on the target, you should run rsync in CoW-friendly mode by letting it update the target in place to a scratch area first. This can result in broken file contents if the process is interrupted (it will leave partially transferred or partially updated files behind) - thus the scratch area. Only if rsync has been successful, finally snapshot the scratch area into your final backup snapshot:

# TARGET=your-final-directory-name

# ensure the directories exist
mkdir -p /mnt/backup/scratch /mnt/backup/snapshots

# run rsync in cow-friendly mode (partial, no-whole-file, inplace)
# HINT: ensure that the source does not include snapshots
rsync -avHXAS --inplace --partial --no-whole-file --inplace /mnt/source/data/. /mnt/backup/scratch/.

# only snapshot after rsync successfully transferred all files, aka no partials left, no previous errors
btrfs subvolume snapshot -r /mnt/backup/scratch /mnt/backup/snapshots/$TARGET

You will see that meta data will likely always be bigger than the source.

You may improve deduplication by running bees on the target but I would not recommend this for a backup destination because it will inflate meta data size, and temporarily it will most likely need a lot of extra space.

You may have better results using a dedicated, deduplicating backup tool like borg or restic.

You may also have better results using btrfs send|receive to run your backups from source to destination because it will be faster and transfer only deltas. But this needs some careful setup of snapshots.