Escaping BTRFS bugs

Hi all!

Today I’d like to share a small yet valuable trick I had to learn while dealing with problems in the btrfs filesystem.
I switched to the filesystem about 8 months ago to better manage space allocation and shutdown times, as it should be quicker to recover upon errors. So far the filesystem performed admirably considering its reputation, but today I hit a nasty bug in the (behold) free space handling code.

Fiddling with large datasets, I managed to fill the whole volume, allocating chunks on the whole RAID1 configuration I setup. After removing the data though, things didn’t go as planned.

I run rebalances from time to time to reclaim large contiguous free space regions, and this time it seemed appropriate, considering I filled the RAID with 2TB worth of data and reclaimed ~1.75TB after a few hours. Should be easy right?

Plain Text

A nice ENOSPC error greets me. I re-checked I had free space available, nothing weird here (sample stat):

Plain Text

There’s even global reserve available, so what is going on here?

Apparently btrfs is unable to detect that there’s plenty of free space for rebalancing, even though the space is available – I can write new files, I tried writing 100GB and everything works. Using the -dusage=99 parameter (with any value) didn’t have any effects, I would still get the “not enough free space” errors.

It seems as if the rebalance code is checking whether there is “Device unallocated” space, and if none is available it’s not even checking that there’s actually “Free (estimated)” space. At this point, I was thinking whether I would have ever been able to rebalance at all again, since I had no way of relocating the whole volume and recreating it from scratch.

After one hour spent browsing bug reports, I begun wondering if there was a simple hack that would be able to force the data in being relocated. It was at that point that I thought of the possibility of shrinking the volume. Btrfs, unlike XFS and ZFS, allows shrinking volumes, and since such operation is a safe operation, it should trigger data rebalancing and moving in case data was present in a zone-to-free area.

Plain Text

Btrfs allows resizing single devices indipendently, the RAID1 replication is enforced at chunk level. I supposed that in order to allow rebalance I would have needed to free space on both devices, otherwise it would have run on one of the two devices but not on the second, because of lack of free space. Easy enough, the chunks were filled mostly with metadata so it didn’t take too long to run the resize:

Plain Text

I had now a smaller filesystem that would still have triggered the bug because there would be no unallocated space. So, there goes the magic:

Plain Text

Expanding the filesystem again we get:

Plain Text

Now the volume has 6GB unallocated! No more ENOSPC!

Note: when I started writing this article, I was running kernel 4.6. I’m now on 4.8 and cannot be sure whether this can still be reproduced.

About the author

Fabio Scaccabarozzi
Fabio Scaccabarozzi

Ever since 2002, I experimented with Linux. After some time spent on Slackware and Mandriva, I found Gentoo, I've been using it since then and I acquired a lot of knowledge on the internals of the distribution. I have created and mantained a custom liveCD targeted to power users and systems engineers, with a lot of tools usually not included in standard distributions and exotic filesystems. I'm currently working as a senior system administrator in Zurich, Switzerland.

Add Comment