Today I’d like to share a small yet valuable trick I had to learn while dealing with problems in the btrfs filesystem.
I switched to the filesystem about 8 months ago to better manage space allocation and shutdown times, as it should be quicker to recover upon errors. So far the filesystem performed admirably considering its reputation, but today I hit a nasty bug in the (behold) free space handling code.
Fiddling with large datasets, I managed to fill the whole volume, allocating chunks on the whole RAID1 configuration I setup. After removing the data though, things didn’t go as planned.
I run rebalances from time to time to reclaim large contiguous free space regions, and this time it seemed appropriate, considering I filled the RAID with 2TB worth of data and reclaimed ~1.75TB after a few hours. Should be easy right?
A nice ENOSPC error greets me. I re-checked I had free space available, nothing weird here (sample stat):
There’s even global reserve available, so what is going on here?
Apparently btrfs is unable to detect that there’s plenty of free space for rebalancing, even though the space is available – I can write new files, I tried writing 100GB and everything works. Using the -dusage=99 parameter (with any value) didn’t have any effects, I would still get the “not enough free space” errors.
It seems as if the rebalance code is checking whether there is “Device unallocated” space, and if none is available it’s not even checking that there’s actually “Free (estimated)” space. At this point, I was thinking whether I would have ever been able to rebalance at all again, since I had no way of relocating the whole volume and recreating it from scratch.
After one hour spent browsing bug reports, I begun wondering if there was a simple hack that would be able to force the data in being relocated. It was at that point that I thought of the possibility of shrinking the volume. Btrfs, unlike XFS and ZFS, allows shrinking volumes, and since such operation is a safe operation, it should trigger data rebalancing and moving in case data was present in a zone-to-free area.
Btrfs allows resizing single devices indipendently, the RAID1 replication is enforced at chunk level. I supposed that in order to allow rebalance I would have needed to free space on both devices, otherwise it would have run on one of the two devices but not on the second, because of lack of free space. Easy enough, the chunks were filled mostly with metadata so it didn’t take too long to run the resize:
I had now a smaller filesystem that would still have triggered the bug because there would be no unallocated space. So, there goes the magic:
Expanding the filesystem again we get:
Now the volume has 6GB unallocated! No more ENOSPC!
Note: when I started writing this article, I was running kernel 4.6. I’m now on 4.8 and cannot be sure whether this can still be reproduced.