Bcachefs File-System Plans to Try Again to Land in Linux 6.6(phoronix.com)
phoronix.com
Bcachefs File-System Plans to Try Again to Land in Linux 6.6
https://www.phoronix.com/news/Bcachefs-Plans-For-Linux-6.6
148 comments
In the early days of SSD adoption there were 'Hybrid HDD' that effectively acted the way you described, using a small SSD as a read/write cache.
Ultimately, the price of SSDs dropped to the level were such solutions filled an ever shrinking niche and they fell out of favor.
Ultimately, the price of SSDs dropped to the level were such solutions filled an ever shrinking niche and they fell out of favor.
[deleted]
I'm not sure about erasure codes, but it seems to be possible to do this in a couple different ways already:
https://wiki.archlinux.org/title/bcache
https://wiki.archlinux.org/title/LVM#Cache
https://wiki.archlinux.org/title/bcache
https://wiki.archlinux.org/title/LVM#Cache
Yes, the desired behavior is "writeback caching".
However, I don't think either of these systems provide a configurable threshold like "max 5 minutes of dirty cache". Instead, you might have to guess as to how large of a cache device you can risk. The larger the cache, the more volatile data might be lost. But, we generally don't have guaranteed writeback throughput either. So you cannot be sure how long a given cache size will take to fully drain to the second storage layer in the worst case.
When I have used the LVM cache mechanism, I made a RAID1 mirror of two SSDs for the writeback cache layer, and hoped they were different enough devices so that they wouldn't simultaneously fail from the mirrored traffic patterns. In fact, I used a RAID1 of SSDs to writeback cache a RAID1 of HDDs.
But these days, I usually just try to have a large enough SSD to hold my entire data and then use a scheduled backup tool like restic. It copies data to another device but also maintains an archive of multiple snapshots for point-in-time recovery scenarios. I accept up to 24 hours of loss here, using a daily backup procedure.
For really important data where I worry about 24 hours, I might manually replicate after making important file updates. E.g. git push to replicate commits to a remote repo, or rsync to replicate specific subdirs to another machine.
Edit to add: when worrying about 24 hours, it is also worthwhile to think about the probabilities. The extra effort to maintain a low latency backup might end up costing more time than the unlikely potential day of redundant work after a loss. And how long will the recovery take anyway? Do you keep spare parts on hand or expect to spend time procuring hardware and rebuilding your system after a period of inaction due to a broken system...?
The things that I consider too important to wait 24 hours would be non-deterministic things where a recovery method is more difficult than just performing the day's tasks again. E.g. losing recently changed passwords/secrets in an encrypted password store, or losing my notes about other infrequent purchases or bill payments that are in progress. I wouldn't just repeat the steps, but instead have to unravel other recovery procedures for each item.
However, I don't think either of these systems provide a configurable threshold like "max 5 minutes of dirty cache". Instead, you might have to guess as to how large of a cache device you can risk. The larger the cache, the more volatile data might be lost. But, we generally don't have guaranteed writeback throughput either. So you cannot be sure how long a given cache size will take to fully drain to the second storage layer in the worst case.
When I have used the LVM cache mechanism, I made a RAID1 mirror of two SSDs for the writeback cache layer, and hoped they were different enough devices so that they wouldn't simultaneously fail from the mirrored traffic patterns. In fact, I used a RAID1 of SSDs to writeback cache a RAID1 of HDDs.
But these days, I usually just try to have a large enough SSD to hold my entire data and then use a scheduled backup tool like restic. It copies data to another device but also maintains an archive of multiple snapshots for point-in-time recovery scenarios. I accept up to 24 hours of loss here, using a daily backup procedure.
For really important data where I worry about 24 hours, I might manually replicate after making important file updates. E.g. git push to replicate commits to a remote repo, or rsync to replicate specific subdirs to another machine.
Edit to add: when worrying about 24 hours, it is also worthwhile to think about the probabilities. The extra effort to maintain a low latency backup might end up costing more time than the unlikely potential day of redundant work after a loss. And how long will the recovery take anyway? Do you keep spare parts on hand or expect to spend time procuring hardware and rebuilding your system after a period of inaction due to a broken system...?
The things that I consider too important to wait 24 hours would be non-deterministic things where a recovery method is more difficult than just performing the day's tasks again. E.g. losing recently changed passwords/secrets in an encrypted password store, or losing my notes about other infrequent purchases or bill payments that are in progress. I wouldn't just repeat the steps, but instead have to unravel other recovery procedures for each item.
[deleted]
Discord did something that seems to be what you're looking for using md:
https://news.ycombinator.com/item?id=32474093
https://discord.com/blog/how-discord-supercharges-network-di...
https://news.ycombinator.com/item?id=32474093
https://discord.com/blog/how-discord-supercharges-network-di...
Multi-device filesystem is a more fancy solution, but you can do this at block level already. This is where bcache started. Make a virtual device with the slow disks behind some fast ones with bcache, then put any filesystem you want on that device. It should do what you described.
I had wanted such a thing too, for reasons you specify and also because I would expect that it would be less loud than multiple spinning hard drives. Either one can fail and if so then the data from the remaining working one can be copied onto a fresh SSD or hard disk, I suppose. Combined with journaling file system can perhaps improve further. With appropriate considerations, you could recover after a power outage.
(This might be able to be done without a specialized file system, although a driver and/or controller might be needed for the mirroring and access.)
(This might be able to be done without a specialized file system, although a driver and/or controller might be needed for the mirroring and access.)
Not quite the same, but snapraid can do something sort of like this. It's not real time though, it's scheduled. It's usually combined with something like mergerfs.
I do something like this with ZFS snapshots and syncoid. It runs every 15 minutes and sends the diff from the latest snapshot onto a second ZFS pool made out of a single HDD (and then updates that snapshot and discards deleted data). Works fine for about a year now.
What's are the usecases of a filesystem like that? I can't think of a situation where I care that my data is mirrored, but I'm okay with losing 5 minutes of it.
Your home desktop PC?
I'd be perfectly fine losing 5 minutes of data in case of a disk failure. After all - I can probably recreate it in about 5 minutes.
Anyone who does daily backups is effectively saying "I care about my data, but I'm happy to lose up to a days worth if there is a failure".
I'd be perfectly fine losing 5 minutes of data in case of a disk failure. After all - I can probably recreate it in about 5 minutes.
Anyone who does daily backups is effectively saying "I care about my data, but I'm happy to lose up to a days worth if there is a failure".
I suppose I would just throw another SSD in there and mirror with RAID1 (or equivalent), and save myself the headache. SSDs are cheap - A Samsung 980 1TB is $50.
Edit: I think you can achieve what you want with current tools, or something pretty close at least, with ZFS. Create a snapshot of your SSD pool every 5 minutes, then stream the delta to an HDD pool in an automated fashion. Delete snapshots more than X hours / days old.
Edit: I think you can achieve what you want with current tools, or something pretty close at least, with ZFS. Create a snapshot of your SSD pool every 5 minutes, then stream the delta to an HDD pool in an automated fashion. Delete snapshots more than X hours / days old.
From what I understand, cheap SSDs behave very badly under RAID. For example, Synology actively discourages Samsung EVO SSDs because they cause timeout issues.
This is exactly how my desktop PC is setup.
I use the HDD ZFS pool to maintain effectively infinite non-expiring snapshots of my (single device) SSD ZFS pool.
If my SSD fails, I can boot directly from the latest backup on the HDD pool.
I use the HDD ZFS pool to maintain effectively infinite non-expiring snapshots of my (single device) SSD ZFS pool.
If my SSD fails, I can boot directly from the latest backup on the HDD pool.
One of the reasons they say "RAID is not backup" is because the kinds of things that take out a single drive, take out multiple drives together often enough that it surprises people. I've seen more than one person report that they had a RAID NAS at home, thinking their data was safe, only to be horrified one day that every drive failed the same day. An electrical surge from a lightning strike or a faulty power supply, for example, can fry all the drives at once.
Root on RAM with the data being flushed out to disk?
ZFS SLOG and L2ARC can approximate this.
Same here except the usage being inside NAS. Having something like 8TB SSD and and two 8TB HDD.
bacachefs does have some modes that is closely related to your first sdd+hdd scenario. Quite elegant imho, but waiting for it to be mainlined to try it out.
Question:
I currently run ZFS via Ubuntu on my home devices. I've had a very positive experience with it. My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time. It's been fantastic for all my other devices, particularly the snapshotting features and ease of which I can ship the snapshots between the devices.
I've read some comments on various forums that seem to hint Ubuntu is moving on from ZFS (see https://discourse.ubuntu.com/t/future-of-zfs-on-ubuntu-deskt... for one example). This is obviously concerning and the lack of support across the whole Linux ecosystem continues to be a nuisance.
Realistically how well could bcachefs serve as a long term ZFS replacement? I know very little about it. I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go.
I currently run ZFS via Ubuntu on my home devices. I've had a very positive experience with it. My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time. It's been fantastic for all my other devices, particularly the snapshotting features and ease of which I can ship the snapshots between the devices.
I've read some comments on various forums that seem to hint Ubuntu is moving on from ZFS (see https://discourse.ubuntu.com/t/future-of-zfs-on-ubuntu-deskt... for one example). This is obviously concerning and the lack of support across the whole Linux ecosystem continues to be a nuisance.
Realistically how well could bcachefs serve as a long term ZFS replacement? I know very little about it. I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go.
I've been a storage engineer since 2000. The industry rule of thumb for file systems is to give them ten years after mainline release to bake. And the longer I've done this, the more I trust it.
> I've used btrfs in the past, and between my negative experiences and the repeated anecdotes posted all over the internet to this day I'm hesitant to give it another go.
What are those issues? I've not followed the conversation, but I've been running btrfs for over a decade on multiple machines with different usage patterns without any issues.
I'm genuinely interested. Are those substantiated claims? Or just the general noisy rants you can find about essentially anything people have opinions about?
What are those issues? I've not followed the conversation, but I've been running btrfs for over a decade on multiple machines with different usage patterns without any issues.
I'm genuinely interested. Are those substantiated claims? Or just the general noisy rants you can find about essentially anything people have opinions about?
For my 2c, I haven't touch BTRFS since using it as the primary storage in a main storage server at work. We had a 6TB system reporting 3TB free (back when this was a lot of storage), yet every write operation (including deletes) was failing with an error indicating the disk was full. This was years ago, so I'm a little fuzzy on the details, but after some really painful downtime and troubleshooting, it turns out that BTRFS marks blocks as either metadata or data blocks. Once a block is marked, it won't be unmarked even if empty. So because this was a high churn system all of the blocks ended up marked as data and there was none available when more metadata was needed. The solution was running `balance` with some specific flags.
My understanding is that this is partially resolved now (empty blocks are automatically freed now), but not completely (partially empty blocks are never compacted without a balance, so you can still have a bunch of half-empty data blocks and no room for metadata), so one is a lot less likely to run into similar issues, but it was scarring enough to me that I've never run a BTRFS system again.
My understanding is that this is partially resolved now (empty blocks are automatically freed now), but not completely (partially empty blocks are never compacted without a balance, so you can still have a bunch of half-empty data blocks and no room for metadata), so one is a lot less likely to run into similar issues, but it was scarring enough to me that I've never run a BTRFS system again.
N=1, though I suspect my case is going to sound a lot like the other complaints you might find. I thought btrfs looked great so I used it for the root and home filesystems on a laptop running OpenSUSE. (This is modestly significant, because SUSE specifically supports BTRFS and I figured it would give me a better experience.) Now the home filesystem never gave me any problems. The root filesystem, however, broke to the degree that I ended up reinstalling. Twice. I couldn't tell you the exact error anymore, unfortunately, but it looked like running out of space left the fs in an unrecoverable state.
Recovery from out of space errors is difficult on any copy-on-write filesystem since you need to write more data to disk to actually delete anything. You'll find similar stories about ZFS if you care to look for them. btrfs has had some protection against out-of-space issues for some time now (look for "global reserve" in `btrfs filesystem usage`). I think ZFS has something similar, but I don't care enough to check since I'm very careful to never let it get that far.
In my own case: Btrfs performance got absolutely terrible once I was up to a few thousand snapshots. Never had it lose data, but I also never ran it long enough to lose a disk.
This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots.
I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.
I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.
I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.
I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.
I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.
I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.
My setup is that I only have subvolumes for root, and (for now) my Doom Emacs config. Whenever I do a system upgrade, or a Doom Emacs upgrade, I manually take a snapshot of the appropriate subvolume first, just in case anything goes wrong. And then I only keep a max of 7 or so snapshots before I go through and delete the old ones.
Has worked well for me so far and saved my butt a few times, and since I don't have a bunch of subvolumes to manage, I'm not overwhelmed. It's nice and simple, and I need nothing other than the default `btrfs` CLI utility.
Has worked well for me so far and saved my butt a few times, and since I don't have a bunch of subvolumes to manage, I'm not overwhelmed. It's nice and simple, and I need nothing other than the default `btrfs` CLI utility.
That's what's great about btrfs. I started using it that way too. It was nice to be able to take instant snapshots of my fs. I just amended my existing backup scripts to act on temporary snapshots instead. Then, I started dividing things up into subvolumes.
I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.
I also highly suggest reflinks. Once you start using them you will use them all the time.
I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.
I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.
I also highly suggest reflinks. Once you start using them you will use them all the time.
I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.
I did keep snapshots pruned to ~300 per subvolume.
But I also had around twenty subvolume I needed to snapshot in this manner.
But I also had around twenty subvolume I needed to snapshot in this manner.
It's been a couple years since I last tried but I had file system corruption and some random nuisance issues that have long since slipped my mind. I never really got to the bottom of it, I had other things to do than fight my file system to get it working properly.
I tried precisely that, and had it corrupt my data… twice.
Bcachefs isn’t ready for prime time. I’m reasonably sure it’s better designed than Btrfs, and it’ll get there, but don’t do it yet. If Ubuntu becomes a problem, NixOS has 100% support for ZFS.
Bcachefs isn’t ready for prime time. I’m reasonably sure it’s better designed than Btrfs, and it’ll get there, but don’t do it yet. If Ubuntu becomes a problem, NixOS has 100% support for ZFS.
Were you able to report this? If you can get me enough information (e.g. via bcachefs dump), I'd like to track this down.
We talked on IRC. :-)
I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix.
For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point.
===
It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.
I'm planning to do that, but it'll have to be after my vacation; don't want to be running bcachefs on a remote server I can't fix.
For those following along, this sort of response is why I'm optimistic overall. And bcachefs was dramatically higher performance than ZFS, although the lack of an ARC is a particular pain point.
===
It's going to be an interesting bug, I'm pretty sure. It affected two sets of files: The Steam Linux runtime (Solder), and everything in /nix/store, but absolutely nothing else. Actually, I think it's probably a bug in the fsck routine; everything worked perfectly until I rebooted.
> And bcachefs was dramatically higher performance than ZFS
Would you care to go into some specifics?
On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio.
My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.
Would you care to go into some specifics?
On my middle of the road machines, the limiting factor has always been the drive. Encrypted ZFS has the same transfer speeds as unencrypted ext4, as tested with fio.
My best machine is a zen 3 u laptop, so pcie 3 with a samsung 990 nvme drive. It reads and writes at over 2 GB/s.
Caching, mostly. My desktop has two HDDs and one NVMe. The NVMe isn't nearly that fast -- it's a 970 Evo which tops out at 1.6 GB/s -- but it's still much faster than the HDDs.
When I write data to the setup, bcachefs and ZFS had very different behaviour.
ZFS:
- Writes to the HDDs first.
- Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.)
- Is in any case limited by how much can be buffered in RAM.
- Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC.
BcacheFS:
- Writes to the SSD first.
- Writes to the HDDs at its leisure.
- Can be told the NVMe is more reliable than the HDDs.
- Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue).
- Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up.
The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
When I write data to the setup, bcachefs and ZFS had very different behaviour.
ZFS:
- Writes to the HDDs first.
- Blocks further writes until the first ones are on disk, unless I use sync=disabled. (Which I do; it's safe for my use-case.)
- Is in any case limited by how much can be buffered in RAM.
- Doesn't write to the SSD (which is configured as cache) unless it first reads data into RAM and then that data gets dropped from the in-core ARC.
BcacheFS:
- Writes to the SSD first.
- Writes to the HDDs at its leisure.
- Can be told the NVMe is more reliable than the HDDs.
- Can be configured to leave data uncompressed on the SSD (because even LZ4 is a notable drain at 1.6 GB/s), but use zstd for the HDDs (because it's async so why not, and decompressing zstd at 300 MB/s is no issue).
- Uses LRU instead of ARC, so if your working set is larger than your cache ZFS will have much better behaviour... once it's warmed up.
The bug I hit is apparently fixed now, so I'm likely to switch back to bcachefs more or less ASAP. My working set is not in fact smaller than my SSD... but I'll just get a second SSD, I think.
I don't think it's fixed yet?
biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet
biggest bugfix in the pipeline is that bfoster just figured out the 'missing backpointer' bug, but that's just in the btree write buffer, it wouldn't cause data corruption - and the fix for that isn't in yet
2GB/s is pretty low performance for storage these days. =-)
I thought we might have :)
If it's a bug in fsck that will be fortunate, that'll be easy to track down.
If it's a bug in fsck that will be fortunate, that'll be easy to track down.
I think it could be a very good replacement. In my opinion ZFS is still very static and disk/block based with some traditional RAID modes on top.
It can provide the same redundancy and safety but with tons more flexibility. For example having different redundancy levels using the same pool of disks and changing redundancy levels on the fly. It does feel like a huge step in in terms of "here are my disks" and then "please store this data this way". bcachefs then just does it in a reasonable and efficient way. Way better than having to set up all of your disk RAID configs upfront.
That being said it definitely still needs stability work. I tried it and it corrupted itself. So I love the idea, the design seems good overall. But it is going to need lots of testing and tweaking to become reliable. However it does seem like a solid addition to the filesystem space and I think being upstream will allow more people to work on it, avoid extra maintenance burden by staying in sync with changes and get the attention it deserves.
I wrote more about why I like the design and how it corrupted for me here: https://kevincox.ca/2023/06/10/bcachefs-attempt/
It can provide the same redundancy and safety but with tons more flexibility. For example having different redundancy levels using the same pool of disks and changing redundancy levels on the fly. It does feel like a huge step in in terms of "here are my disks" and then "please store this data this way". bcachefs then just does it in a reasonable and efficient way. Way better than having to set up all of your disk RAID configs upfront.
That being said it definitely still needs stability work. I tried it and it corrupted itself. So I love the idea, the design seems good overall. But it is going to need lots of testing and tweaking to become reliable. However it does seem like a solid addition to the filesystem space and I think being upstream will allow more people to work on it, avoid extra maintenance burden by staying in sync with changes and get the attention it deserves.
I wrote more about why I like the design and how it corrupted for me here: https://kevincox.ca/2023/06/10/bcachefs-attempt/
> My only (minor) complaint is the extra 30 seconds it adds to my desktop's boot time.
Wait. What? You'll forgive me, I have no experience with ZFS: why??
Wait. What? You'll forgive me, I have no experience with ZFS: why??
Metadata scanning, I presume. It's not quite 30 seconds, but mounting my 4-drive 12TB btrfs volume (spinning rust) also feels like it takes upwards of ten seconds (I've never bothered to actually measure the mount time, so can't say with certainty).
Btrfs mounting of larger arrays became near-instant for me with the new block group tree enabled; see btrfstune: https://btrfs.readthedocs.io/en/latest/btrfstune.html
When importing/opening a ZFS pool, ZFS traverses the last few transaction groups written to the ZFS pool as well as the intent logs as a verification step to make sure that the ZFS pool is in a safe/consistent state before continuing (which could otherwise corrupt a ZFS pool even more and make the pool unrecoverable if something were to be wrong).
I'm not sure if this is the main bottleneck when opening a ZFS pool during boot, but I wouldn't be surprised if it were.
And if it is, personally I'm willing to sacrifice some boot time for some assurances that my data will be safer, but of course, not everyone's priorities are the same as mine :)
(Also, for the record, for me the pool import/opening process is nearly instantaneous on SSDs/NVMe and takes 2.4 seconds on a 12-year-old Intel Atom machine with a single HDD pool).
I'm not sure if this is the main bottleneck when opening a ZFS pool during boot, but I wouldn't be surprised if it were.
And if it is, personally I'm willing to sacrifice some boot time for some assurances that my data will be safer, but of course, not everyone's priorities are the same as mine :)
(Also, for the record, for me the pool import/opening process is nearly instantaneous on SSDs/NVMe and takes 2.4 seconds on a 12-year-old Intel Atom machine with a single HDD pool).
[deleted]
Importing ZFS pools from spinning disks can be slow. I'm guessing there's some serious random access going on because it's nearly instantaneous on SSDs
I have a fair amount of experience with ZFS, and have the same question.
ZFS on Linux will not goes away. It is just that Ubuntu dropping it from the installer. You can still install it: https://launchpad.net/ubuntu/+source/zfs-linux/
And you can install it on a lot of distro too: https://openzfs.github.io/openzfs-docs/Getting%20Started/ind...
My current favorite is to install kABI-tracking kmod on RHEL. (But their kABI-tracking is not a guarantee as they uses more.) In fact I move away from Ubuntu knowing that they knowingly ship a ZFS version with file corruption bug, which is introduced by their own patch of OpenZFS.
So if OpenZFS is serving you well, don't worry about the Ubuntu situation because there's plenty better options.
And you can install it on a lot of distro too: https://openzfs.github.io/openzfs-docs/Getting%20Started/ind...
My current favorite is to install kABI-tracking kmod on RHEL. (But their kABI-tracking is not a guarantee as they uses more.) In fact I move away from Ubuntu knowing that they knowingly ship a ZFS version with file corruption bug, which is introduced by their own patch of OpenZFS.
So if OpenZFS is serving you well, don't worry about the Ubuntu situation because there's plenty better options.
Don't hold your breath.
This is lkml business as usual.
Patches as large as this usually take a few tries to get things into shape.
That said, this isn't the first time bcachefs has been through this process. Though previous runs were mostly just an RFC for finding issues and getting input on how things are structured and seeing if there are any difficult to understand/reason about areas.
I'm not sure I expect it to end up in 6.6 just because reviews like this do take a long time anyway, but the 6.6 merge window is still a decently long time away from happening. If it doesn't make 6.6 and no major issues are found I could definitely see 6.7 being the one it's in. That'd likely mean it'd be sometime in the late fall early winter (northern hemisphere) which is still great news.
I've been looking forward to bcachefs coming in as an alternative to XFS/ext4 on the lower end of features and as an alternative for ZFS for when you don't need or want all of the stuff it has either. It looks like it should shape up to being a very capable middle ground with a lot of good stuff in it.
I'm not sure I expect it to end up in 6.6 just because reviews like this do take a long time anyway, but the 6.6 merge window is still a decently long time away from happening. If it doesn't make 6.6 and no major issues are found I could definitely see 6.7 being the one it's in. That'd likely mean it'd be sometime in the late fall early winter (northern hemisphere) which is still great news.
I've been looking forward to bcachefs coming in as an alternative to XFS/ext4 on the lower end of features and as an alternative for ZFS for when you don't need or want all of the stuff it has either. It looks like it should shape up to being a very capable middle ground with a lot of good stuff in it.
It sounds like one of the biggest issues with merging bcachefs is that the patch is huge and touches systems outside of bcachefs. I suspect if they had taken those external patches and submitted them separately they could have had incremental acceptance (or at least discussion) of those patches and we might be further along, or at least know how far along we actually are.
That's my understanding too, and I believe that's what came out of the previous RFC discussions it had before making it to this merge discussion. The parts that go outside the FS itself I believe have been split out better so that it's easier to review and I think can be taken in parts to verify that it doesn't cause issues in those external places.
Actually I'm shocked things are moving as fast as they are. I expect bcachefs to eventually end up in mainline, but only taking 2ish releases worth of time to do it would be a (pleasant!) surprise. Exciting times:)
I've been using it for the last two years and I think it has potential over btrfs, and zfs specifically. I used zfs on my hobby data array for 10 years prior but decided to downsize everything and use bcachefs in the hopes of it one day being in the kernel (unlike zfs).
Device management is super easy, but is still missing knobs and dials for the most part. Multi device fault tolerance also needs work as I have had to recreate a 16tb array twice now due to journal corruption on power outages.
I'll keep using it and reporting bugs, but I do feel the author is letting some bugs slide to focus on getting the fs in the kernel so that it gets more capable users that can provide patches. I am a sophisticated user, but definitely not a kernel developer.
Device management is super easy, but is still missing knobs and dials for the most part. Multi device fault tolerance also needs work as I have had to recreate a 16tb array twice now due to journal corruption on power outages.
I'll keep using it and reporting bugs, but I do feel the author is letting some bugs slide to focus on getting the fs in the kernel so that it gets more capable users that can provide patches. I am a sophisticated user, but definitely not a kernel developer.
> I do feel the author is letting some bugs slide to focus on getting the fs in the kernel
I'm not involved in any of this besides watching from the sidelines. But it seems to me that Kent's main focus right now is clearly getting the filesystem into the kernel so that it can get the testing and bugfixing it needs. To the point that his determination is annoying all of the veteran kernel maintainers. (Who, to be fair, have a right to be conservative on what they accept into the kernel as they have been burned many times in the past by not being conservative enough.) Even though I am sure he has grand ideas for the future, he seems to be happy enough with the current state of things but hasn't been successful in attracting outside help on his own. Which is why he is trying to drum up interest on the LKML.
I don't think it's likely that Kent is letting known bugs slide, as that would be extremely detrimental toward getting his patches merged.
I'm not involved in any of this besides watching from the sidelines. But it seems to me that Kent's main focus right now is clearly getting the filesystem into the kernel so that it can get the testing and bugfixing it needs. To the point that his determination is annoying all of the veteran kernel maintainers. (Who, to be fair, have a right to be conservative on what they accept into the kernel as they have been burned many times in the past by not being conservative enough.) Even though I am sure he has grand ideas for the future, he seems to be happy enough with the current state of things but hasn't been successful in attracting outside help on his own. Which is why he is trying to drum up interest on the LKML.
I don't think it's likely that Kent is letting known bugs slide, as that would be extremely detrimental toward getting his patches merged.
Can you tell me about the journal corruption you were seeing?
I will probably give bcachefs on one of my machines.
I started using btrfs back when it was way less stable, but that instability was more than offset by the other features.
The last time I lost a btrfs filesystem was four years ago when it was less stable, but it did not matter because it only amounted to about an hour of logs lost, and I reconstructed it in minutes. So btrfs is a must on any embedded devices. It frees me to use cheaper media, because I can have two copies of all data and metadata, and compression, and fine grained external snapshots.
Zfs is great for making big complex arrays and if you plan out how your are storing your data ahead of time. But it has a steeper learning curve and is more seperate from the underlying OS.
Btrfs fills this perfect niche for me of being a drop in replacement for ext4, and being really flexible and offering some of the nicer features of zfs. (differential snapshots, checksumming, compression)
So I'm excited to try bachefs as an alternative to btrfs.
I started using btrfs back when it was way less stable, but that instability was more than offset by the other features.
The last time I lost a btrfs filesystem was four years ago when it was less stable, but it did not matter because it only amounted to about an hour of logs lost, and I reconstructed it in minutes. So btrfs is a must on any embedded devices. It frees me to use cheaper media, because I can have two copies of all data and metadata, and compression, and fine grained external snapshots.
Zfs is great for making big complex arrays and if you plan out how your are storing your data ahead of time. But it has a steeper learning curve and is more seperate from the underlying OS.
Btrfs fills this perfect niche for me of being a drop in replacement for ext4, and being really flexible and offering some of the nicer features of zfs. (differential snapshots, checksumming, compression)
So I'm excited to try bachefs as an alternative to btrfs.
> as I have had to recreate a 16tb array twice now due to journal corruption on power outages.
This isn't a good sign... Journalling should never, if bug-free, lead to data corruption or loss even if there is a power outage.
And, even if there was some bug, one would expect a robust filesystem to say "ah, there is some data corruption here, so we're gonna run an fsck and recover every single file on the disk except perhaps the one or two that the bug clobbered."
This isn't a good sign... Journalling should never, if bug-free, lead to data corruption or loss even if there is a power outage.
And, even if there was some bug, one would expect a robust filesystem to say "ah, there is some data corruption here, so we're gonna run an fsck and recover every single file on the disk except perhaps the one or two that the bug clobbered."
ZFS and Btrfs have demonstrated devices have a variety of transient failures including maintaining write order implied by fsync or fua.
This will thwart any filesystem.
SSDs do not reliably report UNC read errors when data can't be retrieved. Garbage or zeros are returned instead.
There's a reason why ext4 and XFS added journal and metadata checksumming. Storage devices just aren't as reliable at informing the kernel when it suspects the data returned is bad.
This will thwart any filesystem.
SSDs do not reliably report UNC read errors when data can't be retrieved. Garbage or zeros are returned instead.
There's a reason why ext4 and XFS added journal and metadata checksumming. Storage devices just aren't as reliable at informing the kernel when it suspects the data returned is bad.
Incorrect write order shouldn't thwart a CoW filesystem. It can check at mount time whether the last few commits are fully there.
If the write order isn't guaranteed you can get a new super block in place without the updated trees being written. The super points to trees that don't exist.
Recent but no longer current trees, can be partly overwritten when the kernel is informed a super block write was successful. But if the super block write wasn't successful (the device lied), the stale super block on disk points to damaged metadata and recoverability isn't certain.
Recent but no longer current trees, can be partly overwritten when the kernel is informed a super block write was successful. But if the super block write wasn't successful (the device lied), the stale super block on disk points to damaged metadata and recoverability isn't certain.
You can tell if the metadata is correct by checking the hashes of everything committed by that superblock.
If it isn't correct, ignore it and move on to the previous superblock. Keep going until you can verify a contiguous 30 seconds of superblocks.
If writes are being delayed by more than 30 seconds, your problems go beyond "out of order".
This does impose the requirement not to overwrite trees that are only a minute or two old. That should not be hard.
If it isn't correct, ignore it and move on to the previous superblock. Keep going until you can verify a contiguous 30 seconds of superblocks.
If writes are being delayed by more than 30 seconds, your problems go beyond "out of order".
This does impose the requirement not to overwrite trees that are only a minute or two old. That should not be hard.
Nice to see even the graybeards on LKML are as susceptible to bikeshedding as anyone else.
Note how much heated debate is happening around how Kent is behaving, rather than the substance of his patchset. The former requires far less effort and knowledge to criticize and attack...
Note how much heated debate is happening around how Kent is behaving, rather than the substance of his patchset. The former requires far less effort and knowledge to criticize and attack...
The real feedback has already happened. Last year Kent sent out the initial patches, and the main feedback from Linus was about the Git history and locking. All the important stuff is sorted out, it's in good shape to go into the kernel.
The issue now is there are lots of maintainers who don't like that someone else did something impressive that they couldn't have done, and they are dragging their feet getting the various patches outside `fs/bcachefs` integrated. The design and implementation is basically just Kent, there are others onboarding to help with maintenance. There is a "Wireguard" level of craftsmanship in this project, it's a big deal.
A younger Linus, in another time, might have told everyone to cut the bullshit--if you can't comment on the code, shut up, you're probably incompetent--but that's not the way things work anymore.
The issue now is there are lots of maintainers who don't like that someone else did something impressive that they couldn't have done, and they are dragging their feet getting the various patches outside `fs/bcachefs` integrated. The design and implementation is basically just Kent, there are others onboarding to help with maintenance. There is a "Wireguard" level of craftsmanship in this project, it's a big deal.
A younger Linus, in another time, might have told everyone to cut the bullshit--if you can't comment on the code, shut up, you're probably incompetent--but that's not the way things work anymore.
Bcachefs does not yet implement scrubs. Corrupted data can be detected, reading instead from a copy (if available). But the problems get neither reported nor fixed. So does not seem production ready yet...
https://github.com/koverstreet/bcachefs/issues/146
https://github.com/koverstreet/bcachefs/issues/146
Another option is Dragonfly's HAMMER2[0].
0. https://www.dragonflybsd.org/hammer/
0. https://www.dragonflybsd.org/hammer/
Not on Linux. There weren't any serious attempts to port it as far as I know.
It's a bit of a shame that filesystems are so OS-specific.
It's kinda a shame that there isn't at least some FUSE layer to let you run a filesystem driver from windows/mac/bsd/whatever on linux.
It's kinda a shame that there isn't at least some FUSE layer to let you run a filesystem driver from windows/mac/bsd/whatever on linux.
FWIW FUSE kind of is that layer; a filesystem targeting FUSE should at least be trivially portable across the FOSS unix-likes.
I really wish HAMMER2 had more traction. It's BSD licensing means it has the potential to be supported on Linux, Windows, macOS, etc. (in addition to other BSDs.)
Could someone explain what is the expected main benefit over btrfs? I've been running btrfs on all my "fast&busy" disks for quite a while, including hourly snapshots etc and I've been very happy with it.
I have never really paid much attention to filesystems on Linux just installed which ever the installer is set to use, with that in mind do we need another filesystem?
Different file systems are optimized for different workloads and use cases. If you're just installing Linux on a laptop for medium-duty use, then you don't need anything special and can go with whatever general-purpose FS the distribution recommends, usually ext4. (Although I am partial to XFS myself.)
You only need to consider more specialized filesystems if you have unique needs. For example, if you are setting up a file server, you might want a filesystem that is good at disk redundancy and data durability. Or you might need a filesystem that can handle billions of small files. Or you might have a huge database that needs random access to large amounts of flash storage. All of these are very different scenarios that require different (competing) things from the filesystems.
In this case, bcachefs _seems_ to be promising many of the features of ZFS but with a better design and less overhead, potentially bringing these features to general-purpose workloads as well. (ZFS is fairly resource-intensive: in order to make full use of its features, you need a dedicated box for it, or really beef up the hardware on the application server that it will run on.)
You only need to consider more specialized filesystems if you have unique needs. For example, if you are setting up a file server, you might want a filesystem that is good at disk redundancy and data durability. Or you might need a filesystem that can handle billions of small files. Or you might have a huge database that needs random access to large amounts of flash storage. All of these are very different scenarios that require different (competing) things from the filesystems.
In this case, bcachefs _seems_ to be promising many of the features of ZFS but with a better design and less overhead, potentially bringing these features to general-purpose workloads as well. (ZFS is fairly resource-intensive: in order to make full use of its features, you need a dedicated box for it, or really beef up the hardware on the application server that it will run on.)
bcachefs supports a lot of features that would be useful for even home user desktop systems.
For example, the filesystem supports snapshots. This would let you do something like have apt (in Ubuntu) create an automatic snapshot of the filesystem before running an upgrade, so that if the upgrade fails or causes problems you can roll it back trivially.
It also supports native compression, saving users tons of disk space. It supports combining multiple devices together for redundancy, and it can checksum all of your data so that you can know if a file was corrupted without it introducing transparent failures into your system. It supports copy-on-write, so that you can make instantaneous "copies" of files or directories which take no extra space unless you change them.
So yeah, there are a lot of functionalities that we can enable with newer filesystems. ext4 is pretty great all in all, but there are a lot of modern features that modern hardware and concepts enable.
For example, the filesystem supports snapshots. This would let you do something like have apt (in Ubuntu) create an automatic snapshot of the filesystem before running an upgrade, so that if the upgrade fails or causes problems you can roll it back trivially.
It also supports native compression, saving users tons of disk space. It supports combining multiple devices together for redundancy, and it can checksum all of your data so that you can know if a file was corrupted without it introducing transparent failures into your system. It supports copy-on-write, so that you can make instantaneous "copies" of files or directories which take no extra space unless you change them.
So yeah, there are a lot of functionalities that we can enable with newer filesystems. ext4 is pretty great all in all, but there are a lot of modern features that modern hardware and concepts enable.
If you go by features, there aren't many alternatives. Unless I set up a database server, I consider checksums + snapshots + compression + reflinks to be table stakes these days. There is only one mainline filesystem that supports all of that, and it has its share of issues that are unlikely to ever be fixed, and is (probably permanently) tarnished in many users' eyes because it was pushed too early. So the question is, do we really need a second filesystem? (Yes?)
[deleted]
[deleted]
Anyone know how the compression features of bcachefs work? At what level is the compression happening?
There's plenty of information about bcachefs here: https://bcachefs.org/bcachefs-principles-of-operation.pdf
Thanks! This looks perfect.
Reading the peer comment document, the unique thing about compression with bcachefs is that it compresses extents, rather than blocks. Where a compressed and checksummed extent is 64Kb, and a block is typically 2/4/8 Kb.
So better compression at the cost of slightly reduced performance for random reads.
Edit: I was paraphrasing the text in the peer linked document:
"As with data checksumming, we compress entire extents, not individual disk blocks - this gives us better compression ratios than other filesystems, at the cost of reduced small random read performance."
So better compression at the cost of slightly reduced performance for random reads.
Edit: I was paraphrasing the text in the peer linked document:
"As with data checksumming, we compress entire extents, not individual disk blocks - this gives us better compression ratios than other filesystems, at the cost of reduced small random read performance."
That's not unique to bcachefs; it's how transparent compression is implemented in every filesystem that implements it that I'm aware of (for example, I just checked NTFS, BTRFS and ZFS).
ZFS is block based, not extent based, but blocks can be variable size from my understanding. Btrfs does compress individual blocks at 4KiB granularity, but can use the same zstd dictionary for up-to 128KiB runs of contiguous blocks. I don't know about modern NTFS but IIRC it can't compress logical blocks larger than 4KiB anyway, or couldn't a long time ago, so at that point it's kind of moot when compared to the greater ratios of the competitors.
In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice.
In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice.
> In practice being a true extent-based filesystem with compression and snapshotting is one of bcachefs's distinguishing features (snapshotting is rather obvious in a block-based system by comparison). Other systems work similarly at a high level but the details are of course quite different in practice.
How do bcachefs' extents differ from ZFS's logical blocks (whose size are controlled by the dataset's recordsize property) and how do bcachefs' blocks differ from ZFS's physical blocks (whose size granularity are controlled by the ZFS pool's ashift property)?
How do bcachefs' extents differ from ZFS's logical blocks (whose size are controlled by the dataset's recordsize property) and how do bcachefs' blocks differ from ZFS's physical blocks (whose size granularity are controlled by the ZFS pool's ashift property)?
My understanding (Kent is in this thread so maybe I'm totally wrong and he can improve on this) is that extents work like any other filesystem but taken to the extreme: they record a variable, contiguous range of bytes, and that's really all there is in bcachefs, there is no other fixed size unit of data for file data to fit into (I suppose beyond whatever is enforced by the underlying drive geometry and btree index); while a ZFS logical block has several underlying physical blocks of smaller size, but ultimately the logical block is the "unit of operation." It's kind of like the difference between your data type being single scalar number and an interval, I think. bcachefs only has extents and all data is recorded as a variable-sized extent.
In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics.
Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc.
There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later.
The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud.
Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same.
[1] https://bcachefs.org/Architecture/
In contrast, ZFS logical blocks have a fixed size, but may be handled "sparsely" by an underlying number of (smaller) physical blocks. So a 128KiB logical block doesn't mean every 4KiB file takes up 128KiB, naturally. But to change 'recordsize' in ZFS, you also need to rewrite the file, since the logical block size won't be retroactively changed. For good reason, of course. But the indirection between physical/logical makes a big difference in the underlying mechanics.
Snapshots make a lot of sense in a system that uses logical, fixed-sized blocks as the atomic "unit" of data even if there is an underlying physical block layer. For example, COW is trivial -- page is read, written and marked dirty, then written out to a new copy of the block it came from. The original logical block can be left alone, and the cost is understood as proportional to the underlying number of physical blocks rewritten (and some change to update metadata). But how does COW or snapshotting work in an extent system? In a non-COW system, you can update the data inside the extent in-place since you don't copy, but in a COW system you need to copy or potentially split the extent which could be costly. You may actually want to group extents together if they're for multiple versions of a file when doing COW. There is no layer of "indirection" above this, so that's where a lot of the trickiness comes from, I think. If you have indirect "logical pointers" instead, you can defer a lot of work, things like reclamation, fragmentation, splitting and merging, etc.
There are some other interesting design points in the notes[1] I was looking over to refresh myself, but I don't know how some of them relate to ZFS specifically. For example, another thing bcachefs is pretty good at is low tail latency, and the note on eschewing heavyweight transactions/locks is really good example of that. If you want to run 'touch file', you don't need to take a transaction lock, create an inode, then create the dirent, and release the lock. You can instead simply create the inode and then create dirent assuming that they have a proper write barrier between them, and deleting a file is the inverse operation. Then the system can never be left in an inconsistent state, it can only result in a few dangling objects in the key-value store, which can be garbage collected later.
The low tail latency is also assisted by having really small amounts of metadata for each inode, just barely a 64-byte cache line IIRC; so some really weird pathological cases like `ls` in a directory with 500k dirents is way faster in bcachefs than elsewhere. I also wonder if that also falls out from some of its basic design; it basically looks like a POSIX filesystem layered on top of a fast, ordered b-tree, so there's the storage layer and the actual filesystem semantics on top. The filesystem layer probably doesn't actually need to record that much in-band metadata for a given inode when a lot of the underlying consistency and performance issues are handled elsewhere. But that's just me thinking out loud.
Overall, there are lots of differences in the details even if some of the high-level stuff comes across as the same.
[1] https://bcachefs.org/Architecture/
Just a couple of minor corrections:
A ZFS physical block also records a variable, contiguous range of bytes on-disk, not unlike what you describe with bcachefs extents (although, in ZFS the "contiguous" part can get more complicated due to gang blocks and RAID-Z, I think).
That said, if you don't have compression enabled, then in ZFS the physical size of the block usually does correspond to the logical size, regardless of how much data was written to it (except for files smaller than one logical block size).
Also, if I remember correctly, a ZFS logical block usually only maps to one ZFS physical block, although it can map to up to 3 physical blocks, each of them having the same data (the actual number depends on how many redundant physical copies the logical block is configured to have on top of the ZFS pool redundancy).
But in any case, thank you for your long explanation. Some details still escape me, but I can consult the architecture document you linked if I have more doubts.
A ZFS physical block also records a variable, contiguous range of bytes on-disk, not unlike what you describe with bcachefs extents (although, in ZFS the "contiguous" part can get more complicated due to gang blocks and RAID-Z, I think).
That said, if you don't have compression enabled, then in ZFS the physical size of the block usually does correspond to the logical size, regardless of how much data was written to it (except for files smaller than one logical block size).
Also, if I remember correctly, a ZFS logical block usually only maps to one ZFS physical block, although it can map to up to 3 physical blocks, each of them having the same data (the actual number depends on how many redundant physical copies the logical block is configured to have on top of the ZFS pool redundancy).
But in any case, thank you for your long explanation. Some details still escape me, but I can consult the architecture document you linked if I have more doubts.
You're mistaken about NTFS, if I'm interpreting you right. It requires you to have 4KB blocks, but it compresses 64KB at a time.
ZFS compresses records (blocks) independently.
A ZFS block is 128 KB by default (smaller if the file is < 128 KB, but can also be larger if you manually configure it).
So a ZFS block is even larger than a bcachefs "extent".
So a ZFS block is even larger than a bcachefs "extent".
Well yes, my point is that (counter to GP) ZFS doesn't combine together multiple blocks into a compression extent, rather it uses larger blocks by default.
ZFS's larger blocks you are referring to are the logical block size, which is the size unit for compression among other things. I don't know much about bcachefs, but as I understand it, a logical block in ZFS is equivalent to a bcachefs extent (in terms of compression, at least) and a ZFS physical block is equivalent to a (possibly empty) set of bcachefs blocks.
As an example, in terms of actual on-disk storage, a 128 KB logical block in ZFS can be compressed down to (for example) a 3.5 KB "physical" block and it will only actually consume 3.5 KB of disk space (surprisingly, the block size of a ZFS physical block doesn't have to be a power of 2!).
That said, if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes.
So, regardless of the file size, an on-disk ZFS block can actually occupy anywhere from 0 bytes up to 1 MB, with a granularity of up to 512 bytes, depending on how much it is compressed, what the physical block size granularity of the ZFS pool is and what you've selected as the recordsize property (i.e. the logical block size) for the corresponding filesystem in the pool.
As an example, in terms of actual on-disk storage, a 128 KB logical block in ZFS can be compressed down to (for example) a 3.5 KB "physical" block and it will only actually consume 3.5 KB of disk space (surprisingly, the block size of a ZFS physical block doesn't have to be a power of 2!).
That said, if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes.
So, regardless of the file size, an on-disk ZFS block can actually occupy anywhere from 0 bytes up to 1 MB, with a granularity of up to 512 bytes, depending on how much it is compressed, what the physical block size granularity of the ZFS pool is and what you've selected as the recordsize property (i.e. the logical block size) for the corresponding filesystem in the pool.
> if you have 4 KB-sector disks then you should probably use a physical block size granularity of 4 KB for your ZFS pool rather than 512 bytes
According to Allan Jude (OpenZFS & FreeBSD developer), you should never use 512-byte recordsize even on an HDD with 512-byte sectors. Always start from 4 KiBs, probably even from 8.
https://klarasystems.com/articles/tuning-recordsize-in-openz...
According to Allan Jude (OpenZFS & FreeBSD developer), you should never use 512-byte recordsize even on an HDD with 512-byte sectors. Always start from 4 KiBs, probably even from 8.
https://klarasystems.com/articles/tuning-recordsize-in-openz...
I don't think that post is saying that.
I think that post is warning you that it's immutable once done. If you want to be more flexible in the future and can't be sure you won't need to use 4kn/512e disks at some point, yes, use 4kn (or with all SSD storage it gets even messier).
But using 512b can be a significant gain on space efficiency for some people.
I think that post is warning you that it's immutable once done. If you want to be more flexible in the future and can't be sure you won't need to use 4kn/512e disks at some point, yes, use 4kn (or with all SSD storage it gets even messier).
But using 512b can be a significant gain on space efficiency for some people.
> you should never use 512-byte recordsize even on an HDD with 512-byte sectors
I think you meant to refer to the pool's sector size (i.e. "ashift" property) rather than the recordsize.
I agree, but I wouldn't say "never", as that's too strong a word.
It would be useful to use a 512-byte ZFS pool sector size even when the HDD uses 4K sectors if you care a lot about disk space efficiency but don't care at all about performance or HDD wear (for example, if you plan to use a ZFS pool almost exclusively read-only).
That said, I would also recommend a 4K pool sector size in almost all cases, as HDD space is cheap.
I think you meant to refer to the pool's sector size (i.e. "ashift" property) rather than the recordsize.
I agree, but I wouldn't say "never", as that's too strong a word.
It would be useful to use a 512-byte ZFS pool sector size even when the HDD uses 4K sectors if you care a lot about disk space efficiency but don't care at all about performance or HDD wear (for example, if you plan to use a ZFS pool almost exclusively read-only).
That said, I would also recommend a 4K pool sector size in almost all cases, as HDD space is cheap.
I understand all of that. None of that disagrees with what I have said.
I think we're talking past each other. In a practical sense I agree the two filesystems achieve an essentially identical result through very similar methods, as you have said.
I am making a point specifically about the details of the implementation logic of the two filesystems. Bcachefs (as I understand it) combines multiple smaller blocks into a larger extent. ZFS only has one raw record and its corresponding on-disk compressed record, whose size is rounded up to ashift.
I think we're talking past each other. In a practical sense I agree the two filesystems achieve an essentially identical result through very similar methods, as you have said.
I am making a point specifically about the details of the implementation logic of the two filesystems. Bcachefs (as I understand it) combines multiple smaller blocks into a larger extent. ZFS only has one raw record and its corresponding on-disk compressed record, whose size is rounded up to ashift.
16M, not 1M.
Thanks for the correction. The `zfsprops` man page should be updated to reflect that...
It has been.
The default was to limit it to 1M in case there were weird edge cases with the larger sizes (16M is the largest it'll encode on disk without a format change, IIRC), with larger sizes behind a tunable, but after a few years of people who knew this twiddling it, the default in 2.2 will be that the tunable is set to 16M.
The default was to limit it to 1M in case there were weird edge cases with the larger sizes (16M is the largest it'll encode on disk without a format change, IIRC), with larger sizes behind a tunable, but after a few years of people who knew this twiddling it, the default in 2.2 will be that the tunable is set to 16M.
[deleted]
This is similar to how ZFS does it, and probably others, so hardly unique. In ZFS they call it "records" or "blocks" rather than "extents", and the default value record size is 128KiB.
ATM machine
Kind of sort of mostly? The big difference here is that Bcachefs is a filesystem based originally on the design of Bcache which is a block device caching layer, so it'd be very easy to miss the fs on the end and confuse Bcachefs for Bcache. Because of this, calling it Bcache filesystem also doesn't really work because it's not Bcache anymore either.
Read maybe three comments deep in the actual PR[0], and you'll get the impression this is definitely not going to ever be merged.
I've said it 1000x -- OpenZFS is already free and open source. Linux kernel developers should just stop punching themselves in face. The ZFS hair shirt is self imposed by semi-religious Linux wackadoos.
[0]: https://lore.kernel.org/lkml/20230626214656.hcp4puionmtoloat...
I've said it 1000x -- OpenZFS is already free and open source. Linux kernel developers should just stop punching themselves in face. The ZFS hair shirt is self imposed by semi-religious Linux wackadoos.
[0]: https://lore.kernel.org/lkml/20230626214656.hcp4puionmtoloat...
I've read several such LKML threads about bcachefs and while the discussions have been contentious (and Kent could stand to chill out a bit), everyone has been in agreement that they'd like to see bcachefs succeed and be merged. The major technical blockers have been addressed or are being addressed.
Bcachefs also has plenty of features which ZFS does not, so I don't see why you're so critical.
Oracle could relicense ZFS at any time, or make an explicit statement about its legal status. They have not done so after 10 years.
Bcachefs also has plenty of features which ZFS does not, so I don't see why you're so critical.
Oracle could relicense ZFS at any time, or make an explicit statement about its legal status. They have not done so after 10 years.
Oracle could relicense its own ZFS (probably?) But it couldn't relicense the ZFS everyone actually uses because they don't own the copyrights of independent contributors, some of whom oppose license changes.
[deleted]
yeah i love love love ZFS and use it everywhere and am very excited to see Bcachefs get merged. More legit competition in this space is good. Btrfs doesn't apply, it's not even in the same league.
>I've read several such LKML threads about bcachefs and while the discussions have been contentious (and Kent could stand to chill out a bit), everyone has been in agreement that they'd like to see bcachefs succeed and be merged. Bcachefs also has plenty of features which ZFS does not, so I don't see why you're so critical.
Only because I've read this story more than a few times before? See: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
>Oracle could relicense ZFS at any time, or make an explicit statement about its legal status. They have not done so after 10 years.
Any legal difficulty remains from the GPL side, and no suit has been brought in the years upon years of the GPL hair shirts screaming about non-compliance. Unfortunately, the GPL hair shirts don't have much (any) credibility on the subject of what the GPL means.
Bring the suit or STFU. I have no patience for armchair legal opinions from people who won't put their money where their mouth is. Stop spreading legal FUD!
Only because I've read this story more than a few times before? See: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
>Oracle could relicense ZFS at any time, or make an explicit statement about its legal status. They have not done so after 10 years.
Any legal difficulty remains from the GPL side, and no suit has been brought in the years upon years of the GPL hair shirts screaming about non-compliance. Unfortunately, the GPL hair shirts don't have much (any) credibility on the subject of what the GPL means.
Bring the suit or STFU. I have no patience for armchair legal opinions from people who won't put their money where their mouth is. Stop spreading legal FUD!
> Only because I've read this story more than a few times before? See: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Sorry, if we are looking at the same thing, the earliest time of a HN entry of bcachefs trying to be merged upstream was not even 1 month ago. All the other entries are related to features that are called "merge" and not actual merge requests or the author declaring that the code was up to be merged but not an actual merge request.
1 month ago is nothing in terms of kernel time, so I don't think your argument holds up.
Sorry, if we are looking at the same thing, the earliest time of a HN entry of bcachefs trying to be merged upstream was not even 1 month ago. All the other entries are related to features that are called "merge" and not actual merge requests or the author declaring that the code was up to be merged but not an actual merge request.
1 month ago is nothing in terms of kernel time, so I don't think your argument holds up.
I can't believe I have to say this to people. Before you nit pick, expand your search parameters? See: https://news.ycombinator.com/item?id=17138877
That article says "An update on bcachefs" and "it is time to start talking about getting the code upstream".
The article also makes it clear that it's not ready. And for a big patch like this, you're going to "start talking" long before actually submitting.
The only thing that supports your point is the incorrect title that was submitted to HN. But this submission got 6 upvotes and 0 comments, so almost nobody saw this title.
The article also makes it clear that it's not ready. And for a big patch like this, you're going to "start talking" long before actually submitting.
The only thing that supports your point is the incorrect title that was submitted to HN. But this submission got 6 upvotes and 0 comments, so almost nobody saw this title.
There was a previous attempt 4 years ago. It's normal that there's some pushback and things get worked on for some more time. Nothing that indicates a serious problem here. Kent worked on the filesystem overall rather than addressing the comments and resubmitting immediately.
(But that's still a single initial attempt, not "a few" like the OP claimed)
(But that's still a single initial attempt, not "a few" like the OP claimed)
Notably, Oracle does not include ZFS with Oracle Linux. As I said, they could easily clear this up, but they haven't.
I knew your pen name was familiar because I was liking your other comments in the recent RH threads. What's amusing is I previously thought RH was far too precious re: its interpretation of the GPL, especially re: ZFS.
> Notably, Oracle does not include ZFS with Oracle Linux. As I said, they could easily clear this up, but they haven't.
> Oracle could relicense ZFS at any time, or make an explicit statement about its legal status. They have not done so after 10 years.
That would make this easy, but unfortunately easy is not the case we have here. Just as RH has chosen strict compliance with the GPL re RHEL, I think strict compliance with the GPLv2 is what is required here. Someone, I wish it was RH, should say "We've made a mistake. The GPL is not a suicide pact." For instance, interpreting any object which dynamically links to a GPL licensed work as a derived work is fundamentally anti-business, anti-interoperability, and, I'd argue, anti-FOSS. Being GPL compliant does not mean blithely accepting whatever legal claptrap FSF and SFC can summon.
> Notably, Oracle does not include ZFS with Oracle Linux. As I said, they could easily clear this up, but they haven't.
> Oracle could relicense ZFS at any time, or make an explicit statement about its legal status. They have not done so after 10 years.
That would make this easy, but unfortunately easy is not the case we have here. Just as RH has chosen strict compliance with the GPL re RHEL, I think strict compliance with the GPLv2 is what is required here. Someone, I wish it was RH, should say "We've made a mistake. The GPL is not a suicide pact." For instance, interpreting any object which dynamically links to a GPL licensed work as a derived work is fundamentally anti-business, anti-interoperability, and, I'd argue, anti-FOSS. Being GPL compliant does not mean blithely accepting whatever legal claptrap FSF and SFC can summon.
OpenZFS is free and open source but that license has been determined to NOT be GPL compatible by some lawyers. If your lawyer says it is fine then go ahead and make your own distribution where you include it as a separate module. Oracle, the current owner of ZFS, is one of the most litigious companies in the tech world.
There is a bunch of explanations and legal opinions here.
https://openzfs.github.io/openzfs-docs/License.html
There is a bunch of explanations and legal opinions here.
https://openzfs.github.io/openzfs-docs/License.html
> OpenZFS is free and open source but that license has been determined to NOT be GPL compatible by some lawyers.
Oh? And let me tell you there is more than a boat load of lawyers who disagree.
See the links on page you linked to (!): http://www.networkworld.com/article/2301697/smb/encouraging-...
> There is a bunch of explanations and legal opinions here.
Did you read the text of that page or any of the linked articles? The OpenZFS position is clear: "However, there is nothing in either license that prevents distributing it in the form of a binary module or in the form of source code."
Oh? And let me tell you there is more than a boat load of lawyers who disagree.
See the links on page you linked to (!): http://www.networkworld.com/article/2301697/smb/encouraging-...
> There is a bunch of explanations and legal opinions here.
Did you read the text of that page or any of the linked articles? The OpenZFS position is clear: "However, there is nothing in either license that prevents distributing it in the form of a binary module or in the form of source code."
I'd say the main problem with the whole Oracle/ZFS situation: even if merging it is legally sound beyond a doubt, Oracle has the resources (and probably willingness) to make the lives of everyone involved absolutely miserable in the meantime and create legal pains for years and years over this until it gets finally confirmed in court.
I guess that alone could be enough to come to the conclusion that it is not worth the trouble.
I guess that alone could be enough to come to the conclusion that it is not worth the trouble.
The Oracle fears seems overblown to me. If ZFS is shipped with Linux, I believe it would be the GPL that is violated. So Oracle would have no right to sue beyond being one of the many Linux contributors.
Linux had ~5 years to try and add ZFS before it was bought by Oracle, and did not do so. As much as Oracle is often the villain, I don't think they're what the Linux team fears here.
Linux had ~5 years to try and add ZFS before it was bought by Oracle, and did not do so. As much as Oracle is often the villain, I don't think they're what the Linux team fears here.
Having lived through the SCO debacle, this right here is my main concern.
[deleted]
I would like for OpenZFS to get merged into Linux but to be more realistic, even DTrace didn't get merged and that is code that was actually relicensed to GPL (UPL for the user space): https://www.theregister.com/2018/02/19/oracle_open_sources_d...
Frankly, it is more likely, distributions will add/ solidify infrastructure for making OpenZFS really hassle-free and well integrated and cloud/ VPS providers will have alternative images built with Root-on-ZFS. Until then, it is not that hard to convert a VM from EXT4 to OpenZFS or to build your own image. Shameless plug, I have done a video about a procedure how to convert EXT4 to OpenZFS from within the VM: https://www.youtube.com/watch?v=KoRVFRufZI0 Or you can just open the notes and copy&paste from there: https://github.com/KaliszAd/ext4-to-zfs-conversion
Frankly, it is more likely, distributions will add/ solidify infrastructure for making OpenZFS really hassle-free and well integrated and cloud/ VPS providers will have alternative images built with Root-on-ZFS. Until then, it is not that hard to convert a VM from EXT4 to OpenZFS or to build your own image. Shameless plug, I have done a video about a procedure how to convert EXT4 to OpenZFS from within the VM: https://www.youtube.com/watch?v=KoRVFRufZI0 Or you can just open the notes and copy&paste from there: https://github.com/KaliszAd/ext4-to-zfs-conversion
I don't think Linus is interested in merging ZFS.
> And honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's ok to do so and treat the end result as GPL'd.
> Other people think it can be ok to merge ZFS code into the kernel and that the module interface makes it ok, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so.
https://www.realworldtech.com/forum/?threadid=189711&curpost...
> And honestly, there is no way I can merge any of the ZFS efforts until I get an official letter from Oracle that is signed by their main legal counsel or preferably by Larry Ellison himself that says that yes, it's ok to do so and treat the end result as GPL'd.
> Other people think it can be ok to merge ZFS code into the kernel and that the module interface makes it ok, and that's their decision. But considering Oracle's litigious nature, and the questions over licensing, there's no way I can feel safe in ever doing so.
https://www.realworldtech.com/forum/?threadid=189711&curpost...
To me that reads as "he would be interested, if not for the volatile nature of the copyright owner".
Thanks for the info, I know painfully little of modern filesystems: without opening too many worm cans ZFS > XFS as a general purpose ext4 replacement?
It depends, but as a rule of thumb for more traditional servers and development workstations ZFS is a superior choice. You can probably look at https://mwl.io/nonfiction/os#fmzfs to get a quick and straight forward introduction.
For me, ZFS beats other approaches with snapshots/ rollbacks and send/ receive. It really hits you, when you develop something using a database or a lot of files. If you want to test something, just do a snapshot (even without shutting down the database) and just try it out. If you mess up, you can at the very least shut the DB down, rollback (which is instant) and start the DB again. It will be in the state before you messed up. You can also make a clone and spin up another instance and compare. With files, you can even diff snapshots, so as to extract e.g. a deleted file. Also this is a great mitigation against ransomware, if your network drives are backed by ZFS. Also in the case of a database, compression and the integrity guarantees help a lot. You can do some tuning because of it that might make your performance better/ more consistent - because backups can get a lot more efficient and you can cache more data in RAM when it is compressed or buy better SSDs when they don't need to be that large. Now you can probably do something similar with btrfs, e.g. Turris Omnia is using btrfs internally and it saved me once while I was doing some broader reconfiguration. Otherwise, I don't know about it - it is probably reliable enough. You can do LVM snapshots. There are many approaches to solve a particular problem. ZFS was just designed to do some of these things and has about 15 years of production use in some of the most serious settings.
XFS or EXT4 is a decent choice for systems that you just don't want to think about. They are available by default, don't usually need any kind of tuning/ configuration. XFS is probably the better choice instead of EXT4, if you have a typical mdadm/ lvm RAID for e.g. surveillance camera recordings. For VMs that do some computation but the data is stored elsewhere, you can use the usually default EXT4 for the base filesystem. Converting such systems to ZFS might not be worth it. For the company database, filer, archive, backup ZFS should be something you evaluate as the first option. Just be sure you know a bit about how ZFS stores its data and how to configure the values for your use case. ZFS is more rigid in some places however extremely flexible in others be sure to understand where the rigid parts might be a problem to you.
For me, ZFS beats other approaches with snapshots/ rollbacks and send/ receive. It really hits you, when you develop something using a database or a lot of files. If you want to test something, just do a snapshot (even without shutting down the database) and just try it out. If you mess up, you can at the very least shut the DB down, rollback (which is instant) and start the DB again. It will be in the state before you messed up. You can also make a clone and spin up another instance and compare. With files, you can even diff snapshots, so as to extract e.g. a deleted file. Also this is a great mitigation against ransomware, if your network drives are backed by ZFS. Also in the case of a database, compression and the integrity guarantees help a lot. You can do some tuning because of it that might make your performance better/ more consistent - because backups can get a lot more efficient and you can cache more data in RAM when it is compressed or buy better SSDs when they don't need to be that large. Now you can probably do something similar with btrfs, e.g. Turris Omnia is using btrfs internally and it saved me once while I was doing some broader reconfiguration. Otherwise, I don't know about it - it is probably reliable enough. You can do LVM snapshots. There are many approaches to solve a particular problem. ZFS was just designed to do some of these things and has about 15 years of production use in some of the most serious settings.
XFS or EXT4 is a decent choice for systems that you just don't want to think about. They are available by default, don't usually need any kind of tuning/ configuration. XFS is probably the better choice instead of EXT4, if you have a typical mdadm/ lvm RAID for e.g. surveillance camera recordings. For VMs that do some computation but the data is stored elsewhere, you can use the usually default EXT4 for the base filesystem. Converting such systems to ZFS might not be worth it. For the company database, filer, archive, backup ZFS should be something you evaluate as the first option. Just be sure you know a bit about how ZFS stores its data and how to configure the values for your use case. ZFS is more rigid in some places however extremely flexible in others be sure to understand where the rigid parts might be a problem to you.
I'd really like to understand your perspective!
My current/primitive understanding:
* CDDL is 'BSD-like'- allowing the right for CDDL code to co-exist in perhaps proprietary/closed source codebase, so long as the CDDL parts remain CDDL. It does not make demands for non-CDDL parts, allowing dual-license scenarios?
* GPL demands all code in the same codebase are subject to GPL. e.g. This excludes a scenario where some parts are open source, and others proprietary/closed.
* All contributions to the Linux kernel are subject to GPL
* Say OpenZFS is merged into Linux, CDDL demands that OpenZFS parts remain CDDL?
Now the resulting codebase has more restrictions than the spirit of the CDDL allows for. Is this not a conflict?
If you could attempt to ELI5 how I'm potentially getting part of this wrong, or point me to some article, I'd appreciate it! I honestly find this stuff pretty confusing.
My current/primitive understanding:
* CDDL is 'BSD-like'- allowing the right for CDDL code to co-exist in perhaps proprietary/closed source codebase, so long as the CDDL parts remain CDDL. It does not make demands for non-CDDL parts, allowing dual-license scenarios?
* GPL demands all code in the same codebase are subject to GPL. e.g. This excludes a scenario where some parts are open source, and others proprietary/closed.
* All contributions to the Linux kernel are subject to GPL
* Say OpenZFS is merged into Linux, CDDL demands that OpenZFS parts remain CDDL?
Now the resulting codebase has more restrictions than the spirit of the CDDL allows for. Is this not a conflict?
If you could attempt to ELI5 how I'm potentially getting part of this wrong, or point me to some article, I'd appreciate it! I honestly find this stuff pretty confusing.
IANAL, but my understanding of some of the details differs:
> CDDL is 'BSD-like'- allowing the right for CDDL code to co-exist in perhaps proprietary/closed source codebase, so long as the CDDL parts remain CDDL. It does not make demands for non-CDDL parts, allowing dual-license scenarios?
It's more like halfway between GPL and BSD. Importantly, BSD licenses impose sufficiently limited conditions that BSD code can be subject to GPL conditions at the same time without violating either license. CDDL makes mutually exclusive demands with GPL.
(But yes, CDDL does allow for mixed CDDL+proprietary systems; that's what it was created for. Sun wanted to open source Solaris, but needed to be able to keep some parts of it closed-source (because they didn't actually own all the code, having licensed some of it from third parties).)
> All contributions to the Linux kernel are subject to GPL
There's actually non-GPL code in there, but it's all under GPL compatible licenses, so the final result is GPL. But for instance, if someone wanted to put in the work and there was interest, you could almost certainly get HAMMER2 from dragonfly BSD included into Linux because it'll be under a BSD license that's GPL compatible.
> Now the resulting codebase has more restrictions than the spirit of the CDDL allows for. Is this not a conflict?
Yes, that's basically the conflict. Note that there is some difference of opinion on the situation; Canonical, for instance, seems to believe that they can ship ZFS kernel modules under the CDDL without making them part of the GPL kernel and thus not mix the licenses. Others say that compiling the ZFS modules against the kernel makes the result a derivative work of the kernel and therefore is a conflict. Until someone actually tries to file a lawsuit over it, I don't know that we definitively know.
> CDDL is 'BSD-like'- allowing the right for CDDL code to co-exist in perhaps proprietary/closed source codebase, so long as the CDDL parts remain CDDL. It does not make demands for non-CDDL parts, allowing dual-license scenarios?
It's more like halfway between GPL and BSD. Importantly, BSD licenses impose sufficiently limited conditions that BSD code can be subject to GPL conditions at the same time without violating either license. CDDL makes mutually exclusive demands with GPL.
(But yes, CDDL does allow for mixed CDDL+proprietary systems; that's what it was created for. Sun wanted to open source Solaris, but needed to be able to keep some parts of it closed-source (because they didn't actually own all the code, having licensed some of it from third parties).)
> All contributions to the Linux kernel are subject to GPL
There's actually non-GPL code in there, but it's all under GPL compatible licenses, so the final result is GPL. But for instance, if someone wanted to put in the work and there was interest, you could almost certainly get HAMMER2 from dragonfly BSD included into Linux because it'll be under a BSD license that's GPL compatible.
> Now the resulting codebase has more restrictions than the spirit of the CDDL allows for. Is this not a conflict?
Yes, that's basically the conflict. Note that there is some difference of opinion on the situation; Canonical, for instance, seems to believe that they can ship ZFS kernel modules under the CDDL without making them part of the GPL kernel and thus not mix the licenses. Others say that compiling the ZFS modules against the kernel makes the result a derivative work of the kernel and therefore is a conflict. Until someone actually tries to file a lawsuit over it, I don't know that we definitively know.
> If you could attempt to ELI5 how I'm potentially getting part of this wrong, or point me to some article, I'd appreciate it!
Suffice to say: a shallow reading of the GPLv2 might lead one to believe that the CDDL and the GPLv2 are incompatible. And I believe SFC and FSF's readings of the GPLv2 and copyright law, which the GPL incorporates by reference, are shallow. There are quite a few reasons to believe a CDDL and ZFS combination would not be incompatible. If you want learned legal opinions expressing my/this contrary view there are a few. [0], [1], [2]. Moreover, I'd warn you that SFC and FSF are not disinterested parties. They have agendas which extend beyond whether ZFS is actually incompatible to what that might mean for the GPL.
If you want my unschooled personal understanding -- most generally, there is no reason to believe that dynamic linking creates a derived work. Without further reasoning, FSF's analysis obliterates this boundary without reference to case law, and without an argument as why this must be the case. Whereas I think there are sound copyright law reasons to think where one creates an API boundary, through the use of dynamic linking or otherwise (a modular kernel interface), fair use allows one to make use of software at that boundary.
This is a difficult pill to swallow for many FOSS advocates because this would mean closed source software modules would be permissible in combination with the Linux kernel. And while I understand their political reasoning for disfavoring this view, I think their legal reasoning is weak. I also think the FSF view makes it much harder to build software which interoperates with other software, which BTW should be a key goal of the FSF and FOSS advocates.
I'm just not sure your copyright license has this kind of power. Imagine you build some software which links to MegaEvil Corp's libc, whose license specifically states, if you link to this libc, your work is a derived work of MegaCorp libc, you forfeit all copyright to MegaEvil Corp. My feeling is any court would say building any such software at an API boundary is fair use and copyright law simply does not grant any court the power to enforce this copyright term.
More specifically re: ZFS module and the GPL, I'd dig into what is a "derived work". The GPLv2 after all incorporates the copyright term by reference at Section 0 when describing a "work based on the Program". I'd consider the ambiguity of what is the "whole" work as described in Section 2 and how that might be construed in light of the long held copyright distinction between derived and collective works. I'd also consider recent copyright jurisprudence when asking whether any court would take the view that distributing the Linux kernel and ZFS module is a copyright violation. [3], [4].
[0]: https://softwarefreedom.org/resources/2016/linux-kernel-cddl... [1]: http://www.networkworld.com/article/2301697/smb/encouraging-... [2]: https://www.linuxjournal.com/article/6366 [3]: https://en.wikipedia.org/wiki/Sega_v._Accolade [4]: https://en.wikipedia.org/wiki/Lewis_Galoob_Toys,_Inc._v._Nin....
Suffice to say: a shallow reading of the GPLv2 might lead one to believe that the CDDL and the GPLv2 are incompatible. And I believe SFC and FSF's readings of the GPLv2 and copyright law, which the GPL incorporates by reference, are shallow. There are quite a few reasons to believe a CDDL and ZFS combination would not be incompatible. If you want learned legal opinions expressing my/this contrary view there are a few. [0], [1], [2]. Moreover, I'd warn you that SFC and FSF are not disinterested parties. They have agendas which extend beyond whether ZFS is actually incompatible to what that might mean for the GPL.
If you want my unschooled personal understanding -- most generally, there is no reason to believe that dynamic linking creates a derived work. Without further reasoning, FSF's analysis obliterates this boundary without reference to case law, and without an argument as why this must be the case. Whereas I think there are sound copyright law reasons to think where one creates an API boundary, through the use of dynamic linking or otherwise (a modular kernel interface), fair use allows one to make use of software at that boundary.
This is a difficult pill to swallow for many FOSS advocates because this would mean closed source software modules would be permissible in combination with the Linux kernel. And while I understand their political reasoning for disfavoring this view, I think their legal reasoning is weak. I also think the FSF view makes it much harder to build software which interoperates with other software, which BTW should be a key goal of the FSF and FOSS advocates.
I'm just not sure your copyright license has this kind of power. Imagine you build some software which links to MegaEvil Corp's libc, whose license specifically states, if you link to this libc, your work is a derived work of MegaCorp libc, you forfeit all copyright to MegaEvil Corp. My feeling is any court would say building any such software at an API boundary is fair use and copyright law simply does not grant any court the power to enforce this copyright term.
More specifically re: ZFS module and the GPL, I'd dig into what is a "derived work". The GPLv2 after all incorporates the copyright term by reference at Section 0 when describing a "work based on the Program". I'd consider the ambiguity of what is the "whole" work as described in Section 2 and how that might be construed in light of the long held copyright distinction between derived and collective works. I'd also consider recent copyright jurisprudence when asking whether any court would take the view that distributing the Linux kernel and ZFS module is a copyright violation. [3], [4].
[0]: https://softwarefreedom.org/resources/2016/linux-kernel-cddl... [1]: http://www.networkworld.com/article/2301697/smb/encouraging-... [2]: https://www.linuxjournal.com/article/6366 [3]: https://en.wikipedia.org/wiki/Sega_v._Accolade [4]: https://en.wikipedia.org/wiki/Lewis_Galoob_Toys,_Inc._v._Nin....
If I had to choose between Linux and Oracle to determine which one is operating in bad faith, I most certainly would not be thinking that it was Linux.
> If I had to choose between Linux and Oracle to determine which one is operating in bad faith, I most certainly would not be thinking that it was Linux.
I just want to pause for a moment to explain how unhinged this reply is. First, never said Linux (or Oracle for that matter) was operating in bad faith. Second, my comment calls ZFS a "hair shirt" for the kernel. A "hair shirt" is an uncomfortable undergarment which religious people/GPL fanatics use to punish themselves.
I just want to pause for a moment to explain how unhinged this reply is. First, never said Linux (or Oracle for that matter) was operating in bad faith. Second, my comment calls ZFS a "hair shirt" for the kernel. A "hair shirt" is an uncomfortable undergarment which religious people/GPL fanatics use to punish themselves.
ZFS people can just change license that was explicity written to be GPL hostile to something non-retarded
This is a myth that keeps coming up, in spite of being debunked time and time and time again. Solaris could not use the GPL because Sun didn't own the rights to all parts, so the GPL's copyleft wouldn't work and they needed a new license. When OpenSolaris was released it wasn't 100% free/open source (some parts were later rewritten under OpenSolaris, and illumos completed that and is 100% free/open source).
In hindsight, an explicit "GPL compatibility clause" like the EUPL has would clearly been a wiser choice, but that wasn't clear in 2005, and it's too late for that now, unfortunately.
On the other hand, you could also blame the GPL for not recognizing that other copyleft licenses exist; they also could have written a compatibility clause in GPL 3. "CDDL is incompatible with the GPL" could also be phrased as "GPL is incompatible with the CDDL".
In hindsight, an explicit "GPL compatibility clause" like the EUPL has would clearly been a wiser choice, but that wasn't clear in 2005, and it's too late for that now, unfortunately.
On the other hand, you could also blame the GPL for not recognizing that other copyleft licenses exist; they also could have written a compatibility clause in GPL 3. "CDDL is incompatible with the GPL" could also be phrased as "GPL is incompatible with the CDDL".
Thank you for fighting the good fight out here in the wilds of the internet! It's so frustrating that the misperceptions on this seem to be uncorrectable -- and it's to the point that I have sadly concluded that the false narrative around this somehow being a CDDL issue is one that people actually want to believe because to know the truth would be to confront Linux's own NIH syndrome, which remains profound.
> I have sadly concluded that the false narrative around this somehow being a CDDL issue is one that people actually want to believe
Yes, I agree; there's a few other topics like this. I call this "hacker conspiracy theories", because at this point that's basically what they are.
I mostly comment on these things for the benefit of innocent onlookers, who were not yet around when all of this went down, and not to convince the person who posted it.
Yes, I agree; there's a few other topics like this. I call this "hacker conspiracy theories", because at this point that's basically what they are.
I mostly comment on these things for the benefit of innocent onlookers, who were not yet around when all of this went down, and not to convince the person who posted it.
No, they literally cannot. The current OpenZFS codebase is the combination of the original Sun/Oracle-owned code and a massive number of new committers. Now, Oracle could relicense the original code, but they can't do anything about the later additions, because they don't own the copyright on those. The only maybe possible carve out is if the code is licensed with something like "CDDL 1.0 or later" then they could make a new version, but I'm not clear on whether that's an option or whether the code is licensed CDDL 1.0-only. IANAL, YMMV. (EDIT: Apparently the CDDL already has a version 1.0 and 1.1? So take exact versions with a grain of salt above.)
If ZFS was re-licensed to be compatible with the GPL, then I'm sure that it would get accepted into the kernel (assuming the code is tidied up as necessary). Now it can be argued that CDDL and GPL are compatible, but that's a risk and Oracle have been known to be litigious whenever it suits them.
From: https://opensource.stackexchange.com/questions/2094/are-cddl...
> The CDDL is a license that does not try to enforce restrictions that would not stand in court but disallows relicensing of code. The fact that on the other side, the CDDL explicitly limits its scope to the files that contain CDDL'd code, allows us make our first conclusion:
> Code under the CDDL and code under the GPL cannot be mixed in a single file.
>The GPL is a license that limits its scope to the so called "work" limit and forbids relicensing of code. This allows us to make the next conclusion:
> Code under the CDDL and code under the GPL cannot be mixed in a single work.
So, that would seem to leave two options - either re-license Linux under a CDDL-compatible license or re-license ZFS under a GPL-compatible license. Clearly re-licensing ZFS would be much easier as it's controlled by a single entity - Oracle.
My opinion is that Oracle are acting in bad faith (probably a default assumption with Oracle to be fair) and deliberately don't want the code to be in Linux.
From: https://opensource.stackexchange.com/questions/2094/are-cddl...
> The CDDL is a license that does not try to enforce restrictions that would not stand in court but disallows relicensing of code. The fact that on the other side, the CDDL explicitly limits its scope to the files that contain CDDL'd code, allows us make our first conclusion:
> Code under the CDDL and code under the GPL cannot be mixed in a single file.
>The GPL is a license that limits its scope to the so called "work" limit and forbids relicensing of code. This allows us to make the next conclusion:
> Code under the CDDL and code under the GPL cannot be mixed in a single work.
So, that would seem to leave two options - either re-license Linux under a CDDL-compatible license or re-license ZFS under a GPL-compatible license. Clearly re-licensing ZFS would be much easier as it's controlled by a single entity - Oracle.
My opinion is that Oracle are acting in bad faith (probably a default assumption with Oracle to be fair) and deliberately don't want the code to be in Linux.
> So, that would seem to leave two options - either re-license Linux under a CDDL-compatible license or re-license ZFS under a GPL-compatible license. Clearly re-licensing ZFS would be much easier as it's controlled by a single entity - Oracle.
Oracle's ZFS code is controlled by a single entity, but OpenZFS (which is what you probably want) is a community fork with copyrights owned by a huge list of people and companies.
Oracle's ZFS code is controlled by a single entity, but OpenZFS (which is what you probably want) is a community fork with copyrights owned by a huge list of people and companies.
Oh good we have StackExchange being used as a legal authority.
The point is that legal opinions differ on whether CDDL and GPL are compatible, but Oracle are well known for weaponising legal ambiguities.
Edit: Ultimately unless it goes to court, all you can have are legal opinions - there is no legal authority.
Edit: Ultimately unless it goes to court, all you can have are legal opinions - there is no legal authority.
[deleted]
Thanks for clarifying "hair shirt". I'd seen you mention it at least 3 times previously in comments, and was trying to unpack it without doing a search. Thoughts such as "they're quite fond of stylish hair and t-shirts?" and "they like t-shirts with hair designs?". I think my worst take was "they fetishize wearing someone else's chest hair, like a shirt?"
Popped onto DDG just now, and the explains right there. Also "cilice" within the first few links.
Popped onto DDG just now, and the explains right there. Also "cilice" within the first few links.
I just scrolled a little further in the DDG results to see a hit on Amazon.
After my unexpected laughter, I had to click. The first 2 hits align with my prior misunderstanding - cue more laughter. One is a t-shirt with a graphic of a very hairy chest/abdomen, the other has the likeness of Chewbacca with "Messy hair / don't care".
No need for anyone else to see for yourselves; saved you a click.
After my unexpected laughter, I had to click. The first 2 hits align with my prior misunderstanding - cue more laughter. One is a t-shirt with a graphic of a very hairy chest/abdomen, the other has the likeness of Chewbacca with "Messy hair / don't care".
No need for anyone else to see for yourselves; saved you a click.
Eg. I could have an SSD and a slow hard disk. Data is written to the SSD first, and eventually mirrored to the hard disk.
If the SSD fails, the hard disk will still be fully consistent up to the point the mirroring process got to. That mirroring would never be allowed to get more than say 5 minutes behind (writes would be throttled if that ever happened).
This lets you have performance and resilience while paying less money. The scheme can be extended with erasure codes to say 32 drives for a huge database so that you could for example sustain any 3 device failures not losing any fsynced bytes of data, or up to 6 failures but you might lose up to 30 seconds of data.
I think many multi-device filesystems could support this mode, but I don't think any do.