Is ZFS a suitable replacement for other Linux filesystems?(clusterhq.com)
clusterhq.com
Is ZFS a suitable replacement for other Linux filesystems?
https://clusterhq.com/blog/zfs-suitable-replacement-file-systems/
17 comments
This issue was fixed in Illumos a few months ago, although the fix has not made its way to O3X yet:
https://github.com/illumos/illumos-gate/commit/4bb7380495217...
The workaround is to find a file that was written that is outside of a snapshot and truncate it. `zfs diff` can be used to do that. That allows you to delete an old snapshot.
https://github.com/illumos/illumos-gate/commit/4bb7380495217...
The workaround is to find a file that was written that is outside of a snapshot and truncate it. `zfs diff` can be used to do that. That allows you to delete an old snapshot.
I ported this fix to ZoL and opened a pull request to get it merged into HEAD for the next release:
https://github.com/zfsonlinux/zfs/pull/2784
O3X merges fixes from ZoL, so it should get this fix soon after it has been merged to ZoL.
https://github.com/zfsonlinux/zfs/pull/2784
O3X merges fixes from ZoL, so it should get this fix soon after it has been merged to ZoL.
That's great! Thanks for the info.
edit: Didn't realize you meant you just ported it over and submitted the PR just now. Double thanks!
edit: Didn't realize you meant you just ported it over and submitted the PR just now. Double thanks!
There is work in progress to add the ability to remove a device from a pool. So, hopefully in the not too distant future, that work will land in illumos and migrate to all the downstream implementations (e.g. linux, mac, and freebsd).
I have heard that ZFS (at least, ZFS on Linux) requires about 1 GB of memory for every 1 TB of storage. Is this an accurate statement? That's certainly a critical flaw for many use cases, though completely irrelevant for many others. If accurate, what's behind this requirement?
On ZFS, all file data is stored using B-trees, where the leaves store X bytes, where X is the value of recordsize at file creation. When a write is done to a file in a dataset with dedup=on, a lookup is done on the data deduplication table. If the lookup finds an entry, it increments a reference. If it does not find an entry, it creates one. This involves 3 random seeks and consequentially your total write throughput is [average record size * IOPS / 3 random seeks]. If you have sufficient memory that the entire DDT can stay cached, then we avoid this limit.
The 1GB of RAM for every 1TB of storage is a rule of thumb for avoiding this limit that someone made a very long time ago. Unfortunately, that rule is wrong because it is impossible to estimate memory requirements by such a simple rule, but it has stuck with us for years despite being wrong.
The amount of memory needed to store a DDT is a function of the average record size. If your dataset has the default recorsize=128K (e.g. you are storing many >1MB files), then you can multiply your system memory by 153.6 to determine the total amount of unique data that you can store before hitting the limit I described. If you are storing many small <128KB files, then you need to calculate the average file size and then calculate [average file size * system memory * 12 / 10355] to obtain the total amount of unique data. This assumes the module settings for zfs_arc_max and zfs_arc_meta_max were left at their ZoL defaults. It is important to keep in mind that the unique data is different from total data, especially in the case of duplicate files. It also excludes metadata required for maintaining indirect block trees, dnodes (i.e. inodes), directories and the DDT itself.
The total amount of data that you can store on a pool before hitting the limit is [unique data * deduplication multipler], where unique data is what I described how to calculate and the deduplication multipler is a number that is either 1 or greater. A measure of the deduplication multipler is provided by `zpool list` as DEDUP, so you should be able to see that yourself. If your pool has data that was written without dedup=on, then any duplicates in that data will be counted as unique data for the purposes of that calculation. To provide a simple example of the deduplication multipler, imagine a pool with only two files that store the same data. The deduplication multipler for that pool would be 2, provided that both were written with dedup=on set on their dataset. If one or both were written with dedup=off, then the deduplication multiper would be 1. You can use zdb to calculate the theoretical deduplication statistics for an entire pool by running `zdb -D $POOLNAME`. Note that this will require significant memory because it constructs a full DDT in userland memory.
The 1GB of RAM for every 1TB of storage is a rule of thumb for avoiding this limit that someone made a very long time ago. Unfortunately, that rule is wrong because it is impossible to estimate memory requirements by such a simple rule, but it has stuck with us for years despite being wrong.
The amount of memory needed to store a DDT is a function of the average record size. If your dataset has the default recorsize=128K (e.g. you are storing many >1MB files), then you can multiply your system memory by 153.6 to determine the total amount of unique data that you can store before hitting the limit I described. If you are storing many small <128KB files, then you need to calculate the average file size and then calculate [average file size * system memory * 12 / 10355] to obtain the total amount of unique data. This assumes the module settings for zfs_arc_max and zfs_arc_meta_max were left at their ZoL defaults. It is important to keep in mind that the unique data is different from total data, especially in the case of duplicate files. It also excludes metadata required for maintaining indirect block trees, dnodes (i.e. inodes), directories and the DDT itself.
The total amount of data that you can store on a pool before hitting the limit is [unique data * deduplication multipler], where unique data is what I described how to calculate and the deduplication multipler is a number that is either 1 or greater. A measure of the deduplication multipler is provided by `zpool list` as DEDUP, so you should be able to see that yourself. If your pool has data that was written without dedup=on, then any duplicates in that data will be counted as unique data for the purposes of that calculation. To provide a simple example of the deduplication multipler, imagine a pool with only two files that store the same data. The deduplication multipler for that pool would be 2, provided that both were written with dedup=on set on their dataset. If one or both were written with dedup=off, then the deduplication multiper would be 1. You can use zdb to calculate the theoretical deduplication statistics for an entire pool by running `zdb -D $POOLNAME`. Note that this will require significant memory because it constructs a full DDT in userland memory.
No, that's flat out not true.
I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.
I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.
How much memory does ZFS (and/or ZoL) need per 1TB of storage when dedup is off?
Also, bup ("it backs things up!") efficiently dedups across an ssh connection (using bloom filters) Scales are differrent, but it might work for ZFS as well.
Also, bup ("it backs things up!") efficiently dedups across an ssh connection (using bloom filters) Scales are differrent, but it might work for ZFS as well.
I answered this question here:
https://news.ycombinator.com/item?id=8437921
https://news.ycombinator.com/item?id=8437921
The recommendations I've read say you need 1GB RAM for system use (assuming a dedicated file server), and then as much RAM (ideally ECC RAM) as you want to give it for caching data.
If you're short on RAM (sub 4GB), you might need to change some of the default settings to avoid problems, but RAM's fairly cheap nowadays, so unless you've got an old machine, it's not likely to be a problem :)
If you're short on RAM (sub 4GB), you might need to change some of the default settings to avoid problems, but RAM's fairly cheap nowadays, so unless you've got an old machine, it's not likely to be a problem :)
I'd point out that cheap ram is about $10/GB right now, and a cheap HDD is $30/TB. So if you need an extra GB of ram for each TB of storage, you're increasing storage costs by a third.
Ah ha! Thanks. Data deduplication. Well, yeah, that makes sense to me. Thank you very much for clearing that up.
No problem. It's really a shame to have such a high bar for using dedup. It either fits a given workload extremely well, or can be extremely detrimental.
There's been talk in the developer community about ways to address the usability of dedup, but so far nothing has gone further than small prototypes.
There's been talk in the developer community about ways to address the usability of dedup, but so far nothing has gone further than small prototypes.
Setting zfs_dedup_prefetch=0 has been found to help systems using ZFS data deduplication. There is a patch in ZoL HEAD for the next release that makes zfs_dedup_prefetch=0 by default:
https://github.com/zfsonlinux/zfs/commit/0dfc732416922e1dd59...
Aside from that, you are right that there has not been much done here. Making data deduplication more performant is a difficult task and so far, no one has proposed improvements beyond trivial changes.
https://github.com/zfsonlinux/zfs/commit/0dfc732416922e1dd59...
Aside from that, you are right that there has not been much done here. Making data deduplication more performant is a difficult task and so far, no one has proposed improvements beyond trivial changes.
I think for many or most use-cases, it would make more sense to have "off-line" deduplication (like, I believe, BTRFS does), as you can free-up space on-demand, when you judge it would yield you the most benefit and the system is the least busy.
I'm not sure how much benefit compared to the "real time" deduplication this approach can provide however, as the mapping table would still need to exist in memory, but I think there should be an increase to the write performance of non-duplicate data.
PS. My use case is that I have a few dozens of (linux-vserver) gentoo containers that obviously share many files and, unfortunately, trying to maintain a shared read-only mount of the core system doesn't seem to be practical/viable (as it does i.e. for FreeBSD jails, due to mostly the clear system seperation). The waste is not significant enough to really bother me, it would just be nice to avoid. The solutions that I am aware of are (not sure if I'm missing any, I'd be happy to be pointed to something else, if there is):
- integrated FS-level deduplication (ZoL, BTRFS)
- higher-level deduplication (lessfs and opendedup)
- hard-linking scripts (obviously at the file-level)
I'm not sure how much benefit compared to the "real time" deduplication this approach can provide however, as the mapping table would still need to exist in memory, but I think there should be an increase to the write performance of non-duplicate data.
PS. My use case is that I have a few dozens of (linux-vserver) gentoo containers that obviously share many files and, unfortunately, trying to maintain a shared read-only mount of the core system doesn't seem to be practical/viable (as it does i.e. for FreeBSD jails, due to mostly the clear system seperation). The waste is not significant enough to really bother me, it would just be nice to avoid. The solutions that I am aware of are (not sure if I'm missing any, I'd be happy to be pointed to something else, if there is):
- integrated FS-level deduplication (ZoL, BTRFS)
- higher-level deduplication (lessfs and opendedup)
- hard-linking scripts (obviously at the file-level)
Cloned snapshots carcass good way to dediplicate similar FS trees and have no RAM overhead. Its functionally the same as hardlinking.
Thanks, you mean LVM2/ZFS/BTRFS based snapshots right? I've seen that mentioned but I was thinking that eventually, with system updates, the snapshots will end up having less and less common blocks with their original source, so I would have to frequently recreate new snapshots from a more similar source and then copy the unique data on top of them again to raise the hit rate -but that sounds a pretty inconvenient thing to do.
another anecdote: I've only heard this mentioned specifically about using de-duplication.
All the extra stuff ZFS does, and does performantly, comes at a cost. That's the cost (I can't vouch for the amount.. but the principle is right. CPU power matters too.)
You don't get all those features for free... but you do get them.
You don't get all those features for free... but you do get them.
While this is true, ZFSOnLinux has made progress on reducing CPU overhead. Additional improvements will be in the next release. In particular, the performance of NFS exports of ZFS datasets will increase considerably (some are seeing a factor of 2 increase) while CPU utilization has dropped:
https://github.com/zfsonlinux/spl/pull/369#issuecomment-5839...
https://github.com/zfsonlinux/spl/pull/369#issuecomment-5839...
I wrote the above article. I will try to answer any questions that I see here.
According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...
Oracle advice on how to configure ZFS and Oracle Database
http://www.oracle.com/technetwork/server-storage/solaris/con...
Some picks:
Free space is important 10-20%.
>In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database.
>The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size
http://www.oracle.com/technetwork/server-storage/solaris/con...
Some picks:
Free space is important 10-20%.
>In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database.
>The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size
Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.
As another commenter noted it's not a complete durability solution. However it is useful in a number of places - and not just to do with durability.
AIO is also very extensively used, and bmap is pretty important.
It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important.
I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
AIO is also very extensively used, and bmap is pretty important.
It's really important when writing comparison pages like this to not play down the weaknesses of the thing you're trying to sell - it comes across badly. O_DIRECT, AIO and to a lesser extent bmap are feature that are required of a general purpose linux filesystem. I'm sure you'll have them all working in ZoL soon enough, but please don't try to pretend they aren't important.
I'm also not sure why you have NFS in there - network filesystems are not local filesystems, and aren't generally going to have identical feature-sets. It feels a bit like you just stuck NFS in there so that there'd at least be something with more missing features. The same could kind of be said of JFS - I haven't heard of anyone using JFS on linux in a serious way in a very long time.
AIO is in HEAD for the next release. I implemented it myself.
As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS.
As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good.
As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).
As for bmap, I can say unequivocally that bmap will never be supported in ZFS. bmap is a nonstandard extension that is incompatible with filesystems that span multiple disks, incompatible with network filesystems that have no local disks and incompatible when doing writes to filesystems that support checksums. ZFS is disqualified from supporting bmap by virtue of #1 and #3. The same is true for others such as btrfs and NFS.
As for O_DIRECT, there are no clearly defined semantics for O_DIRECT other than how it is implemented in XFS. My understanding of O_DIRECT is that the I/O is intended to go straight to/from the underlying disk. A true implementation of those semantics cannot be implemented in filesystem that spans multiple disks, CoW or checksums. A lesser form could probably be implemented where we try to make I/Os using O_DIRECT operate as if they were run with primarycache=metadata when primarycache=data and secondarycache=metadata when secondarycache=data. A third option would be to just ignore the O_DIRECT flag. A fourth would be to let users choose between the second and third. We might support that in the future, but I am inclined to think it will do more harm than good.
As for why NFS is in the VFS API support chart, NFS is least common denominator for VFS API support that is widely deployed. Nearly all software that works on NFS can be known to work on other filesystems. Those using the chart for its intended purpose can quickly deduce whether software will likely run on ZFSOnLinux when they know that it already works on NFS. The sole exceptions are software that uses O_DIRECT or NFSv4 ACLs. The former can typically be toggled on/off in software that supports it while the latter can be dismissed with the knowledge that it runs on NFSv3 (or any other Linux filesystem at the present time).
O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used.
O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.
Bypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
I can't read the tables on my phone as can't shrink them down (Firefox Android)... So as you don't seem to spell it out is it the case that SElinux should be fine on ZFS but PAX might not be?
They both work, with two caveats:
1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release:
https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411
The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014.
The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release:
https://github.com/zfsonlinux/spl/pull/395
2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release:
https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...
1. ZoL is currently incompatible with kernels built with either PAX_USERCOPY_SLABS or PAX_KERNEXEC_PLUGIN_METHOD_OR. The next release should fix PAX_USERCOPY_SLABS and might fix PAX_KERNEXEC_PLUGIN_METHOD_OR. The former should be fixed by a kmem-rework that the project plans to merge before the next release:
https://github.com/zfsonlinux/spl/pull/369 https://github.com/zfsonlinux/zfs/pull/2411
The fix requires both pull requests, but I have yet to refresh the second pull request with the fix. I plan to refresh it before I leave for LinuxCon Europe 2014.
The latter might be fixed by changes to drop support for pre-2.6.32 kernels that the project lead would prefer to merge for the next release:
https://github.com/zfsonlinux/spl/pull/395
2. There is a bug specific to ZoL where unlinking a file with a directory extended attribute (Solaris-style) will trigger a full traversal of all objects (i.e. inodes) within a dataset. It manifests itself as the zfs_iput_taskq kernle thread consuming excessive CPU time. It is fixed for the next release:
https://github.com/zfsonlinux/zfs/commit/0d5c500d6cf4dd9e133...
Who offers production Level support? Certification? No one AFAICS. I wonder why you didn't mention that.
It was not the purpose of the blog post. Anyway, you mentioned a couple of different things here. First, there are a few companies that offer production level support. SoftNAS is one. For full disclosure, my 2014 tax return will include consulting fees earned from fixing bugs for SoftNAS. It is in use at various large companies, including Netflix and Boeing:
http://www.softnas.com/
Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS:
http://www.pogostorage.com/products/nexenta/overview/whycert...
They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support:
http://www.pogostorage.com/products/pogostor/pogostor-overvi...
As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki:
http://open-zfs.org/wiki/Hardware
http://www.softnas.com/
Another is PogoStor, which advertises that the hardware that they sell has been certified by Nexenta for use with ZFS:
http://www.pogostorage.com/products/nexenta/overview/whycert...
They also have their own Linux distribution based on ZoL, for which I assume that they offer commercial support:
http://www.pogostorage.com/products/pogostor/pogostor-overvi...
As for certification, it is not clear if you mean hardware certification or system administrator certification. I am not aware of any filesystems that have system administrator certifications. As for hardware certifications, ZFS is designed for use with commodity hardware. It probably does not make sense to try to certify it for the full range of hardware it supports (although a subset is certified by Nexenta). Specific hardware guidelines are available on the official Open ZFS wiki:
http://open-zfs.org/wiki/Hardware
ixsystems also provides support for ZFS on FreeNAS, which is based on FreeBSD.
[deleted]
The license is pretty concerning. If anyone using ZFS sues Oracle for a patent issue, then Oracle can revoke their license to use ZFS in 60 days. (Not a lawyer, my interpretation might be wrong.)
Probably OK for home use, but not much more. Even a small startup with no patents could end up getting bought by a larger company that does have patents. Then that company could be vulnerable, perhaps even if they are trying to use their patents defensively.
Probably OK for home use, but not much more. Even a small startup with no patents could end up getting bought by a larger company that does have patents. Then that company could be vulnerable, perhaps even if they are trying to use their patents defensively.
ZFS on Linux requires more memory than most other FSes, due to the Solaris compatibility layer. The cache is a bit less responsive to low memory conditions.
I run it on a NAS with 4G of memory, and I've had to limit usage and flush caches occasionally, otherwise I get to the point that I can't run new programs.
I run it on a NAS with 4G of memory, and I've had to limit usage and flush caches occasionally, otherwise I get to the point that I can't run new programs.
For private use, yes. I've been using ZoL for over a year on my home server with a simple ZFS raid pool and a number of volumes. Relatively stable. Snapshots every 15 minutes or so. Other than a few briefly scary kernel upgrades - nothing actually broke, just had to rebuild modules. Anytime your system comes up without pools mounted, it's scary. If only the ZFS license were compatible. Anyway, those weren't real breakages, so no problems.
I have not used zfs because when I read about it I found the expand-contract procedure overly complicated. (1) Caveats when planning for expansion should be addressed.
(1) http://arstechnica.com/information-technology/2014/02/ars-wa...
(1) http://arstechnica.com/information-technology/2014/02/ars-wa...
No, it doesn't has shrink, a future you want on a desktop imo. And occasionally on server when the customer doesn't provide enough storage (on the account of being cheap) and you have move things around to get things done.
Not being an expert in this kind of area, I would really like to read a short summary. Is there one and I just have missed it, somewhere in the article? I didn't see a headline like summary or conclusions.
Nearly all software works on ZoL. A small number of applications will likely need configuration of the application(s) to turn off AIO (until the next release), configuration of the application(s) to turn off DirectIO, or setting acltype=posix on the dataset to provide POSIX ACLs.
zfs is a great FS, which add many modern concept such as zpool, and also put volume manager and fs together. storage and fs should be easy as this way. and it's in good quality on Solaris, I miss Solaris.
try one of the Illumos distributions like OmniOS or SmartOS. The legacy of open-solaris is well and alive.
Has anyone tried ZFS together with Hadoop/HDFS? I find it to be a really interesting combination because of compression and self-healing of bad data-blocks.
Can anyone point me to a guide for migrating from software RAID to ZoL?
I suggest emailing the zfs-discuss mailing list for this one:
http://zfsonlinux.org/lists.html
The answer to your question depends on your existing setup.
http://zfsonlinux.org/lists.html
The answer to your question depends on your existing setup.
I think F2FS is.
I'm a bit confused. I always thought that proper use of ZFS required multiple drives just like RAID. Then I read an article where RAID was used with ZFS. Now I'm really confused so I'm taking the lazy way and asking here: is ZFS appropriate for a single drive system?
Sure, COW(copy on write), snapshots, error checksum on files, compression, support for SSD's. You just don't have redundancy.
I thought the error checking (scrubbing) only works if you have multiple drives set up in a RAID.
Error detection can always be used, but error correction may or may not be available (it depends on the type of block). Metadata blocks are redundant even on a single drive pool; so if you just have a partial failure (e.g. overwrite a metadata block) it might be able to correct the block using another redundant copy on the same drive. Data blocks will require a redundant pool configuration, though, as these are not store redundantly by default (e.g. multiple drives in a raidz or mirror).
Actually you can correct data blocks, on a single-drive ZFS pool, if you set the attribute copies=2 (ditto blocks): https://blogs.oracle.com/relling/entry/zfs_copies_and_data_p... Obviously this redundancy feature makes you use twice the disk space you would normally use.
yup, that's why I added "by default". a user won't get that functionality without explicitly asking for it.
You can also use it on a single drive. You need to set copies=n for a ZFS dataset. If one copy gets corrupted a zpool scrub fixes this problem on a single drive. However HDDs are usually dying fast after signalling bad blocks to the OS so it's not a good strategy to avoid data loss.
Besides that the metadata is still replicated in a single-drive setup. So if you loose data you'll at least know exactly where the error was. This is still a benefit over ext4.
Besides that the metadata is still replicated in a single-drive setup. So if you loose data you'll at least know exactly where the error was. This is still a benefit over ext4.
It only works in the sense that if it finds an error it could recover when it is a multi-drive RAID, but a scrub can verify that the data on disk is still whole and matches what it expects.
Not only that, but a scrub can have ZFS tell you exactly what files are now "damaged".
Not only that, but a scrub can have ZFS tell you exactly what files are now "damaged".
ZFS writes a checksum when it writes data. So you can tell whether your data is corrupted, even if you don't have the redundancy to restore the corrupted data.
Yes, I run ZFS on Linux on my single drive laptop currently.
ZFS doesn't _need_ multiple drives to work well, but it is capable of using them if they're available. One can also run a HW raid solution underneath ZFS, where the RAID engine just presents ZFS with a small number of LUNs. It's all up to the admin; and what sort of performance, redundancy, and maintenance guarantees are required.
ZFS doesn't _need_ multiple drives to work well, but it is capable of using them if they're available. One can also run a HW raid solution underneath ZFS, where the RAID engine just presents ZFS with a small number of LUNs. It's all up to the admin; and what sort of performance, redundancy, and maintenance guarantees are required.
No.
Care to elaborate?
The article is missing many things that make a FS suitable for production, it's listing every features but lack the reliability / cons / waknesses ect...
ZFS is nowhere near production ready for Linux. I know XFS and ext4 under heavy load with different scenarios, could you tell the same for ZoL?
Yes, I can. I am a system administrator who also knows a great deal about storage, and manages a good chuck of the storage servers at my employment. We use ZFS for our backup servers, both onsite, and offsite, and we use them for a couple generic storage servers as well, one of which is constantly under heavy stress, all the time. That server is http://mirrors.xmission.com.
I am personally using it on my workstation for a /home mount, and I have it on a highly available KVM 2-node cluster, replicated with GlusterFS using InfiniBand. While I have had networking issues with GlusterFS, which have since been ironed out, I have not had any stability, reliability, or data corruption issues with ZFS. At all. I've been running this cluster for 3 years straight, and it's the cluster that is housing my ZFS documentation at https://pthree.org/category/zfs
ZFS on Linux is absolutely "production ready". While it's true there are some ARC issues floating about, trim support is missing for SSDs, and some other things, remember that ZFS has been stable for a decade. It's just been brought into Linux kernel space with the help of the Solaris Porting Layer (spl) module.
Claiming that ZFS on Linux is not stable, and not production ready is nothing more than FUD.
I am personally using it on my workstation for a /home mount, and I have it on a highly available KVM 2-node cluster, replicated with GlusterFS using InfiniBand. While I have had networking issues with GlusterFS, which have since been ironed out, I have not had any stability, reliability, or data corruption issues with ZFS. At all. I've been running this cluster for 3 years straight, and it's the cluster that is housing my ZFS documentation at https://pthree.org/category/zfs
ZFS on Linux is absolutely "production ready". While it's true there are some ARC issues floating about, trim support is missing for SSDs, and some other things, remember that ZFS has been stable for a decade. It's just been brought into Linux kernel space with the help of the Solaris Porting Layer (spl) module.
Claiming that ZFS on Linux is not stable, and not production ready is nothing more than FUD.
is lack of trim support not a substantial issue? (for those using SSDs)
It can cause performance degradation when all the banks in the SSD have been written to, and a new bank needs to be re-written. TRIM allows cleaning the bank when it is no longer needed, and hopefully, before it needs to be written again. As such, the new rewrite will be native speed, and not bottle-necked
Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened).
This happens a lot more often then we really care to think about. (Ram corruption)
For most day-to-day linux users who are just using standard consumer desktop/laptops they don't have this protection. And ZFS can do more harm then good.
References:
https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
This happens a lot more often then we really care to think about. (Ram corruption)
For most day-to-day linux users who are just using standard consumer desktop/laptops they don't have this protection. And ZFS can do more harm then good.
References:
https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.
A person at the FreeNAS forums wrote a forum post explaining his mistaken believed that ZFS is somehow more prone to catastrophic data loss when bit flips occurred than other filesystems:
https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
The reality is that the worst case consequences for a bit flip is the complete loss of all data, regardless of the filesystem used. While the automated repair routined bundled in fsck utilities are often able to fix problems on other file systems, the problems that they do fix simply do not occur on ZFS. The kinds of problems that kill ZFS are not among those that an automated repair tool can fix. e.g. overwrite all ext4 superblocks with random data and then see if fsck.ext4 can fix it.
That said, I am usually able to resuscitate a pool that another person would have considered to have been killed by a bit flip. I not only find such failure modes to be incredibly rare, but I find those that I cannot fix to be a rarity among the cases where things did in fact go wrong.
https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
The reality is that the worst case consequences for a bit flip is the complete loss of all data, regardless of the filesystem used. While the automated repair routined bundled in fsck utilities are often able to fix problems on other file systems, the problems that they do fix simply do not occur on ZFS. The kinds of problems that kill ZFS are not among those that an automated repair tool can fix. e.g. overwrite all ext4 superblocks with random data and then see if fsck.ext4 can fix it.
That said, I am usually able to resuscitate a pool that another person would have considered to have been killed by a bit flip. I not only find such failure modes to be incredibly rare, but I find those that I cannot fix to be a rarity among the cases where things did in fact go wrong.
Indeed. I can reiterate this point. I have helped a few administrators, who initially were compiling ZFS on Linux from source, then decided to switch to the Launchpad PPA repository, and "upgrade" their userspace tools. Next thing I know, they are asking me if there is anything they can do to recover their pool. After a bit of research, and a little cleaning up, I am able to get their pool back online, with zero data loss.
I have had a VERY hard time finding a situation where there was corrupted data in ZFS, and where there are ZFS pools that absolutely will not import back into full operation.
In other words, it is pretty difficult to "accidentally" corrupt a ZFS pool to the point of not being able to recover your data.
ECC RAM greatly minimizes the potential for a corrupted file in ZFS, but ECC RAM also greatly minimizes the potential for a corrupted file in ext4 or XFS.
So long story short, I have experienced the same thing as ryao, and come to the same conclusions.
I have had a VERY hard time finding a situation where there was corrupted data in ZFS, and where there are ZFS pools that absolutely will not import back into full operation.
In other words, it is pretty difficult to "accidentally" corrupt a ZFS pool to the point of not being able to recover your data.
ECC RAM greatly minimizes the potential for a corrupted file in ZFS, but ECC RAM also greatly minimizes the potential for a corrupted file in ext4 or XFS.
So long story short, I have experienced the same thing as ryao, and come to the same conclusions.
Care to elaborate?
ZFS warned me of that while I reading its installation guide. Which is why I never installed it.
ZFS warned me of that while I reading its installation guide. Which is why I never installed it.
If you're doing RAID, you should have ECC. All hardware RAID controllers have ECC cache, and if you're doing software RAID, you should have ECC system RAM. But this is just common sense, I guess, since it's useless to write data to redundant storage if it's already been bit-flipped in RAM.
I thought that I had replied to this, but I seem to have replied to its parent by mistake. Here is a link to the response:
https://news.ycombinator.com/item?id=8438416
Would you provide a link to that guide? If there is a guide out there that says that you should use something other than ZFS when a system lacks ECC, I would like to know so that I can try to get it corrected.
Not using ZFS because it cannot provide full protection without ECC is like not getting a flu shot because you can still get sick anyway. It is true, but opting to be even less safe in the name of safety is counterproductive.
https://news.ycombinator.com/item?id=8438416
Would you provide a link to that guide? If there is a guide out there that says that you should use something other than ZFS when a system lacks ECC, I would like to know so that I can try to get it corrected.
Not using ZFS because it cannot provide full protection without ECC is like not getting a flu shot because you can still get sick anyway. It is true, but opting to be even less safe in the name of safety is counterproductive.
The trouble is, once you have a corrupted ZFS, there's no good way to recover it. That's why it's vital to make sure nothing bad happens in the first place.
The same can be said for other filesystems. The issues that fsck has been abused to automatically fix on them simply do not happen on ZFS. Failures so severe that they kill ZFS have should have analogous failure states on those filesystems too. People do not hear about such failures because they cannot be distinguished from the more typical issues that affect such filesystems.
This is a myth started by someone who did not understand how filesystems work.
You are wrong. Don't spread potentially dangerous maladvice on the internet when you have no idea what you're talking about.
The official ZFS documentation[1] tells you to use ECC Ram and why. The first google hit for "zfs ecc ram"[2] further elaborates on the risks of using ZFS without ECC memory.
[1] https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
[2] http://louwrentius.com/please-use-zfs-with-ecc-memory.html
You are wrong. Don't spread potentially dangerous maladvice on the internet when you have no idea what you're talking about.
The official ZFS documentation[1] tells you to use ECC Ram and why. The first google hit for "zfs ecc ram"[2] further elaborates on the risks of using ZFS without ECC memory.
[1] https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
[2] http://louwrentius.com/please-use-zfs-with-ecc-memory.html
RAID is not a backup, zfs especially so. Zfs (and RAID) is designed to protect your data from a small, enumerable and specific set of failures, namely the right-there-in-the-acronymn "inexpensive disks".
If your house burns down ZFS will not save your data. If too many disks fail zfs will not save your data. If your disks' unrecoverable read error rate is too high and your array is rebuilding zfs will not save your data.
If your computer is fundamentally broken (which is what a machine with a persistent memory error is) zfs will not save your data.
Take backups.
If you had a 'stuck bit' anywhere in your memory space you'd be way deep into nasal demon unspecified behavior the first time you tried to dereference a pointer that crossed that bit. When your hardware is that broken you can't count on the OS to not stab your dog much less ZFS to do anything sane.
Note that this is just as true for all storage systems. Regular filesystems might corrupt themselves or do other insane things. Hardware RAID or kernel software raid will happily propogate the error. How many bits separate the kernel record for "this disk is totally cool" from "this is a new disk and should be zeroed"?
If your house burns down ZFS will not save your data. If too many disks fail zfs will not save your data. If your disks' unrecoverable read error rate is too high and your array is rebuilding zfs will not save your data.
If your computer is fundamentally broken (which is what a machine with a persistent memory error is) zfs will not save your data.
Take backups.
If you had a 'stuck bit' anywhere in your memory space you'd be way deep into nasal demon unspecified behavior the first time you tried to dereference a pointer that crossed that bit. When your hardware is that broken you can't count on the OS to not stab your dog much less ZFS to do anything sane.
Note that this is just as true for all storage systems. Regular filesystems might corrupt themselves or do other insane things. Hardware RAID or kernel software raid will happily propogate the error. How many bits separate the kernel record for "this disk is totally cool" from "this is a new disk and should be zeroed"?
While it is possible to have an unimportable pool, it is also possible to have ext4 and XFS filesystems that can neither be mounted nor repaired with fsck. When dealing with undefined behavior caused by bit flips, virtually any failure is possible, especially when you consider bit flips to kernel data structures. No software can save you from bit flips and if they are your concern (as they should be), then you should refuse to use computers that lack ECC RAM.
As for the articles that you link, they are correct to say that you want to use ECC RAM. However, there is nothing specific to ZFS that makes it require ECC any more than any other filesystem. It should also be noted that I wrote the official documentation on this subject. It can be found at the Open ZFS wiki, rather than the pages you linked:
http://open-zfs.org/wiki/Hardware#ECC_Memory
As for the articles that you link, they are correct to say that you want to use ECC RAM. However, there is nothing specific to ZFS that makes it require ECC any more than any other filesystem. It should also be noted that I wrote the official documentation on this subject. It can be found at the Open ZFS wiki, rather than the pages you linked:
http://open-zfs.org/wiki/Hardware#ECC_Memory
I apologize. I was sure to be dealing with a troll and was sloppy with my sources (i.e. I was sure to have once read it in the official docs, but here I just linked the first official looking google hit that was not the solaris docs).
there is nothing specific to ZFS that makes it require ECC any more than any other filesystem
This statement still troubles me. I was under the assumption that ZFS scrubbing may indeed lead to compounding corruption as described in the blog post I linked. Hence the ongoing(?) debate in the ZFS community.
However, I consider myself disqualified from this discussion and sorry again for my tone. I'll read up on the current state of the argument.
there is nothing specific to ZFS that makes it require ECC any more than any other filesystem
This statement still troubles me. I was under the assumption that ZFS scrubbing may indeed lead to compounding corruption as described in the blog post I linked. Hence the ongoing(?) debate in the ZFS community.
However, I consider myself disqualified from this discussion and sorry again for my tone. I'll read up on the current state of the argument.
It is easy to get confused and it could happen to anyone.
That said, the only thing in the code remotely similar to "compound corruption" is logic for preventing such a scenario, which has the following comment:
That touches more than obvious at a glance because the mirror code is used for processing both mirrors and ditto blocks.
That said, the only thing in the code remotely similar to "compound corruption" is logic for preventing such a scenario, which has the following comment:
/*
* Don't rewrite known good children.
* Not only is it unnecessary, it could
* actually be harmful: if the system lost
* power while rewriting the only good copy,
* there would be no good copies left!
*/
That touches more than obvious at a glance because the mirror code is used for processing both mirrors and ditto blocks.
Sorry for the very late reply that you probably won't see.
I'm not familiar with ZFS at all, so I don't know what the terminology 'ditto blocks' and 'mirrors' mean, so there's a good chance I'm not talking sense here at all.
The main argument for ZFS on non-ECC using systems being worse than other less sophisticated file systems seems to be that just reading a file (or during periodic 'scrubbing') while you have bad ram can cause good files to be corrupted when ZFS notices checksum errors and tries to correct them. (There are other arguments about not being able to mount pools due to bad ram, but that seems no different to any other FS).
People seem to be quite insistent that this is possible, and at first glance the code you linked above doesn't seem to be relevant since it refers to 'known good' data, but the problem scenario is when data has (erroneously) been determined to be bad due to faulty ram. Will ZFS overwrite good data in that situation in an attempt to correct what it sees as a disk error?
A few links where people are making this claim:
https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
I'm not familiar with ZFS at all, so I don't know what the terminology 'ditto blocks' and 'mirrors' mean, so there's a good chance I'm not talking sense here at all.
The main argument for ZFS on non-ECC using systems being worse than other less sophisticated file systems seems to be that just reading a file (or during periodic 'scrubbing') while you have bad ram can cause good files to be corrupted when ZFS notices checksum errors and tries to correct them. (There are other arguments about not being able to mount pools due to bad ram, but that seems no different to any other FS).
People seem to be quite insistent that this is possible, and at first glance the code you linked above doesn't seem to be relevant since it refers to 'known good' data, but the problem scenario is when data has (erroneously) been determined to be bad due to faulty ram. Will ZFS overwrite good data in that situation in an attempt to correct what it sees as a disk error?
A few links where people are making this claim:
https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...
The pthree.org article you linked to is my blog. It is not the official ZFS documentation nor the official ZFS on Linux documentation. It is only a storage administrator (me) explaining what I know of the inner workings of ZFS.
However, thanks for linking to my blog! :)
However, thanks for linking to my blog! :)
Are you referencing this thread? https://groups.google.com/forum/#!topic/zfs-macos/qguq6LCf1Q...
I've heard this but I haven't actually confirmed anywhere official that ZFS will try to scrub your data due to a parity flip in ram values. Where in the manuals do they talk about it? (I'm actually curious about if it's just a rumor or if it's acknowledged)
I've heard this but I haven't actually confirmed anywhere official that ZFS will try to scrub your data due to a parity flip in ram values. Where in the manuals do they talk about it? (I'm actually curious about if it's just a rumor or if it's acknowledged)
Referencing this
https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
http://louwrentius.com/please-use-zfs-with-ecc-memory.html
https://pthree.org/2013/12/10/zfs-administration-appendix-c-...
http://louwrentius.com/please-use-zfs-with-ecc-memory.html
Even after reading that, I'm still not convinced it's any worse than something like btrfs+non-ECC memory. The most convincing part is the 'there are less tools' to recover from corruption angle.
This should be fairly obvious...
ZFS has it's own requirements... to work well it needs lots of resources (cpu/ram) and EEC memory.
If you have those, then sure, it gives you many advantages.
If you don't, then you are opening yourself up for struggle.
ZFS has it's own requirements... to work well it needs lots of resources (cpu/ram) and EEC memory.
If you have those, then sure, it gives you many advantages.
If you don't, then you are opening yourself up for struggle.
ZFS doesn't need any more "resources" than any other kernel space filesystem. Deduplication on the other hand, does need a good amount of RAM, if you want to prevent the deduplication table (DDT) from spilling to slow spinning rust. However, if you have a fast, and large SSD L2ARC, the DDT spilling over to the L2ARC might not have drastic consequences.
Regardless, unless using the DDT feature of ZFS, it doesn't need anymore resources than ext4 or XFS.
Regardless, unless using the DDT feature of ZFS, it doesn't need anymore resources than ext4 or XFS.
I kind of screwed myself over though because I to do a bunch of file cleanup that I was hoping to perhaps roll back if it didn't go well so I created a snapshot and then got to work with some copying…
I wasn't paying attention and I ended up running out of space on the disk. The way ZFS snapshots work is that ZFS still does what it would normally do on a copy: writes data, then verifies the write is completed before deleting the old data. When there is a snapshot, it just skips the second step and leaves the old data around. This is how snapshots have zero performance impact on data write.
Turns out that I couldn't even delete the old snapshot (or rollback the new one) because I didn't have enough space to make any changes to the disk. I couldn't even delete files because they would stay in the snapshot.
Aha! I can add a disk to the pool! So I added a small disk to the zpool and then I could delete the snapshot ok. However, I forgot that you can't remove a disk from a zpool. I ended up having to move the data to another disk, drop the entire zpool and recreate it, and move it back.
Honestly, everything worked rather well except for forgetting a few of the limitations of ZFS. Given how often I have issues with HFS+ I really think it's pretty tempting for OS X.