We successfully hired from HN in the previous round and are looking for another OpenZFS Developer (3+ years of experience) to join our team!
Klara Inc. provides development & solutions focused on open source software and the community-driven development of OpenZFS and FreeBSD. We develop new features, investigate/fix bugs, and support the community of these important open source infrastructure projects. Some of our recent work includes major ZFS features such as Fast Deduplication (OpenZFS 2.3: https://github.com/openzfs/zfs/discussions/15896) and AnyRAID: https://github.com/openzfs/zfs/pull/17567.
We’re looking for an OpenZFS Developer with:
- Strong C programming skills and solid understanding of data structures
- Experience with file systems, VFS, and OS internals (threading, locking, IPC, memory management)
Klara provides open source development services with a focus on ZFS, FreeBSD, and Arm. Our mission is to advance technology through community-driven development while maintaining the ethics and creativity of open source. We help customers standardize and accelerate platforms built on ZFS by combining internal expertise with active participation in the community.
We are excited to share that we are looking to expand our OpenZFS team with an additional full-time Developer.
Our ZFS developer team works directly on OpenZFS for customers and with upstream to add features, investigating performance issues, and resolve complex bugs. Recently our team has upstreamed Fast Dedup, critical fixes for ZFS native encryption, improvements to gang block allocation, and has even more out for review (the new AnyRAID feature).
The ideal candidate will have experience working with ZFS or other Open Source projects in the kernel.
The "M" at the end of that revision number suggests a "modified" tree, meaning they had their own patches that were not part of the upstream repository as well.
ZFS has supported online adding of vdevs since the start too, this is specifically modifying an existing vdev and widening it, which is much less common, and much more complex
There are advantages to doing the cloning at the block level, rather than the VFS layer. The feature was originally written for FreeBSD using the copy_file_range() syscall, then extended to work with the existing interfaces in Linux from btrfs.
There are a few different use cases, but cloning a VM image file is definitely a popular one.
Also, `mv` between different filesystems in the same ZFS pool. Traditionally when crossing filesystems doesn't allow just using `rename()`, `mv` resorted to effectively `cp` then `rm`, so at least temporarily required 2x the space, and that space might not be freed for a long time if you have snapshots.
With BRT, the copy to the 2nd filesystem doesn't need to write anything more than a bit of metadata, and then when you remove the source copy, it actually removes the BRT entry, so there is no long-term overhead.
One of the original developer's use cases was restoring a file from a snapshot, without having to copy it and have it take up additional space.
So you make a file (foo) 2 days ago.
You change it each day.
Today, the change you made was bad, and you want to restore the version from yesterday.
before BRT:
you copied the file from the snapshot back to the live filesystem, and it took up all new space.
after BRT:
we reference the existing blocks in the snapshot, so the copy to the live filesystem takes no additional space on disk. A small BRT entry is maintained in memory (and on disk).
If you remove the snapshot, the BRT entry is removed, and the file remains intact. No long term overhead.
The reason the parity ratio stays the same, is that all of the references to the data are by DVA (Data Virtual Address, effectively the LBA within the RAID-Z vdev).
So the data will occupy the same amount of space and parity as it did before.
All stripes in RAID-Z are dynamic, so if your stripe is 5 wide and your array is 6 wide, the 2nd stripe will start on the last disk and wrap around.
So if your 5x10 TB disks are 90% full, after the expansion they will contain the same 5.4 TB of data and 3.6 TB of parity, and the pool will now be 10 TB bigger.
New writes, will be 4+2 instead, but the old data won't change (they is how this feature is able to work without needing block-pointer rewrite).
A post where we discuss some strategies and tools to make managing disk arrays on FreeBSD (and related platforms like TrueNAS Core) much easier. These concepts also apply to other operating systems, but the tools might differ slightly.
In previous talks Netflix has mentioned that due to serving so many 1000s of people from each box, that they basically do 0 caching in memory, all of the system memory is needed for buffers that are enroute to users, and they purposely avoid keeping any buffer cache beyond what is needed for sendfile()
I think the implication is: TCP kind of assumed you will either keep transmitting, or close the connection.
The many video-streaming type workloads, the connection will go idle, for seconds or even minutes at a time. If the loss is at the tail end of some activity, before a period of idle, the recovery takes a lot longer than it would if there are further activity on the connection.
Sure, the ZFS features are the same, but the integration is still not all there on Linux, and is on FreeBSD.
FreeBSD has features like 'bootonce', to switch ZFS boot environments on the next boot only. Better integration with the boot loader, more boot environment tooling, and other things that can only come when the filesystem is part of the OS.
Without a checksum, hardware RAID has no way to KNOW it needs to use the parity to correct the block.