It’s fun to build things like this but if you want to nourish a user base you need to fully understand the landscape of similar tools and then explain your differentiating value. This is /particularly/ important for security related tools.
Specifically you should compare and contrast to tools like SOPS, Ansible Vault, pass, etc.
This is an unnecessary optimization, particularly for the article's use case (small files that are read immediately after being written). Just use /tmp. The linux buffer cache is more than performant enough for casual usage and, indeed, most heavy usage too. It's far too easy to clog up memory with forgotten files by defaulting to /dev/shm, for instance, and you potentially also take memory away from the rest of the system until the next reboot.
For the author's purposes, any benefit is just placebo.
There absolutely are times where /dev/shm is what you want, but it requires understanding nuances and tradeoffs (e.g. you are already thinking a lot about the memory management going on, including potentially swap).
Woohoo, one of the highlights of this time of year. I had to do mine from an eastbound flight over the pacific. This has become a fun tradition not just for me personally but for many friends, colleagues, and fellow HNers. Big props once again to wastl and his helper elves for making this!
I encourage anyone who gets value from this to donate to support it if they can. It is a passion project but nonetheless comes with real costs.
Architecture matters because while deep learning can conceivably fit a curve with a single, huge layer (in theory... Universal approximation theorem), the amount of compute and data needed to get there is prohibitive. Having a good architecture means the theoretical possibility of deep learning finding the right N dimensional curve becomes a practical reality.
Another thing about the architecture is we inherently bias it with the way we structure the data. For instance, take a dataset of (car) traffic patterns. If you only track the date as a feature, you miss that some events follow not just the day-of-year pattern but also holiday patterns. You could learn this with deep learning with enough data, but if we bake it into the dataset, you can build a model on it _much_ simpler and faster.
So, architecture matters. Data/feature representation matters.
But not all things you might do with a dotfile (or, more generally, per-user customization) are just replacing files. Things like cronjobs, brew installs, `defaults` in MacOS, etc. Viewing dotfile-based customization as strictly files to obliterate with pre-existing files is needlessly myopic.
For this broader problem, there are other more complete solutions that are more robust and flexible. Personally I like dotbot (https://github.com/anishathalye/dotbot) as a balance between power and simplicity, particularly when managing files across multiple OS homedirs (e.g. linux server, macos laptop).
Reread the story. The child wasn’t left in the car for an extended period (by a grandparent, not parent). The child had just been buckled into a car seat and the driver closed the door, walked around to the drivers side, and couldn’t get in.
Absolutely no indication of improper adult behavior.
We give away potatoes to trick or treaters on Halloween. They are immensely popular and we’ve become known as the potato house in our city’s Facebook groups. The weird delight on the faces of kids of all ages was hugely unexpected but surprisingly consistent.
Careful, your homedir has a CloudStorage folder and if you are using, say, Dropbox or Google Drive then that find will be incredibly slow (in addition to security software possibly slowing it down).
I find it very useful. I made a tool similar to mcfly (before knowing it existed) and use this workflow (`--here`) constantly. Also hostname context and shell session can be useful at times, too, to reconstruct something in the past.
While I doubt I'd quit my day job for it, over the past couple of years I've been poking at my own database-backed shell history. The key requirements for me were that it be extremely fast and that it support syncing across multiple systems.
The former is easy(ish); the latter is trickier since I didn't want to provide a hosted service but there aren't easily usable APIs like s3 that are "bring your own wallet" that could be used. So I punted and made it directory based and compatible with Dropbox and similar shared storage.
Being able to quickly search history, including tricks like 'show me the last 50 commands I ran in this directory that contained `git`' has been quite useful for my own workflows, and performance is quite fine on my ~400k history across multiple machines starting around 2011. (pxhist is able to import your history file so you can maintain that continuity)
What a great historical summary. Compression has moved on now but having grown up marveling at PKZip and maximizing usable space on very early computers, as well as compression in modems (v42bis ftw!), this field has always seemed magical.
These days it generally is better to prefer Zstandard to zlib/gzip for many reasons. And if you need seekable format, consider squashfs as a reasonable choice. These stand on the shoulders of the giants of zlib and zip but do indeed stand much higher in the modern world.
Mark is one of the world's top experts on practical MySQL performance at scale, having spent a huge amount of time optimizing MySQL at Google and Facebook. There's a question in this thread about whether this has real world impact... yes, if Mark noticed it, yes, yes it does. This will materially improve many common workloads for InnoDB.
This article seems like it is an AI summary of another document. It doesn’t offer anything new, just restating someone else’s analysis. And that analysis comes down to… storage demand will outpace SSDs price point enough that the demand will be met by both SSDs and traditional spinning platters.
Which isn’t surprising as even today there is demand for tape. The question is how long the demand will remain vaguely mainstream vs when it will become more niche. The 2028 estimate of storage being dominated by SSDs seems vaguely reasonable to me.
Pure’s bravado of staying zero hard drives will be sold after 2028 seems silly but inline with what a flash storage company would say. But from a directional standpoint it probably is right that many use cases will get further eroded by SSDs. One big challenge with hard drives is access speed (throughout and latency) compared to nvme, and hard drives being used more as cold and near line storage is definitely going to continue. Write once access never in many cases.
In addition to the great advice in this thread, it’s worth being aware of something called “Rejection Sensitivity Dysphoria” (RDD). It isn’t a diagnosable condition according to the DSM-V but is often observed coincident with ADHD. Basically many forms of criticism or even observation can trigger a deep and intense fight-or-flight response due to challenges regulating emotions.
Therapy and medication can help. If you think this might be in play it is worth talking to a mental health professional.
Unless the recipient of whatever you are compressing absolutely requires gzip, you should not use gzip or pigz.
Instead you should use zstd as it compresses faster, decompresses faster, and yields smaller files. It also supports parallelism (via “-T”) which supplants the pigz use case. There literally are no trade-offs; it is better in every objective way.
I consider myself very fortunate to have had the opportunity to work with Yann. He is amazingly humble for someone who has made something so impactful. One thing also that sometimes gets lost is wins as substantial as Zstandard not only save the big tech companies money but also reduce energy costs overall as well as local costs such as bandwidth on mobile devices (which affects individuals as much as it does companies).
To be fair, for every one of these sub-30 wunderkinds there are dozens of 50+ year olds hoping to make a fortune by propping them up… who often avoid such public derision.
Specifically you should compare and contrast to tools like SOPS, Ansible Vault, pass, etc.