Harddrive Is Probably Full(marginalia.nu)
marginalia.nu
Harddrive Is Probably Full
https://www.marginalia.nu/log/a_139_hdd/
16 コメント
If you're on Windows, WizTree (https://diskanalyzer.com/) is great for figuring out what's consuming your storage space.
On Linux/macOS you can use `du` from the command line (or ask an AI to make a convenience wrapper for you).
On Linux/macOS you can use `du` from the command line (or ask an AI to make a convenience wrapper for you).
On Linux, ncdu is really good in the terminal. Filelight on KDE and Baobab on Gnome are really nice GUIs.
Yes, I always use `du . -h 2>/dev/null | sort -k1hr | head -nN` where that last N is however many directories I want to show. I run this regularly and have for a very long time precisely because of the issue that the post talks about.
I've been using SpaceMonger (220 KB) since... Win95 I think.
https://www.portablefreeware.com/index.php?id=150
https://www.portablefreeware.com/index.php?id=150
I've yet to encounter any issues with gdu for linux. Highly reccomended
> 46x faster than windirstat
Thats a pretty bold claim, but is it really that much faster?
Thats a pretty bold claim, but is it really that much faster?
I just ran it for the first time... I've run windirstat many times including in the last two weeks and yes, this is very fast. According to its timing it took 3.63 seconds to scan my drive... I'm not sure if it really took almost 3 minutes for windirstat, but it did take a long time, so it's plausible. More importantly, ~ 4 seconds is much easier to wait for.
windirstat currently doesn't take 3min to index 500k+ files on an 8x8tb drive raidz2 that's 90% full. When I stopped using windoratat 3ish years ago it took a few minutes to index the same array.
I think you might he using an old version, or have an issue with your storage.
I think you might he using an old version, or have an issue with your storage.
It is - it ‘cheats’ by reading off the NTFS MFT, similarly to another indispensable piece of Windows software Everything Search, both of which I miss keenly on my underprovisioned work Mac.
I like erdtree or dust instead of du.
DaisyDisk is also a great option on Mac.
I recently ran `df` and was surprised to see that I'd used more than half of the two terabyte SSD in my laptop. So, being apparently atypically proactive, my first action was delete all of the target directories in my Rust projects and was equally surprised to see that it freed up nearly 10% of my disk usage (~200 GB)!
Are there many tiny files? They may take 4KiB of space each even though they're less than a KiB big (https://en.wikipedia.org/wiki/Disk_sector).
On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.
On a SanDisk portable SSD I own, the cluster size is even 4MiB. If you put a node modules directory in there with thousands of tiny files, it can "take up" several GiB of space.
This can be an important reason to enable compression (on some file systems it “solves” the problem) and why virtual machine file format can be important.
I’ve also noticed this with rust. Those target directories can get huge.
I think about this problem often in life because of my tendency to procrastinate. Only the long term effects that cause me serious paranoia get any real attention (failing to meet social expectations, major health issues, large-figure finance problems). It’s absolutely necessary to define the constraints before they define you. This requires a lot of forward thinking and effort but considering how little time we get on this earth… worth it
Mine is mostly full and it's not stuff I've saved but stuff software has stored there without asking me. It's quite hard to figure what's going on. I once got a message from Chrome saying it couldn't go on because it had stored like 200gb for websites, would I like to delete it all, and I said yes and noticed no difference.
It's annoying because I'd rather store say family photos than have it full of random junk that means nothing to me. Dunno the answer - better os control? Partition the drive?
It's annoying because I'd rather store say family photos than have it full of random junk that means nothing to me. Dunno the answer - better os control? Partition the drive?
It's not just hard drive problem. If you buy a new storage rack at your home, it will be full in no time. No matter how many racks you buy, you will always find things fill those racks. This is like rain water filling up the lakes. if there is a lake, it gets filled. Or maybe it is called a lake because it gets filled. So, one can say that the disks create or cause the data. If there is no disk, there is no data.
This is a variation of Parkinson's Law https://en.wikipedia.org/wiki/Parkinson%27s_Law: "Data expands to fill the space available for storage."
If you're on Windows, MacOS or Linux, a great tool for interactively discovering and deleting large folders and files is dua:
https://github.com/Byron/dua-cli
If you're on MacOS and suspect a lot of space is taken by multiple copies of the same files, use fclones to turn them into APFS clones. (You can also use this on Linux, as ext4 supports hard links, but only if you're sure the duplicates won't diverge.)
https://github.com/Byron/dua-cli
If you're on MacOS and suspect a lot of space is taken by multiple copies of the same files, use fclones to turn them into APFS clones. (You can also use this on Linux, as ext4 supports hard links, but only if you're sure the duplicates won't diverge.)
I have less than one terabyte of disk space, and less than 10% of the full capacity is in use, so it is closer to being empty. (The hard drives they sell have more capacity than I require.) And, there are files that I do not use (some of which come with the operating system, such as PulseAudio).
My harddrive at 1tb has around the same amount of free space in % as my previous computer with just 256gb.
Asked Claude to look at my full disk. Thought for 1 minute
It found 114GB of Hugging Face models, 17GB worth of models in an obscure app I tested once. In places I would've never thought to look.
I'd never have never found these myself.
Problem solved. For now!
It found 114GB of Hugging Face models, 17GB worth of models in an obscure app I tested once. In places I would've never thought to look.
I'd never have never found these myself.
Problem solved. For now!
Just looking with ncdu/du or another tool which sorts folders by size would have been much faster and easier, was there any need to use Claude here?
It is so wild that someone would let Claude index all of the files on their computers for such a bane request. We are really turning into the people on WALL·E
When your only tool is a hammer everything looks like a nail.
> 20+ years of software engineering experience across full-stack, backend, infrastructure, Site Reliability Enigneering, and AI engineering. Past work includes Uber, Airbnb, Canva, and a wide variety of startups - several of which I co-founded.
> I'd never have never found these myself.
You are either the most experienced most incompetent software engineer in the world, or you are lying.
> I'd never have never found these myself.
You are either the most experienced most incompetent software engineer in the world, or you are lying.
The problem with filling up a ssd is that it writes and rewrites the same blocks over and over defeating the built in wear leveling.
Filling an ssd creates the one situation where nand wear out is a problem.
Filling an ssd creates the one situation where nand wear out is a problem.
The SSD firmware is smart enough to rotate the hot sectors over the entire space gradually.
It is a problem to have a full SSD from a performance perspective. Modern SSDs use pSLC (psuedo-single-level-cells) because they can write much faster. When you write to an SSD, you generally write in pSLC mode first, and once the data goes cold, it will get written back to MLC/TLC storage.
Once you are very full, pSLC gets disabled or becomes less effective, affecting write speeds.
There is also an effect known as write amplification that is exacerbated when a disk is nearly full. You want to write 4kb, but the available erasure zones are all full (either valid or invalidated sectors), so now the SSD has to erase a 4MB zone and compact the data there (along with your new 4kb) to write your 4kb.
This gets even worse when you have a filesystem that isn't SSD optimized, such as EXT4. It will cause further write amplication.
Keep your SSD's <80% full if you want good performance.
It is a problem to have a full SSD from a performance perspective. Modern SSDs use pSLC (psuedo-single-level-cells) because they can write much faster. When you write to an SSD, you generally write in pSLC mode first, and once the data goes cold, it will get written back to MLC/TLC storage.
Once you are very full, pSLC gets disabled or becomes less effective, affecting write speeds.
There is also an effect known as write amplification that is exacerbated when a disk is nearly full. You want to write 4kb, but the available erasure zones are all full (either valid or invalidated sectors), so now the SSD has to erase a 4MB zone and compact the data there (along with your new 4kb) to write your 4kb.
This gets even worse when you have a filesystem that isn't SSD optimized, such as EXT4. It will cause further write amplication.
Keep your SSD's <80% full if you want good performance.
Won't modern wear leveling migrate rarely written data to spread new writes out to its blocks?
And i am wondering what do people even fill their storage with, and why do they need so much. In my case it is ~200-300 out of 1TB on 2 laptops, 37 out of 128gb on phone. And i don't even actively manage it.
I have one single video game I play that takes up around 650GB, so I have a dedicated drive for it. And between 1 and 2 TB of movies, TV, and photos. Everything else is ephemeral/scratch space for whatever I'm working on.
I used to be into amateur movie making and editing, and back then I needed many TB of space for the stuff I pulled off my cameras before it got edited down.
I used to be into amateur movie making and editing, and back then I needed many TB of space for the stuff I pulled off my cameras before it got edited down.
What video game!? I've seen games that reach up to the already crazy number of 100gb, but I usually just don't install those, with few exceptions
One of the popular flight simulators. Earth terrain data takes up a huge amount of space.
My photo library is around 700 gb.
Photos, movies, MP3, SDKs for various kinds of systems, PDFs (and PS) of computing papers collected during the last 35 years, and naturally games.
I have 2TB filled and I need more. I make music so I need a lot of storage for my instruments and samples. This is measured in tens and often hundreds of gigabytes. I also play games. Games' sizes range wildly from 10 to 200 GBs. I also do gamedev. The textures get big and get duplicated between projects. Programming (npm modules for example), add to this personal data, media archives and 2TB is not enough
probably. wish there was a way to find out. hint: not logs or updates.
I bought a 8tb drive just before prices exploded so I'm alright
... and It is ok.
I bought 16TB disk because I needed that much storage.
If I were using, say, only 10TB, I would be wasting 6TB.
I can expand my LVM as much as I can. But I definitely want my disk to be full. Same goes for my RAM.
And I want my CPU ide time to stay as close as possible to 0% and my load average as close as possible to my no. of CPUs.
Anything else would mean oversizing and overspending.
I bought 16TB disk because I needed that much storage.
If I were using, say, only 10TB, I would be wasting 6TB.
I can expand my LVM as much as I can. But I definitely want my disk to be full. Same goes for my RAM.
And I want my CPU ide time to stay as close as possible to 0% and my load average as close as possible to my no. of CPUs.
Anything else would mean oversizing and overspending.
Slack has value, though. Your full disk is likely to be very fragmented, making file access slow. A CPU running a full throttle cannot deal with spikes in demand. full RAM means less cache space available. And these issues get asymptotically worse as you approach 100% utilization (which can have catastrophic effects in organizations which are too heavily optimized for high utilization).
Slack has value, though. Your full disk is likely to be very fragmented, making file access slow. A CPU running a full throttle cannot deal with spikes in demand and latency suffers, as well as power consumption. full RAM means less cache space available. And these issues get asymptotically worse as you approach 100% utilization.
Delete is expensive. "Am I positive I'll never need this again?"
As a consequence, "cheap" actions (saves) accumulate, "expensive" ones (deletes) accrue until cheap becomes expensive as space becomes tight.
Ultimately, options become:
- Find criteria for rapidly assessing what data can be deleted.
- Purchase cheap archival storage and migrate old data to that.
- A media or device crash induces loss of arbitrarily-selected data.