The thinking behind the 32GB Windows Format limit on FAT32(theregister.com)
theregister.com
The thinking behind the 32GB Windows Format limit on FAT32
https://www.theregister.com/2021/01/04/windows_format_fat32/
15 comments
He made a decision and assumed when making future NT versions this would be revised and updated with the times. I can only image that future teams looked at this and thought "someone chose 32GB for a reason, better not change it"...
Well, Windows 95/98/ME, 2000 and XP shared the same old font installer that Windows 3.11 . I wonder why no body changed it before.
Likely if it ain't broke don't fix it.
Maybe, but it could also be about just not getting around to doing it.
Changes, especially in a large system with a large user base, require extensive testing, and even small changes may come with a lot of red tape. I don't know how things work at Microsoft, but I wouldn't be surprised if making changes to the UI for a technology that's being superseded anyway just didn't make anybody's list.
Changes, especially in a large system with a large user base, require extensive testing, and even small changes may come with a lot of red tape. I don't know how things work at Microsoft, but I wouldn't be surprised if making changes to the UI for a technology that's being superseded anyway just didn't make anybody's list.
In other words, he failed to comment his code adequately.
TIL: In 2019, Microsoft released the exFAT [1] specs to a large group of vendors [2], including prominent Linux vendors. I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems.
[1] https://en.wikipedia.org/wiki/ExFAT
[2] https://en.wikipedia.org/wiki/Open_Invention_Network
[1] https://en.wikipedia.org/wiki/ExFAT
[2] https://en.wikipedia.org/wiki/Open_Invention_Network
The patents on FAT32 are now fully expired, including the final lingering LongFileName one.
Therefore FAT32 will remain preferred until the patents expire on exFAT, even though the specification for exFAT has been released, things like that have proven to be not good enough until after the patents expire.
The purpose of exFAT appears mainly to be an attempt to foist a new encumbered filesystem on developers now that the first mainstream Microsoft filesystem is in the public domain.
We will know when Microsoft has reduced their anti-user stance on filesystems after they have released very detailed full NTFS specifications in the most useful way possible.
Also I have the greatest respect for Dave, and like he mentioned he was only doing the UI not the bitmoving of the actual Format process.
Those who were developing the NT version of FAT32 formatting failed in much more serious ways where Dave did not.
The inconsistency compared to FAT32 formatted by W9x is actually intolerable.
Best reliability still requires booting to the DOS from W98SE to properly format FAT32.
Therefore FAT32 will remain preferred until the patents expire on exFAT, even though the specification for exFAT has been released, things like that have proven to be not good enough until after the patents expire.
The purpose of exFAT appears mainly to be an attempt to foist a new encumbered filesystem on developers now that the first mainstream Microsoft filesystem is in the public domain.
We will know when Microsoft has reduced their anti-user stance on filesystems after they have released very detailed full NTFS specifications in the most useful way possible.
Also I have the greatest respect for Dave, and like he mentioned he was only doing the UI not the bitmoving of the actual Format process.
Those who were developing the NT version of FAT32 formatting failed in much more serious ways where Dave did not.
The inconsistency compared to FAT32 formatted by W9x is actually intolerable.
Best reliability still requires booting to the DOS from W98SE to properly format FAT32.
The main use for FAT32 and exFAT is to transfer files between different operating systems or between a computer and some dedicated device, e.g. a photographic camera.
They are not useful for anything else except memory cards, USB sticks and external SSDs.
Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating the parts at destination. That wastes time and it is annoying and sometimes it may be even impossible, when the destination lacks a concatenation program or an archiver that can deal with split archives.
After Microsoft released it, now exFAT is supported in the Linux kernel. Because exFAT is also supported by all commercial operating systems, it is now the best way for file transfer.
It does not matter any more if there are still valid patents for exFAT.
They are not useful for anything else except memory cards, USB sticks and external SSDs.
Unfortunately, nowadays it is frequently necessary to copy files larger than 4 GB. When using FAT32, you cannot do that without splitting the file and concatenating the parts at destination. That wastes time and it is annoying and sometimes it may be even impossible, when the destination lacks a concatenation program or an archiver that can deal with split archives.
After Microsoft released it, now exFAT is supported in the Linux kernel. Because exFAT is also supported by all commercial operating systems, it is now the best way for file transfer.
It does not matter any more if there are still valid patents for exFAT.
> It does not matter any more if there are still valid patents for exFAT.
Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?
Would you care to elaborate for why that is? Would anyone else with knowledge about the patent system and any possible risks imposed by it also like to do so?
Because, effectively, the exFAT filesystem is OSS since August 2019 and Microsoft released all the specifications to it [0]. Microsoft has essentially taken public, legal steps to ensure that anyone else can use exFAT without legal encumbrances and without threat of lawsuit.
IANAL - so there's likely some caveats (you probably can't just use the exFAT name unless certain qualifications are met, just like you can't use Firefox's trademarks willy-nilly), but that should be good enough for just about anyone. Including paranoid OSS peeps like me :)
0: https://cloudblogs.microsoft.com/opensource/2019/08/28/exfat...
IANAL - so there's likely some caveats (you probably can't just use the exFAT name unless certain qualifications are met, just like you can't use Firefox's trademarks willy-nilly), but that should be good enough for just about anyone. Including paranoid OSS peeps like me :)
0: https://cloudblogs.microsoft.com/opensource/2019/08/28/exfat...
> Because, effectively, the exFAT filesystem is OSS since August 2019
No its inclusion in Linux was the result of a decision by Microsoft, not a granting of an open license. Any other implementation still has to apply for a license:
"For customers wanting only a patent license, you can contact us to learn more about the exFAT licensing program and request a pricing proposal."
No its inclusion in Linux was the result of a decision by Microsoft, not a granting of an open license. Any other implementation still has to apply for a license:
"For customers wanting only a patent license, you can contact us to learn more about the exFAT licensing program and request a pricing proposal."
I'm also not a lawyer, but AFAIK publishing a specification has nothing to do with whether the tech can be legally used without a patent license.
I don't know how the release to Open Invention Network affects things, though.
I don't know how the release to Open Invention Network affects things, though.
> Would you care to elaborate for why that is? Would you care to elaborate for why that is?
To put it bluntly, if Microsoft's guarantees are good enough for Linus so as to merge the driver in the Linux mainline, they're good enough for me too.
To put it bluntly, if Microsoft's guarantees are good enough for Linus so as to merge the driver in the Linux mainline, they're good enough for me too.
Much before MS did that, macOS had builtin exFAT support presumably licensed from MS, Paragon had exFAT support for android for a one time fee which works with other file manager apps(also licensed from MS), On Linux Exfat-utils enabled support for exFAT(open-source implementation).
Has there been any changes to these since MS has opened the license to exFAT?
Has there been any changes to these since MS has opened the license to exFAT?
Third-party exFAT support did exist before 2019, but vendors had to pay a licence fee to Microsoft. Apple apparently paid the fee.
Regarding Linux, exfat-utils only provided limited functionality. The kernel mainline lacked a native exFAT driver. Mounting exFAT volumes was possible only through FUSE (with all its shortcomings).
Regarding Linux, exfat-utils only provided limited functionality. The kernel mainline lacked a native exFAT driver. Mounting exFAT volumes was possible only through FUSE (with all its shortcomings).
The change is that now there is an exFAT driver included in the Linux kernel.
Therefore, if you have a recent Linux kernel, you no longer need to install a user-mode driver for exFAT and the kernel driver has better performance.
Therefore, if you have a recent Linux kernel, you no longer need to install a user-mode driver for exFAT and the kernel driver has better performance.
Huh, I thought exFAT was much older, not 2019.
It is – it dates back to 2006. But before 2019, the spec was proprietary and patent-encumbered. And the Linux kernel didn't include a good driver for it until 5.7, released in May of last year[0].
[0]: https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system
[0]: https://kernelnewbies.org/Linux_5.7#New_exFAT_file_system
>TIL: In 2019, Microsoft released the exFAT [1] specs [...] I.e. exFAT should now be preferred over FAT/NTFS for media used across different operating systems.
You seem to be very optimistic about how fast these get implemented into embedded devices.
You seem to be very optimistic about how fast these get implemented into embedded devices.
I have a 2011 car which has a USB port where I can plug a USB stick with MP3s (remember those, kids?) on them, I doubt it will ever be made ExFAT-compatible.
OTOH, I do have the Windows logo on my steering wheel, and somewhere inside the car there's a Windows CE system installed (it doesn't have a multi-color LCD screen and interacts using speech recognition, so it's not really obvious).
OTOH, I do have the Windows logo on my steering wheel, and somewhere inside the car there's a Windows CE system installed (it doesn't have a multi-color LCD screen and interacts using speech recognition, so it's not really obvious).
Maybe there are even capable embedded implementations even before Microsoft implements all supported features like ACLs in their OS.
I would imagine if embedded devices are being assembled and configured with filesize concerns, then exFAT would be the natural implementation they would opt for.
Or do you mean updating/transitioning legacy embedded?
The spec doesn't mention backwards compat, but does mention as a goal
>Retain the simplicity of FAT-based file systems.
I would like to test out if an input embed system is exFAT compat. And what might be required to update.
The spec doesn't mention backwards compat, but does mention as a goal
>Retain the simplicity of FAT-based file systems.
I would like to test out if an input embed system is exFAT compat. And what might be required to update.
I believe the implication is SD card/USB drive support on microcontrollers without an OS.
SD under 32GB uses FAT32 so you can’t drop supports, and FAT32 stack is often already pushing the maximum ROM size, so no room is left for a novel filesystem. Lots of developers aren’t themselves proficient at writing filesystem code as well, and there aren’t as many copy-pastable exFAT implementations as there are for FAT32.
So unless they know what they’re doing and unless it’s critical to their product, the majority won’t easily move onto exFAT.
As for the use within the device “firmware”, the user defined program is usually just a giant baremetal a.out in the middle of the ROM at a specified address, so no FS is involved there.
SD under 32GB uses FAT32 so you can’t drop supports, and FAT32 stack is often already pushing the maximum ROM size, so no room is left for a novel filesystem. Lots of developers aren’t themselves proficient at writing filesystem code as well, and there aren’t as many copy-pastable exFAT implementations as there are for FAT32.
So unless they know what they’re doing and unless it’s critical to their product, the majority won’t easily move onto exFAT.
As for the use within the device “firmware”, the user defined program is usually just a giant baremetal a.out in the middle of the ROM at a specified address, so no FS is involved there.
Embedded devices often have a lifespan measured in decades. And for short-lived devices like consumer all-in-one printers (~4 years) it's not like every new model has a new firmware written from scratch. They get whatever the last model had, with a few updates for new device features. That might include exFAT, or it might not.
Unfortunately, this only applies to systems that actually have this (i.e. they were updated after exFAT released or released after and included this update). There will be sadly many systems that will never receive this update.
But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). FAT32 can go up to 2 TB. So I have found that reformatting a larger media back to FAT32 makes it recognizable to that system with all of the storage space.
But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT). FAT32 can go up to 2 TB. So I have found that reformatting a larger media back to FAT32 makes it recognizable to that system with all of the storage space.
> But for a fun fact, if you see many embedded systems that say they can only support an SD card up to 32 GB, it is a soft limitation due to the fact that media over 32 GB are pre-formatted to exFAT (and the system lacks support for exFAT).
There might also be the issue of the card reading hardware only supprting SDHC instead of SDXC (or SDUC).
There might also be the issue of the card reading hardware only supprting SDHC instead of SDXC (or SDUC).
Even then it is actually only a driver or firmware limitation most of the time.
My old laptop has Ricoh SDHC card reader. Windows can't read 64GB SD cards (properly formatted with exFAT according to the standard) on it because it's using the proprietary Ricoh driver.
On Linux it works just fine because it's using the generic SD card reader driver which supports SDXC.
My old laptop has Ricoh SDHC card reader. Windows can't read 64GB SD cards (properly formatted with exFAT according to the standard) on it because it's using the proprietary Ricoh driver.
On Linux it works just fine because it's using the generic SD card reader driver which supports SDXC.
That is true, though I have yet to run into that issue though for the devices I have tried it on (which admittedly is not a large sample size).
I guess that's the best option for compatibility. But it's not journaled and single fat, very prone to corruption right and fixing errors could take a while for lager disk like external hdd. There's not really any good option in terms of reliability and compatibility I guess.
I have had issues with ntfs getting corrupted and losing file both in mac and linux at times so there's that risk with implementation too.
I have had issues with ntfs getting corrupted and losing file both in mac and linux at times so there's that risk with implementation too.
The original video this article is referencing is here if anyone wants to just watch it instead: https://www.youtube.com/watch?v=bikbJPI-7Kg
If my memory serves well, when 40+ GB HDDs started being common, you were either using Windows 2000 and NTFS already, or it was probably a good practice to have several partitions, given the overall unreliability of Windows 9x and the not so infrequent clean reinstallations. And then came Windows XP and the issue was moot.
It's a bit of a shame Microsoft didn't push and open up exFAT earlier and allowed FAT32 to become an issue again, only with removable flash drives and memory cards this time.
It's a bit of a shame Microsoft didn't push and open up exFAT earlier and allowed FAT32 to become an issue again, only with removable flash drives and memory cards this time.
> It's a bit of a shame Microsoft didn't push and open up exFAT earlier and allowed FAT32 to become an issue again, only with removable flash drives and memory cards this time
Erm, because that was the idea? SDXC is a big racket and Microsoft has been collecting patent fees from any kind of consumer device that's capable of reading memory cards thanks to the filesystem requirements in the standard.
Erm, because that was the idea? SDXC is a big racket and Microsoft has been collecting patent fees from any kind of consumer device that's capable of reading memory cards thanks to the filesystem requirements in the standard.
The problem with removable drives is that no one came up with a better filesystem than FAT32 that could be implemented easily by everyone. All the *NIX filesystems embedded permissions that were useless on non-POSIX systems and caused issues on POSIX ones when you actually moved a drive from one computer to another. So MS extended FAT again and kept it under its IP hat, as was their wont[0] at the time.
[0] Non-native english speakers may be confused by this: https://www.merriam-webster.com/dictionary/as%20is%20someone...
[0] Non-native english speakers may be confused by this: https://www.merriam-webster.com/dictionary/as%20is%20someone...
FAT embedded a long/short name mapping algorithm that a modern system shouldn't need, but Microsoft had patents on. The main feature of FAT is simple native support on all platforms, Microsoft never had a financial incentive to add support for alternatives on Windows.
The choice he made was probably a reasonable balance -- the alternative was reinventing the filesystem, which was probably a bit out of spec for implementing the GUI format dialog. And eventually happened, but it took a while.
Its not reinventing. Its just setting a limit on max cluster slack. Like he says in the video, you can totally use the CLI and format any disk size you want, its just a soft block
I hit this a couple of days ago when helping a friend set up their new DJ decks. The decks only supported FAT32 but would happily read a 128gb USB stick.
The article states that you can format as fat32 from the command line - this wasn't my experience. Disk Util (cli) saw my raw partition and started formatting it as FAT32, but seemed to fail about an hour in saying the disk was too big.
I ended up finding a random abandoned GUI utility which seemed to work, though it's extremely infuriating.
The article states that you can format as fat32 from the command line - this wasn't my experience. Disk Util (cli) saw my raw partition and started formatting it as FAT32, but seemed to fail about an hour in saying the disk was too big.
I ended up finding a random abandoned GUI utility which seemed to work, though it's extremely infuriating.
I suspect the command the article said is diskpart.
Hopefully exFAT implementations will get more robust. At least with the macOS implementation, it has always been prone to reporting corruption when devices are not ejected properly. I can usually fix it by running check disk or whatever on Windows, but sometimes not.
One change from previous FAT filesystems to exFAT is the removal of the second, redundant file allocation table. Disks are less unreliable now so it is less necessary, but there are still some scenarios, especially when the filesystem is in a dirty state, where the table can be corrupted.
This can also really hurt in unexpected ways with otherwise badly written drivers. Nintendo's driver for the switch would constantly write to the table unnecessarily, and while this doesn't do all that much in normal operation (other than waste a small number of writes), it made it drastically more likely (in some cases for some games almost guaranteed) that the table would be partially written and corrupt if the system crashed or lost power unexpectedly. When some new Pokemon games came out and were initially a bit unstable this came up a lot.
This can also really hurt in unexpected ways with otherwise badly written drivers. Nintendo's driver for the switch would constantly write to the table unnecessarily, and while this doesn't do all that much in normal operation (other than waste a small number of writes), it made it drastically more likely (in some cases for some games almost guaranteed) that the table would be partially written and corrupt if the system crashed or lost power unexpectedly. When some new Pokemon games came out and were initially a bit unstable this came up a lot.
I've had more filesystem corruption trouble with exFAT on Linux than with any other filesystem, ever, including Btrfs. Really crossing my fingers and hoping for the quality of the Samsung driver in Linux 5.7. It's going to be great to have an option for high-capacity, cross-platform read/write storage. The day I never have to use NTFS-3G again cannot come too soon.
So, since we're on kernel 5.10, has this new Samsung exFAT driver been tested? Is it at least as reliable as btrfs?
> The day I never have to use NTFS-3G again cannot come too soon.
That day might be soon
https://lkml.org/lkml/2020/12/31/126
That day might be soon
https://lkml.org/lkml/2020/12/31/126
I know everyone talks about 2020 having felt like a long year, but I had totally forgotten that Paragon had started trying to submit that back in August[0]. I'm really glad to see this is progressing. What a monumental set of changes. This won't change the NTFS situation on macOS, of course but even in a post-exFAT world, NTFS will remain a relevant concern for Linux/Windows dual-booting, and I'd love to see this get merged.
[0]: https://lore.kernel.org/linux-fsdevel/2911ac5cd20b46e397be50...
[0]: https://lore.kernel.org/linux-fsdevel/2911ac5cd20b46e397be50...
Were you using the FUSE driver?
That's a great question. I kind of skirted around indicating one way or the other in my parent post because I've forgotten. My troubles were as recent as the beginning of 2020, but I encountered them on a Raspbian machine, so Lord knows what kernel version it was running at the time. Probably pre-5.4, in which case, yes, it would've been the FUSE driver.
Thanks it was probably the FUSE driver. I haven't had a problem with it but I'm not a heavy user.
On the other hand I've been a heavy user of exFat formatted sd cards on Samsung phones for long years and had no problems with it, so hopefully the new Linux driver (which Samsung upstreamed) will be solid.
On the other hand I've been a heavy user of exFat formatted sd cards on Samsung phones for long years and had no problems with it, so hopefully the new Linux driver (which Samsung upstreamed) will be solid.
Many embedded devices can read FAT32 and that's it. If you need to use a drive larger than 32GB for these, there are tools that will let you format them as FAT32 from within Windows.
One example: FAT32Format GUI - https://www.softpedia.com/get/System/Hard-Disk-Utils/FAT32fo...
One example: FAT32Format GUI - https://www.softpedia.com/get/System/Hard-Disk-Utils/FAT32fo...
It also conveniently allowed Microsoft to collect license fees from ExFAT patents on all SDXC devices.
Maybe it's not for the worst, as otherwise there might have been less push for exFAT and this inefficiency would have persisted much later..?
I rarely read The Register. Is that the usual tone -- as if the author wishes they were Lady Whistledown, the gossip writer on Bridgerton?
Maybe with a little Variety mixed in, but you're not far off the mark. Definitely in the standard British vein of snarky tabloid writing.
Yes, pretty much.
It is, it's similar to Private Eye in the UK or
Le Canard enchaîné in France.
TL;DR: "simple lack of foresight combined with the age-old problem of the temporary solution becoming de-facto permanent"
More directly: He only expected his code to be used in Windows NT and made the fatal flaw of thinking that just because he slapped it together in an hour it would be easy for someone to rewrite in the next version of Windows to up the limit.
Instead the code sat untouched for decades as the limit slowly became more and more unreasonable.
IMHO the real issue is the artificial limit. Instead of straight up disallowing you from creating large filesystems, it should have had a dialog box warning about small files being inefficient but letting you make it anyway.
Instead the code sat untouched for decades as the limit slowly became more and more unreasonable.
IMHO the real issue is the artificial limit. Instead of straight up disallowing you from creating large filesystems, it should have had a dialog box warning about small files being inefficient but letting you make it anyway.
>Instead the code sat untouched for decades as the limit slowly became more and more unreasonable.
My projects likely won't span decades but when I know I'm putting in some opinionated value I leave behind a comment like:
// XXX picking 32gb because it'll probably be 5 years until we have >32gb hard drives. Change this when the world changes!
My projects likely won't span decades but when I know I'm putting in some opinionated value I leave behind a comment like:
// XXX picking 32gb because it'll probably be 5 years until we have >32gb hard drives. Change this when the world changes!
Exactly. Just another case of "640K should be enough for anybody".
But we have to admit that storage has leapt hugely in size and very rapidly. In 1990 or thereabouts, a common hard disk size was about 30 megabytes. Today I have files that are many times that size, and we think that that is quite normal. A mere five or so years later around 1996, that size had jumped to 4000 megabytes! These days my laptop has 6 million megabytes.
We can't blame filesystem planners for lack of foresight. They were blind-sided by progress just like the rest of us were.
But we have to admit that storage has leapt hugely in size and very rapidly. In 1990 or thereabouts, a common hard disk size was about 30 megabytes. Today I have files that are many times that size, and we think that that is quite normal. A mere five or so years later around 1996, that size had jumped to 4000 megabytes! These days my laptop has 6 million megabytes.
We can't blame filesystem planners for lack of foresight. They were blind-sided by progress just like the rest of us were.
He had already seen exponential growth in storage capacity and could easily extrapolate from that. Moore's law was coined in the '60s, after all.
This wasn't a lack of foresight, this was being prudent. If I can't come close to testing an upper limit, it's correct to acknowledge it may degrade in a way I haven't anticipated.
Especially since he was writing a GUI which should generally avoid handing users footguns, and which can be updated over time.
This wasn't a lack of foresight, this was being prudent. If I can't come close to testing an upper limit, it's correct to acknowledge it may degrade in a way I haven't anticipated.
Especially since he was writing a GUI which should generally avoid handing users footguns, and which can be updated over time.
I have a single file that is 6 million megabytes in size.
who/what reads that file?
I haven't managed a 6TB file before, but I did have a >100GB log file I ended up parsing with awk just to be sure that the memory consumption on the consuming process wouldn't go crazy.
Not my file, but I have produced HDF5 files in that size range using simulation codes on supercomputers and paraview can open them and render 3d view as video from it (eventually).
tar
That stretches the definition of "a single file" when it is basically a filesystem itself.
Not necessarily, it's all relative really. But the point is, the limitations or expected limitations are only relevant until they're not..
Also why not tar a drive for archive if I want :}
Also why not tar a drive for archive if I want :}
What is the practical difference between exFAT and FAT32 when you can format FAT32 up to many GB?
Is it just that they are different formats, and exFAT has taken over, in terms of pre-formatted USB media?
Is it just that they are different formats, and exFAT has taken over, in terms of pre-formatted USB media?
FAT32 also has a 4 Go file size limit (think large video/VM files, DVD/Blu-ray images…)
Actually NT4.0 did not support FAT32 at all (unless you installed a little-known driver from Sysinternals I've never actually seen used in the wild).
So maybe his work was on Windows 2000?
So maybe his work was on Windows 2000?
"Prototypes have a bad habit of becoming the product".