unqueued · HackerTrans

unqueued

no profile record

comments

l’année dernièrediscuss

> The alternative is that people complain "netflix doesn't work on firefox", switch to chrome instead, which is even worse.

So what if users complain? How is it better for Firefox to do something bad just because Google is doing something bad?

Firefox is supposed to provide an alternative to what's out there. Firefox also didn't support some popular proprietary Internet Explorer features, and they never attempted to. For a time, much of MySpace didn't work as well in Firefox. But I'm glad that Firefox didn't cave, even if some users complained that they couldn't make the scrollbars neon green or make music autoplay.

Not letting Microsoft or Google dictate how they implement a web browser worked out really well for them. Chasing proprietary platforms has ruined them.

At the time EME was adopted Firefox was much more popular, I think 20% back in 2012. Video platforms were using Silverlight for DRM. There's a good chance that EME would not have gotten off the ground if Mozilla didn't embrace it, or at least not as quickly.

Mozilla should have taken a stand and refused to support EME when they had the chance. They would be better off than they are now. And there's a good chance Netflix would not have thrown away a double digit percentage of subscribers.

Instead they embraced DRM and now they have nothing.

l’année dernièrediscuss

I wish more were done to push back against this consolidation of power by these platforms.

We're like frogs that have been stoking the fuel of our own pot.

It used to be that DRM was considered to be in conflict with the browser, because it was not acting on behalf of the user. If you must have DRM, then it is on the platform to shoehorn it in through an external plugin, like Silverlight.

When Firefox adopted EME extensions, I knew it was the beginning of the end; they were rolling out the red carpet for DRM. If we make DRM a switch that can simply be thrown, then it will become the norm, not the exception. And there have been proposals for years to DRM fonts and other absurdities. If a company insists on using DRM, then they should have to shoulder the burden of doing something that a browser was never support to support.

The nightmare that we're racing toward is you will only be permitted to cache a trickle of video at a time and your TPM attestation hardware must include a token in every HTTP request. Your browser will just be a software cablebox.

They aren't happy about URLs either and would love to require that if you want to share a reference to something, you have to do it on their terms, like generating a url in their app with a hash that expires and limited in how many times it can be viewed. I'm sure influencers will still have the privilege of unlimited sharing.

They've been slowly rolling this infrastructure out for the last decade. These are not isolated inconveniences, these are coffin nails.

il y a 2 ansdiscuss

I think symlinks are really underappreciated for organizing files.

One cool thing you can do is to treat directories as tags when you symlink into them, since a file can 'belong' to multiple directories.

If you're using Windows, you can just use WSL and then you don't need administrative access as the OP's program does.

Here are some simple tools to let you use symlinks as tags:

https://github.com/perses76/refmat-symlink

https://mikeknowl.es/projects/ztag/

https://github.com/michaelknowles/ztag

But if anyone is curious about ways to better organize their files in general, I would check the writings of Karl Voit:

https://archive.baty.net/2018/using-karl-voits-file-naming-s...

https://karl-voit.at/2022/01/29/How-to-Use-Tags/

Simple and portable self describing filenames has served me really well for a long time across many platforms.

This script has done a lot of lifting for me and it will work with WSL with little effort:

https://github.com/mdom/squaretag

il y a 2 ansdiscuss

If you are iterating over a lot of files, a read while loop can be a major bottleneck. As long as you use the null options from find and pipe into xargs, you should be safe with any filename.

I've found it can reduce minutes down to seconds for large operations.

If you have to process a large number of files, you can let xargs minimize the number of times a program is run, instead of running it once per file.

Something like:

  # Set the setgid bit for owner and group of all folders
  find . -type d -print0 | xargs -0 chmod g+s

  # Make the targets of symlinks immutable
  find . -type l -print 0 | xargs -0 readlink -z | xargs -0 chattr +i

Way faster. But there are lots of caveats. Make sure your programs support it. Maybe read the xargs man page.

il y a 2 ansdiscuss

There is a pretty good syntax for dealing with nasty filenames, if you must: ANSI-C quoting[1].

If you have to output in a shellscript in this format, use printf %q

from man printf:

       %q     ARGUMENT is printed in a format that can be reused as shell input, escaping non-printable
              characters with the proposed POSIX $'' syntax.

It is just $'<nasty ansi-c escaped chars>'

$ touch $'\nHello\tWorld\n' $ ls

One thing I do like about a filesystem that fully supports POSIX filenames is that at the end of the day a filesystem is supposed to represent data. I think it is totally sensible to exclude certain characters, but that it should be done higher up in the stack if possible. Or have a flag that is set at mount time. Perhaps even by subvolume/dataset.

One thing I haven't seen mentioned is that POSIX filenames are so permissive that they allow you to have bytes as filenames that are invalid UTF-8. That's why the popular ncdu[2] program does NOT use json as it's file format, although most think it does. It's actually json but with raw POSIX bytes in filename fields, which is outside of the official json spec. That does not stop folks from using json tools to parse ncdu output though.

Another standard that is also very permissive with filenames is git. When I started exploring new ways to encode data into a git repo, it was only natural that I encountered issues with limitations of filesystems that I would check out in.

Try cloning this repo, and see if you are able to check it out: https://github.com/benibela/nasty-files

It is amazing how many things it breaks.

If you are writing software that deals with git filenames or POSIX filenames (that includes things like parsing a zip file footer), you can not rely on your standard json encoding function, because the input may contain invalid utf-8. So you may need to do extra encoding/filtering.

[1]: https://www.gnu.org/s/bash/manual/html_node/ANSI_002dC-Quoti...

[2]: https://dev.yorhel.nl/ncdu/jsonfmt

il y a 2 ansdiscuss

Both Finder and explorer.exe will mung mtimes from casual browsing, usually due to metadata updates. It is a common enough problem that there are utilities to bind mtimes to files.

IMO mtime the de-facto file timetsamp metadata, since it is most widely supported. The other file timestamp metadata I found were not as useful or portable.

And people use git for storing more than just source code.

I personally find it very useful to be able to clone my podcast index and have accurate timestamps not just from the commit, but from the file itself.

Timestamps can be important for a collections of files, but they are fragile and often an afterthought.

il y a 2 ansdiscuss

Sometimes search indexing and build tools get it wrong, so if you know what you're doing, then tools like this allow you to provide an override.

Just browsing an old photo collection without making any changes can cause a 15 year old file to show up as a "new file", even though it's contents have not changed. This is especially true with media files that the os might take it upon itself to regenerate metadata for.

Or you may wish to store an accurate snapshot of the timestamps of cached or intermediate build artifacts.

il y a 2 ansdiscuss

Thanks for sharing, I love stuff like this. I get frustrated at mtimes being munged. MacOS finder is terrible about this.

If anyone is interested, I wrote a more generalized tool for storing metadata called storetouch[1] which takes a snapshot of file's mtimes, and stores them in a nicely formatted file for convenient restoration. You can run it's output by itself with bash, or easily parse it's output.

I found that the commit or author date's did not always make sense to represent the true age of a file. Especially for non-source-code documents. For example, a repo of scanned documents. Or a git-annex repo of podcasts going back 20 years.

Instead, I treat the commit date as the commit date, and I have a versioned sidecar that stores the actual original mtimes of the files.

It can also handle all valid POSIX filenames (a surprising amount of utilities don't, including tools like git-restore-mtime, last time I checked). If you're already using a tool like git-annex or datalad though, they have the ability to store metadata for each file as well.

I would also suggest checking out git-store-meta[2] which was the original inspiration. It stores more than just mtime, but it is more closely tied to git.

[1]: https://github.com/unqueued/storetouch

[2]: https://github.com/danny0838/git-store-meta

il y a 2 ansdiscuss

You can already use it with git-annex to store binaries using the git-annex-remote-git-aafs[1] special remote.

Although I would be careful and make sure you understand what it is doing to your branch namespace. Even though in the worst case it would not save any space over directly committing binaries, they are in orphan branches that can be pruned without rewriting history.

But even so, you can just use any number of git-annex special remotes to bypass using git for sharing files.

They may eventually add first-party support for git-annex. But nothing is stopping you from using it now.

[1]: https://github.com/domo141/git-annex-remote-git-aafs

il y a 3 ansdiscuss

I think "confabulation" is a way more accurate term, I wish it had stuck instead of "hallucination".

A hallucination is a problem with input. Confabulation is false output.

Confabulation is when a person mistakenly recalls details and tries to "fill in the blanks", without realizing what they are saying is untrue.

il y a 3 ansdiscuss

That's what's great about btrfs. I started using it that way too. It was nice to be able to take instant snapshots of my fs. I just amended my existing backup scripts to act on temporary snapshots instead. Then, I started dividing things up into subvolumes.

I suggest moving your filesystem root to its own subvolume (the convention is to call it @root), then optionally mounting the actual filesystem root as like /mnt/fsroot, and mounting it by passing subvol=/. That is where I keep my snapshots. You can still have /home be nested in @root. I have subvolumes for /var/cache, /var/lib/docker, etc. And more fragmented qemu images just go on a NoCOW subvolume with +C set, and I back up normally.

I also highly suggest reflinks. Once you start using them you will use them all the time.

I don't know if zfs supports reflinks yet, at the time it didn't. But combining reflinks with subvolumes solves a lot of problems. Since btrfs doesn't support recursive snapshotting, you can just do cp --reflink=always -rp @root /path/to/new-subvolume, and it will combine the contents of the subvolumes without consuming any more space. And of course, btrfs-send will (usually) not duplicate reflink data.

il y a 3 ansdiscuss

I will probably give bcachefs on one of my machines.

I started using btrfs back when it was way less stable, but that instability was more than offset by the other features.

The last time I lost a btrfs filesystem was four years ago when it was less stable, but it did not matter because it only amounted to about an hour of logs lost, and I reconstructed it in minutes. So btrfs is a must on any embedded devices. It frees me to use cheaper media, because I can have two copies of all data and metadata, and compression, and fine grained external snapshots.

Zfs is great for making big complex arrays and if you plan out how your are storing your data ahead of time. But it has a steeper learning curve and is more seperate from the underlying OS.

Btrfs fills this perfect niche for me of being a drop in replacement for ext4, and being really flexible and offering some of the nicer features of zfs. (differential snapshots, checksumming, compression)

So I'm excited to try bachefs as an alternative to btrfs.

il y a 3 ansdiscuss

This is an example of how btrfs is poorly handled by many userland tools by default. You should never ever need to have thousands of snapshots.

I use btrbk to manage my snapshots. It keeps my snapshots pruned to 300, pruning intermediate snapshots, so I can still easily step back 52 weeks, or I can purge them if I want to improve performance and reclaim space. While at the same time, every single snapshot is sent to two compressed encrypted backup images over ssh. The diff algorithm is pretty good, but for some types of files that fragment frequently, it is best to put them on their own subvolumes. My places.sqlite files from my browser testing profiles were adding tens or hundreds of MiB to snapshots, so they went in a seperate subvolume. I don't think the zfs would perform any better.

I don't blame people for having bad experiences with btrfs, it is very inconsistently documented.

I think it is especially annoying how many userland tools keep every single snapshot as a subvolume of the root, so that you have tens of millions of read-only files stored in /.snapshots. Awful.

il y a 3 ansdiscuss

I actually have something I've been meaning to put on github. I wrote it over a decade ago and I need to dockerize it. I only made it in Vagrant.

But it can do something iVentoy can not.

https://www.iventoy.com/en/faq.html:

   The computer which run iVentoy must be in the same LAN with them.
   Besides, there must no other DHCP servers in the same LAN.

It can coexist with an existing DHCP server. That means you can just start it up on your own network without any modifications, and start netbooting machines.

It turns out that multiple DHCP servers can co-exist on the same network segment. So I just have a second dhcp server solely for a pxe bootstrapper. At the time, at least the initial download must be tftp, but I used a stripped down ipxe stub that took advantage of the universal undi network drivers, and then loaded the rest (usually syslinux) over a fast HTTPS connection.

From there, iSCSI was no problem, and I could also use tools like plop, memdisk, or grub4dos, as well as Windows 7/8/10 with wimboot.

I remember for some distros like Knoppix, we would use an NBI driver to just mount the iso directly over the network, which could be far simpler and more performant than NFS.

EDIT: To be honest, it is simple enough that it doesn't have to use Docker. It is mostly dnsmasq and PHP.

il y a 3 ansdiscuss

My biggest mistake after being laid off was to not be discerning enough about my next job. You need to be willing to say no to a crummy job offer, even as the bills pile up.

il y a 3 ansdiscuss

John, at around that time you mentioned, you gifted me with my first shell account on one of your coffeshop servers, Reva. I just wanted to say THANK YOU. Up until then, I had read books about unix from the library, and had failed to install Mandrake on my laptop, but being able to access such a well configured NetBSD environment was really helpful and set me on a good path.

I had never been truant from school before up to that point, but sometimes life rewards you for taking risks.

il y a 4 ansdiscuss

I was hoping that using hardware keys would eliminate some of the security hoops that we have to jump through. And it does seem to help. But the whole reason that I have a key is so I do not have to supply my phone number, and I have a more trusted way of proving my identity, even if I am connecting from an unusual location.

il y a 4 ansdiscuss

This makes me worried, because I am pretty sure Google is going to start removing keys based on attestation certificates.

I believe that this is much more about rate limiting than about security for the end users.

il y a 4 ansdiscuss

No, that is not correct, git-annex uses a variety of special remotes[2], some of which support deduplication. Mentioned in another comment[1]

When you have checked something out and fetched it, then it consumes space on disk, but that is true with git-lfs, and most other tools like it. It does NOT consume any space in any git object files.

I regularly use a git-annex repo that contains about 60G of files, which I can use with github or any git host, and uses about 6G in its annex, and 1M in the actual git repo itself. I chain git-annex to an internal .bup repo, so I can keep track of the location, and benefit from dedup.

I honestly have not found anything that comes close to the versatility of git-annex.

[1]: https://news.ycombinator.com/item?id=33976418

[2]: https://git-annex.branchable.com/special_remotes/

il y a 4 ansdiscuss

I have a 1.96 TB git repo: https://github.com/unqueued/repo.macintoshgarden.org-fileset (It is a mirror of a Macintosh abandoneware site)

  git annex info .

Of course, it uses pointer files for the binary blobs that are not going to change much anyway.

And the datalad project has neuro imaging repos that are tens of TB in size.

Consider whether you actually need to track differences in all of your files. Honestly git-annex is one of the most powerful tools I have ever used. You can use git for tracking changes in text, but use a different system for tracking binaries.

I love how satisfying it is to be able to store the index for hundreds of gigs of files on a floppy disk if I wanted.