Good question. I agree with what I think you're implying, which is that local generation is not the right choice if you want to maximize results per time/$ spent. In my experience, hosted models like Claude Opus 4.6 are just so effective that it's hard to justify using much else.
Nevertheless, I spend a lot of time with local models because of:
1. Pure engineering/academic curiosity. It's a blast to experiment with low-level settings/finetunes/lora's/etc. (I have a Cog Sci/ML/software eng background.)
2. I prefer not to share my data with 3rd party services, and it's also nice to not have to worry too much about accidentally pasting sensitive data into prompts (like personal health notes), or if I'm wasting $ with silly experiments, or if I'm accidentally poisoning some stateful cross-session 'memories' linked to an account.
3. It's nice to be able solve simple tasks without having to reason about any external 'side-effects' outside my machine.
Supporting all the various devices does sound quite challenging.
I wouldn't expect a perfect single measurement of "quality" to exist, but it seems like it could be approximated enough to at least be directionally useful. (e.g. comparing subsequent releases of the same model family)
This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question:
"What is the highest-quality model that I can run on my hardware, with tok/s greater than <x>, and context limit greater than <y>"
(My personal approach has just devolved into guess-and-check, which is time consuming.) When using TFA/llmfit, I am immediately skeptical because I already know that Qwen 3.5 27B Q6 @ 100k context works great on my machine, but it's buried behind relatively obsolete suggestions like the Qwen 2.5 series.
I'm assuming this is because the tok/s is much higher, but I don't really get much marginal utility out of tok/s speeds beyond ~50 t/s, and there's no way to sort results by quality.
Thanks for the response! I hear what you're saying, and I apologize for my joke.
I re-read your essay, and what you say about needing sophistication reminds me of the concept of proof-of-work -- "sophistication" could be a way to convey that effort was spent by a writer, even if it doesn't add meaning. That is kind of inherently annoying, because it implies a lack of trust between the author and reader, and in the thesaurus example, the reader would be rightfully annoyed to spent time parsing a sentence only to find that the "proof of effort" was actually just a "facade of effort".
It's clever that the the author provides both his essay and an example at the same time! Sorry, that joke felt obligatory.
Miscellaneous reactions, in an elegant bulleted list:
- "Simple" sentences are certainly expressive, but "elegant" wording expands the set of meanings that can be conveyed. And vice-versa
- I think a lot of the meat of a sentence is conveyed in the connotations of words and not their literal meaning. "Simple" wording is necessarily more common, and therefore will necessarily have a less specific or reliable connotation. This is a blessing and a curse.
- More subjectively, I think ideal writing is also a window into the author's experience of the world (or moreso whatever topic they're writing about), and as a reader, I want that to come through in an authentic way that matches the author's experience. So, using a thesaurus and agonizing over sentence structure might end up 'elegant' but still vaguely bad, but on the other hand if you agonize over a sentence and come up with something more "sophisticated" that ultimately rings truer to you, then go for it.
- ^ The above points aren't direct rebuttals to TFA, but I think they relate to why elegance can be appealing.
I might be misunderstanding your point, but quantization can have a dramatic impact on the quality of the model's output.
For example, in diffusion, there are some models where a Q8 quant dramatically changes what you can achieve compared to fp16. (I'm thinking of the Wan video models.) The point I'm trying to make is that it's a noticeable model change, and can be make-or-break.
Yeah. In the example I shared, my charitable interpretation would be that it's identifying the trick question as "a setup" where the punch line is the confusion the audience experiences. And in a meta sense, that would also describe the form of the entire chat.
The recursive one that I have actually been really liking recently, and I think is a real enough challenge is: "Answer the question 'What do you get when you cross a joke with a rhetorical question?'".
I append my own version of a chain-of-thought prompt, and I've gotten some responses that are quite satisfying and frankly enjoyable to read.
I checked out your docs, and I agree that they flow very nicely! So often it seems that docs are either frustratingly vague to the point where it almost seems like the company is embarrassed to admit that their tool is a tool and not a "transformative synergy experience" (or similar nonsense), or docs immediately get overly specific without covering "why this product exists".
Minor note: The only thing in your docs that made me pause was the repeated use of "CDC", which I had to google the definition of. (For context, I have implemented "CDC" several times in my career, but wasn't familiar with the acronym)
True, perhaps "messy" was the wrong word. I think what I was trying to get at was that I feel "There Is No Antimemetics Division" is more accessible for the average reader -- More narrowly focused, with a more immediate hook.
Regarding recommendations similar to Ra, it's not exactly the same thing, but https://unsongbook.com/ is fantastic and has a similar flavor I think.
I sent out about 8 copies of this book to friends/relatives for christmas a couple years ago, and it was very well received. It's not without flaws, but it's absolutely packed with novel concepts, relatively short, and provides a unique experience for unsuspecting readers. The physical book cover is quite good looking as well in my opinion.
I also loved Ra by the same author, but it felt a little messier plot-wise, so I hesitate to recommend it to an audience who isn't already accustomed to reading "out-there" online/sci-fi/rationalist fiction.
I don't think this is a good example. The bus is clearly a simplified representation. If the answer/explanation given was actually correct then you'd also expect the bus to have other details like side view mirrors.
Edit: I hate to dwell on this too much, but even if a door and mirrors were visible, the bus could be in reverse. This seems to be more of a case of accidentally picking the intended answer due to a lack of knowledge. (Of levels of abstraction in representation and of vehicle design.) That said, I don't necessarily disagree with the underlying point being made.
Are you also saying that thoughts are formed using gradient descent? I don't think gradient descent is an accurate way to describe either process in nature. Also, we don't know that we "see" everything that is happening, we don't even understand the brain yet.
Idea: It would be super cool if the camera angle also shifted a bit when you tilt your phone, making the level look more like a real object. You would have to assume the relative position of the phone from the viewer, but that assumption is already being made (given that the view of the level is not from directly overhead currently).
Butter sticks in the US have marks down the side of the wrapper, like a ruler, that show you tablespoon increments. Google image search "tablespoons butter stick".
Interesting idea, thanks! I will probably try this. I had stopped tweaking things too much after implementing MPAA, because most of the searches end up "short circuit"-ing relatively quickly.
With MPAA you preserve all previous shortest paths that have been found (in the form of an Array where array(nodeIndex) points to the nodeIndex of the next node on the path). When attempting to optimistically follow one of these paths, if the algorithm hits a new wall, it will null out the path it took up until the wall. However a future search (or iteration in the same search) may still reuse the part of the shortest path that still exists after the wall.
That said, you're right that it's a tiny grid, and cache behavior can dominate any abstract theoretical properties in surprising ways.
I've spent a lot of time thinking about fast pathfinding in order to speed up my Scala Quoridor AI*[0], and here are some tips/tricks I've learned:
- MPAA (multipath adaptive A*) is great if you need to re-search the same area multiple times as obstacles are introduced. It allows you to feed in the results of previous searches in order to speed up pathfinding.
- JPS (jump point search) looks very appealing in theory because it allows you to significantly reduce the number of "nodes" considered, but I found that the increased overhead in identifying jump points meant it didn't provide a speedup. There might be a way to marry the ideas of MPAA and JPS, but it is very easy to conceptually shoot yourself in the foot with minor conceptual details when getting creative with the algorithms. (As a contrived example, using a ">" when you needed a ">=" might mean you are no longer guaranteed to output the real shortest path in certain situations)
- Instead of using a proper heap for storing open nodes, consider using a bucketed priority queue if your max priority value is a relatively low integer. This means there's an underlying array indexed by priority, which makes push'ing and pop'ing quite fast.
[0] Quoridor takes place on a 9x9 grid, and repeated pathfinding is essential in order to determine how close a player is to their goal, and more broadly whether the goal is even reachable. (In order to even determine the valid moves from a given position, all moves must be checked to see if they make a goal unreachable). I plan to release this within the next few months, including at least 3 decision making "engines": mtdf (a min-max variant), MCTS (parallel with some tricks), and a hybrid involving catboost.
Nevertheless, I spend a lot of time with local models because of:
1. Pure engineering/academic curiosity. It's a blast to experiment with low-level settings/finetunes/lora's/etc. (I have a Cog Sci/ML/software eng background.)
2. I prefer not to share my data with 3rd party services, and it's also nice to not have to worry too much about accidentally pasting sensitive data into prompts (like personal health notes), or if I'm wasting $ with silly experiments, or if I'm accidentally poisoning some stateful cross-session 'memories' linked to an account.
3. It's nice to be able solve simple tasks without having to reason about any external 'side-effects' outside my machine.