This doesn't feel truthful, it sounds like this tool is a hack that unlocks something. If I understand it correctly, it's using the same FoundationModels framework that powers Apple Intelligence, but for CLI and OpenAI compatible REST endpoint. Which is fine, just the marketing goes hard a bit.
> Runs on Neural Engine
Also unsure if this runs on ANE, when I tried Apple Intelligence I saw that it ran on the GPU (Metal).
Yeah, the initial experience with no colors doesn’t look great. I can implement this when I have some free time, if you feel like doing it please feel free to open a PR. Thanks!
14B Qwen was a good choice, but it became outdated a bit and seems like the new version of 4B surpassed it in benchmarks somehow.
It's a balancing game, how slow a token generation speed can you tolerate? Would you rather get an answer quick, or wait for a few seconds (or sometimes minutes) for reasoning?
For quick answers, Gemma 3 12B is still good. GPT-OSS 20B is pretty quick when reasoning is set to low, which usually doesn't think longer than one sentence. I haven't gotten much use out of Qwen3 4B Thinking (2507) but at least it's fast while reasoning.
Ollama adding a paid cloud version made me postpone this post for a few weeks at least. I don't object them to make money, but it was hard to recommend a tool for local usage and make the first instruction to go to settings and enable airplane mode.
Luckily llama.cpp has come a long way and was at a point that I could easily recommend as the open source option instead.
I'm interested in this, my impression was that the newer chips have unified memory and high memory bandwidth. Do you do inference on the CPU or the external GPU?
According to the benchmarks, this one is improved in every one of them compared to the previous version, some better than 30B-A3B. Definitely worth a try, it’ll easily fit into memory and token generation speed will be pleasantly fast.
I'm really interested in `home-manager` honestly, and while I agree about the benefits of declarative package management, I find it easier to type commands at a terminal like `nix profile upgrade pkg` or `nix registry pin nixpkgs`, with the guarantee of rollbacks.
Since when I'm editing a text file to update a package, I have to look for the latest version separately and copy/paste it into my editor. If I somehow mess it up the file is broken, while no harm is done I still find this workflow a bit brittle.
If there are home-manager commands I missed to do this, I'd be eager to give it a try. That kind of workflow would remind me of running `npm upgrade pkg` and have it reflected on a `package.json` file.
If you don't care about political views of presenters, there's no need to bring inclusion to the table; just consider their technical talents and call it a day.
If you actually care about inclusion, you need to think about how including backwards-views that oppose inclusion itself could affect your effort. Banning only physically violent behavior is behind the times, it's obvious there are many ways of harming someone without lifting a finger. If you're supporting inclusion for inclusions's sake, including everyone may make sense. But if you really care about people who depend on you including them, it's obvious some measures must be taken to create a safe environment for the underrepresented.