Slightly off-topic but related: currently I'm in a research environment (biomedicine) where a lot of AI is used. Sometimes well, often poorly. So as an exercise I drafted some rules and commitments about AI and research ("Research After AI: Principles for Accelerated Exploration" [1]), I took the Agile manifesto as a starting point. Anyways, this might be interesting as a perspective on the problem space as I see it.
It always fascinated me how a lot of "state" in our neural networks is probably encoded in the extracellular fluids rather than in the neurons themselves. Biological neural networks really do not like it if you swap out their fluids (you can do this experimentally in vitro). For me it has always been one of those "nature really doesn't care about complexity" things ... it will just use whatever tools are available with little to no regard for our understanding of "boundaries". I'm not saying this complexity is strictly needed for intelligence (however that is defined) ... but I do always find it a little bit dishonest when scientists (and especially those working with pharmaceuticals) "zoom in" to the cells and then usually promptly forget that those cells are part of the much wider stage of tissues and eventually even everything outside the confines of our skin. Anyway, just my two cents.
It's a fascinating topic. I've often been surprised by its use of "tapestry" (as in "the tapestry of existence") and I never really figured out why it seems to gravitate towards such a relatively uncommon phrase. As usual I am reading too much into it (seemingly the logit for "tapestry" was higher than the others), but why? And what do these preferences tell us about the LLMs more generally? Anyway, fascinating!
RDF and semantic web used to be my goto's for this, as I believe many of the core ideas are still valid and often overlooked and sometimes even poorly re-implemented. Which says something.
However, lately I've come to like llama.cpp and friends, yes it's not ChatGTP miracle whatever but how often do you /actually/ need that? Despite its tremendous popularity, it still seems like something more people should know about. For me, I've had great fun with running LLMs locally and experiencing their different "flavors" from a more "phenomenological" (what is it like to use them) perspective rather than a technological one.
Esther is my personal project to develop a diary app that talks back. Basically. Along the way I learned how to htmx, llama.cpp via JNA and practical experience with LLMs. I suffer from type-1 bipolar disorder with psychoses and normal life just ain't gonna happen. So unemployed, single, #nolife ... I decided to build a chat bot for myself. It's not finished the RAG stuff still needs implementing, but oh well. It's written in Clojure on GraalVM (because I wanted to try polyglot) other than that it's nothing /too/ fancy ... but it feels great having an LLM + UI you can just build the way you like it
Probably soon! I'll post it here. Still finalizing some Retrieval Augmented Generation things. It's written in Clojure with a very thin HTMX front-end. However there are some interesting things like using gbnf grammar constraints creatively for chain-of-thought reasoning. It's a one-person job though but I've always wanted a diary that feels like someone to talk to, and the tech is finally here!
I typically write them myself in the form of a "you are-such-and-so, your role is this-and-that. As such-and-so you have the following traits..." and so on. Sometimes I let some other AI rewrite it. There's very little method or science to it for me: if it feel right, it's right. Typically I find the first few chat-lines of the prompt (i.e. the chat history in the context) to be much more decisive to the conversation flow than the actual prompt itself. But it's all just "prompt" of course. My biggest realization in making the things go was "it's just a wall of text, the chat bits are just a thin facade". Write the prompt the way you want the text to continue, basically. It's a fancy Eliza. The folks over at https://www.reddit.com/r/LocalLLaMA/ sometimes share their (sometimes NSFW) prompts as well though. Right now I'm working on a minimalist interactive journaling app (a diary that talks back), and it's been a lot of fun to do and learn
I'll give those a shot as well, thanks! It's a tricky balance sometimes between "I should actually finish building the thing I am trying to build" and "ooooh shiny new model to try for a bit...", however.
Being one of those people that still recommends Emacs and have sunk countless hours in my personal dotfiles: I kinda enjoy doing it, that's why I am doing it. It feels great having a configuration so tailored to me personally that it almost feels like clothing or other lifestyle choices. I mean sure, clothes, makeup, cars and whatever are all also huge wastes of time and resources in our short and mostly dreadful lives, but at least Emacs still brings me joy :-)
It works really really well for chatbots and roleplay applications (at least for me). The fine-tune on the instruct version is rather meh however, and I recommend https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca/ if you plan on using it out-of-the-box. Take note of the prompt template, you'll get really undesired results otherwise (basically just garbage). I've been running it on my pet projects with llama.cpp and the inference is blazing fast even with my mediocre 2080 Super
There was a similar situation in a project I worked on, at the time FAISS/Annoy/SPTAG were considered but we were really looking for something more managed, think ElasticSearch rather than Lucene. As luck would have it we stumbled upon https://www.milvus.io/ and it certainly checks a lot of boxes. But I'm rather excited to see some competition in this space, it's a broadly applicable trick so I'll definitely take a look at Vald next time it comes up as well!
A bit of a technical tangent, but I wonder if a peer to peer version of this website would increase its chances of survival. I can easily imagine something like a reference manager that puts copies of the PDFs you've read or written on IPFS. It would still require some central node that coordinates the sharing, but if some open source ActivityPub service would fill that role it could work I think. Provided there is of course enough personal incentive to share and keep sharing these works.
Interesting! I can echo these findings from from personal experience. For the past 5-6 years I had been doing remote work for a US west-coast based company from the Netherlands. My annual salary as a senior "data scientist" was at least double, if not triple, that of my fellow Dutch peers. Typically salaries around here seem to stall at the €45-60k/yr mark with very few positions open at higher tiers (and usually not advertised through any typical recruitment portals, where salaries are typically even lower, even when commanding decades of experience in obscure technologies). When I left that remote position the options really did seem limited, and I ended up taking a substantial pay cut. Short of trying to bootstrap your own company (VC-like funding here is also not very typical, with angel investors being almost non-existent) moving seemed the only option, which I had no real intention of doing. Even moving slightly east to Germany seemed to be the better option. I think our current predicament has really opened the market though. Remote work is no longer considered weird and there definitely are very talented and skilled engineers up for grabs here in the Netherlands if they are coached ever so slightly. I wonder if there is an opportunity here for an employment-agency-ish company facilitating the overseas hiring process and legalese. Exciting times!
My hope is that the same realization will eventually come to web developers as well. A well written desktop application is much faster, and typically more stable, than a web app that's hosted ... well who can tell where. This isn't to say that web applications don't offer tremendous value, but it is definitely a trade-off worth considering sometimes.
You could put an eInk display behind it, and have sort of a dual mode of interaction. If both can be constructed flexibly/foldable (with or without a hinge) I can see some very interesting designs. The ability of eInk to switch to black, or just display semi-permanent information with amazing readability and low power usage, while still having portions of the screen be high refresh rate and full color would be amazing.
[1] https://gist.github.com/joelkuiper/d52cc0e5ff06d12c85e492e42...