It was fun writing our book because I SAW you do that. And I had a different approach - I would outline obsessively and hold the whole chapter in my head at once before I started writing. Holding a whole chapter and cross referencing everything with everything else was O(N^2). You're approach for writing one instance of the chapter was linear O(N) but you did it M times... so O(M*N) ... maybe about the same :P
You're absolutely right. This was a post I tossed together quickly just to see what could be done without thinking too much. In retrospect, I think this would be better implemented using Elasticsearch sparse vector fields which allow you to specify the value of every token. Maybe I'l make an update post to try again.
Practical advice! So many good products are lost by people that become fixated on unnecessary evals too early. You need to build your eval muscle AS you release product and get real feedback.
RAG is a pain to set up, so I tried something different. Instead of dealing with vector DBs and all that complexity, just let the LLM navigate well-structured docs like a human—exploring outlines and diving into sections. It’s simple, and works great for stuff like technical manuals or llms.txt.
Feels like it's a dopaminergic response to hearing a word but not knowing what it is. It's a novelty seeking thing. But once the work seems to be well understood, the novelty wears off and the novelty seeking mechanisms in humans quit responding.
If you get tinnitus, spend the first couple of months looking for a solution. Get opinions and second opinions. If that doesn't work, then wait for 6 months and then try again. If that doesn't work then wait for 9 more months then try again. If that doesn't work wait for 12 more months and then try again.
Most likely you'll habituate to the sound (as I have). What was once a living nightmare will become (miraculously) a non-issue. Occasionally, as one person said it even becomes "comfy" - or as I told my wife recently, it's just the sound that the world makes when you're alive. Really, it's not much different from other signal processing affects in other senses. If you close your eyes and concentrate you can see visual noise. It's also there when you open your eyes (floaters for example). But you don't worry about that because it's commonplace. Sound seems different because it's new - but it will become the same type of thing.
An important part of getting better is accepting that (likely...) you're not going to get better. And then, after just being with it for a while and not trying to push it away, you realize that it's not so bad - you get better. Just let life suck for a while. My experience hasn't been pleasant, but I've learned a lot about the nature of life. Fair trade. Wouldn't want to do it again, but glad to have done it.
I once presented with softwaredoug before a room of 400 people on a project that we didn't quite get finished! Yes... massive amounts of stress as we approached the deadline and realized that we were not going to hit it. Nevertheless, b/c of the stress we pushed harder than we normally would have, we learned a lot more than we otherwise would have, and _despite_ not finishing, the talk was one of the better received of my career. (We turned it into a lessons learned talk. We had plenty of great questions.)
How did we deal with the stress? I think we just sat with it. It was always there. Over time you recognize it for what it is, an illusion. That doesn't make it feel much better, but it gives you a bit more control and equanimity. And being able to push through hard situations despite the illusory feelings of dread opens the doors for doing some really interesting things.
In Nashville (and growing elsewhere) we started a community around self-organizing peer-mentorship. Checkout it out http://pennyuniversity.org
It's very rudimentary and grass roots at this point. (I mean.. we're a Slack Team and a Google Forum.) But the idea is that everyone has things they want to learn and thing that they can share. PennyUniversity serves as the community for people that want to make In-Real-Life connections to learn new things. It's self organized, mostly it's individuals asking to learn something and then setting up a coffee chat or a lunch conversation. Occasionally the conversations grow to larger groups (say 10 people). Often the conversations are one-off, but sometimes they form into longer-term mentorship or topical-groups or reading groups. We encourage face-to-face meetings, but occasionally the meetings are online and recorded - like this: https://www.youtube.com/watch?v=Czy7bgDd7Hc