This difference between medical board examinations and real world practice is something that mirrors my real-world experience too, having finished med school and started residency a year ago.
I’ve heard others say before that real clinical education starts after medical school and once residency starts.
I wrote about some similar observations in the clinical domain -- I call it the "human -> AI reasoning shunt" [0]. Explicitly requesting an AI tool to perform reasoning is one thing, but a concern I have is that, with the increasing prevalence of these AI tools, even tasks that theoretically are not reasoning-based (ie helping write clinical notes or answer simple questions) can surreptitiously offload some degree of reasoning away from humans by allowing these systems to determine what bits of information are important or not.
This doesn’t necessarily apply to this particular offering, but having working in clinical AI previously from a CS POV and currently from as a resident physician, something I’m a little wary of is the “shunting” of reasoning away from physicians to these tools (implicitly). One can argue that it’s not always a bad thing, but I think the danger can lie in this happening surreptitiously by these tools deciding what’s important and what’s not.
I wrote a little bit more of my thoughts here, in case it’s of interest to anyone: [0]
On that same vein, I recently made a tool I wrote for myself public [1] - it’s a “copilot” for writing medical notes that’s heavily focused on letting the clinician do the clinical reasoning, with the tool exclusively augmenting the flow rather than attempting to replace even a little bit of it.
Does anyone know how this “user decides how much compute” is implemented architecturally? I assume it’s the same underlying model, so what factor pushes the model to <think> for longer or shorter? Just a prompt-time modification or something else?
Does anyone know, how "reasoning effort" is implemented technically - does this involve differences in the pre-training, RL, or prompting phases (or all)?
“A programming language is for thinking of programs, not for expressing programs you’ve already thought of. It should be a pencil, not a pen.” - from PG’s “Hackers & Painters”
Thanks for trying it out! Honestly, no current idea of business model - this was just something I thought would be helpful in my daily workflow so I built it out.
The buttons are stored to localstorage, as is the editor text - everything stays locally (besides what is sent to the LLM). I'm planning on a simple import/export mechanism for transferring buttons across different browsers/computers!
I'm still trying this out, but I've been using Superlist [0] (no affiliation) for different projects. For me, having a single app that is both a Markdown document <-> todo app has been helpful. I'm able to jot down notes in greater structure and depth than in another todo app and interweave todo items throughout, which helps contextualize tasks a lot better (especially when working across different projects).
Reminds me of org-mode in Emacs - overall, I'm really liking the cross-platform sync and Markdown interface out-of-the-box.
>> “you've tried response_format: "json" and function calling and been disappointed by the results”
Can anyone share any examples of disappointments or issues with these techniques? Overall I’ve been pretty happy with JSON mode via OpenAI API so I’m curious to hear about any drawbacks with it.
I see a couple comments comparing llama.cpp and Ollama, and I think both have utility for different purposes. Having used both llama.cpp (which is fantastic) and Ollama, a couple things that I find valuable about Ollama out-of-the-box --
- Automatically loading/unloading models from memory - just running the Ollama server is a relatively small footprint; every time a particular model is called it is loaded into memory, and then unloaded after 5 mins of no further usage. It makes it very convenient to spin up different models for different use-cases without having to worry about memory management or manually shutting down those tools when not in use.
- OpenAI API compatibility - I run Ollama on a headless machine that has better hardware and connect via SSH port forwarding from my laptop, and with a 1 line change I can reroute any scripts on my laptop from GPT to Llama-3 (or anything else).
Overall, at least for tinkering with multiple local models and building small, personal tools, I've found the utility:maintenance ratio of Ollama to be very positive -- thanks to the team for building something so valuable! :)
I commented on this on HN a couple months ago, but I had a similar conclusion regarding the value of memorization when I joined med school after studying computer science in undergrad and grad.
It took me a while to buy in to high-volume memorization as a learning technique (especially coming from CS, where memorizing facts is not a huge emphasis). After a while though, I started recognizing how the quick recall encouraged by the system enhanced my understanding of concepts vs replacing it (I wrote about this a couple years ago [0]).
I used Anki quite a bit in med school, where it’s extremely popular. Admittedly, it took me a couple years to fully buy in to it as a tool for cramming hundreds of facts, especially coming from an academic background (computer science) where memorizing facts at a high rate was not a huge emphasis at all.
After using it for a while though, I began to value how the quick recall encouraged by the system actually seemed to /enhance/ my deeper understanding of concepts, rather than replace it (I wrote a short post about this a couple years ago [0]).
“Developed by Tencent's ARC Lab, LLaMA-Pro is an 8.3 billion parameter model. It's an expansion of LLaMA2-7B, further trained on code and math corpora totaling 80 billion tokens.”
I just finished up building a working version of a personal-use tool a few days ago which I'm planning on releasing publicly soon in case it's useful to anyone else (however, the primary motivation was my own use-case) -
I often hear something in a podcast that I want to write down, but don't want to peck out notes on my phone, so I built a GPT Vision-powered tool to help.
Workflow:
- hear something interesting → just screenshot lock screen
- use screenshot to auto-identify podcast, episode, and timestamp
- summarized bullet points at that timestamp via GPT, logged to a virtual notebook
I built a local developer-mode plugin that gives ChatGPT access to a local directory [0] -- it's pretty basic in terms of functionality (just simple bridge to access/read files) but I've found a surprising amount of value from it [1]!
I wrote a simple implementation to do this in ChatGPT via local plugin [0]. Obviously it doesn’t hit the “fully private” requirement but I imagine it would be relatively straightforward to integrate into a local LLM. The question is whether a local LLM would be as good at grabbing enough context and nuance from the project to answer meaningfully as GPT-4 is able to do with plugins.
Per this previous post on HN [0][1], at least some of the restrictive behavior in the default chat model is attributed to the default system prompt. It would be interesting to see how the default model performed if the system prompt was changed to encourage providing answers rather than deflecting.