If I remember well, the starting point was that memory is only useful if it changes the system behaviour. There are many ideas of automatic 'tips' or 'rules' that are injected into the prompts so that the agent acts accordingly to some retained lesson. But there are many failure modes for this.
But maybe I'll need to revise the phrasing on that page and in instructions.
I am not sure if this is sarcasm - but just in case - https://github.com/sshwarts/skillscript does not offer a unified namespace for functions and prompts (agents) like https://github.com/zby/llm-do (currently in suspended development).
There are now also the 'dynamic workflows' in Claude Code that are pretty interesting approach - they are like a compiled prompt in many aspects.
I believe there should be easy ways to move logic between prompts and code in a smooth ways. Moving from prompts to code is for getting deterministic, fast, well defined and cheap execution - the other way is when you want to quickly extend your system or when you want to relax a pre-condition (which is extending if you look into it deeply).
There are some inroads into this vision - but I haven't seen anything build directly for this (beside my own experiment).
It is interesting to compare this to LLMs - they also have the bounded context that you can see as the analogue to our working memory. It can contain enormously more bits of information than the 4 things the article says is the capacity of our working memory - but the 4 things can probably be much more complex internally - they are more like 4 pointers probably.
But at some level context engineering is very similar to what this article talks about.
The feature I am waiting for in all of these editors is integrating 'red lining' as a channel for LLM input. This is the best interface for working on a text. https://www.roughdraft.md/ does the core idea pretty well - but is not well integrated with the rest (browsing, etc).
If the action is decided by code based on metadata - then what is really the LLM task? And if you say that it is only the type of action that is decided by code - then this is maybe a mitigation - but the llm still can do a lot of harm. And also it is very limiting - using the llm to decide the action is very useful. This is different from SQL injection - where the action is determined by the code and the injection is really making a code parsing error.
It might still be the way to go - but calling it 'the real solution' is overselling it.
They do predict what injections might be effective - so it is a theory. I don't know how novel it is and it is not very deep (as you noted the general mechanism is quite obvious) - but they do it quite systematically so it is useful.
I am open to changing my mind and I am looking forward for answers to your question - but I think it is like a click bait - looks interesting - but not really useful.
I am building my self-hosting llm-wiki system (https://gist.github.com/karpathy/442a6bf555914893e9891c11519...). My approach is to start with a theory of how such systems could work. Then since llms can interpret theory - this theory becomes an executable llm-wiki system itself.
"In the 890s, having recently converted to Orthodox Christianity, Boris ensured his church would be independent from the Patriarchate of Constantinople." --- I thought Orthodox Christianity was created by the Great Schism in 1054.
meet.hn/city/pl-Warsaw
Socials:
- github.com/zby
- linkedin.com/in/zbigniewlukasiak
- x.com/zby ---