A Bitter Lesson for Memory(personal-website-3bed.onrender.com)
personal-website-3bed.onrender.com
A Bitter Lesson for Memory
https://personal-website-3bed.onrender.com/blog-viewer.html?slug=A%20Bitter%20Lesson%20for%20Memory
2 comments
I've always been amazed at how terrible most frontier LLMs are at compaction given how embarrassingly easy it is to come up with half a dozen different RL training evals which would teach models to generate useful context summaries. Heck, you could bolt it onto any existing RL eval by just forcing a compaction every three turns.
Yep. Or even better, compact after a random number of turns. The model must then learn to preserve useful context at arbitrary context lengths.
If understanding emerges from pre-training, then perhaps memory is what emerges from post-training.