Been wrangling a RAG pipeline for the past few weeks and I swear the model looks like it’s working, but then drops logic mid-sentence, forgets context it saw 10 seconds ago, or hallucinates citations from chunks that were actually relevant — just… semantically wrong…….
The worst part? No errors. Nothing crashes. You just sit there wondering if you’re going crazy or if “LLMs are just like that.”
Reading your list was like watching someone read my bug reports back to me, but actually organized. Especially the stuff on memory gaps and “interpretation collapse” — we’ve hit those exact issues and kept patching them with duct tape (reranking, re-chunking, embedding tweaks, all the usual).
So yeah, big thanks for putting this together. Even just having the names of these failure modes helps explain things to my team.
If anything is unclear, you can just take a screenshot and ask Dr. WFGY directly.
If you run into issues or missing cases, please let me know
this is still an MVP version, and feedback is very welcome.