"Any attempt to build AI generated content (deep fake) detection systems is flawed, since the outputs of such a system may be used to train an even better fake data generator. This leads to an equilibrium state of digital uncertainty: nothing in the digital realm can be deemed as real anymore - only as digital. I do not care if a digital artifact is human or AI made - I only care if it is useful to me. Useful content is on point, factual and at best surprising (teaches something new)." - https://seanpedersen.github.io/posts/digital-uncertainty/
Maybe fight fire with fire and respond as default with a non-sensical question and see if the bug reporter responds genuinely confused or happily tries to engage in a non-sense convo…
I am working on a visual search & exploration engine: https://digger.lol
The goal is to create beautiful and useful maps of interesting data, empowering the user to explore more intuitively guided by semantic similarity. No user data needs to be tracked for this to work, the data speaks for itself.
This roughly works by translating semantic (visual or textual) similarity into spatial proximity. Diggers major features are: semantic mapping, text search and image search. The text and image search works bidirectionally, allowing to search for images (e.g. product images) using text and for text (e.g. books) using images.
I agree in that a perfectly consistent dataset won't completely stop statistical language models from hallucinating but it will reduce it. I think it is established that data quality is more important than quantity. Bullshit in -> bullshit out, so a focus on data quality is good and needed IMO.
I am also saying LMs output should cite sources and give confidence scores (which reflects how much the output is in or out of the training distrtibution).
I wrote up this blog post in 30 mins, that's why it reads a little rough. I could not find explicit research on the impact of contradicting training data, only on the general need for high-quality training data.
May be it is a pipe dream to drastically improve on hallucinations by curating a self-consistent data set but I am still interested in how much it actually impacts the quality of the final model.
I described one possible way to create such a self-consistent data set in this very blog post.