Check out OpenThoughts. It has a widely used dataset, a model that beats the deepseek's smaller reasoning models, and a paper that talks in detail about the data curation methodology.
Synthetic data got a bad reputation last year, but it is now an important component for all modern LLMs! In fact, we had also trained one model for -- ironically -- detecting hallucinations, and it was also trained on synthetic data.
Say if you have some PDFs, and want to generate questions and answers to test your RAG pipeline, that's synthetic data! Distillation is mostly synthetic data and works great as well!
Our hope is that this becomes steroids rather than LSD for LLMs :)
Not sure why you are so upset about a small and neat study ("please, please stop").
If you ask it to summarize (without feeding the entire bible), it needs to know the bible. Knowledge and reasoning are not entirely disconnected.
For the specific form of hallucination, which is called grounded factuality, we have trained a pretty good model that can detect if a claim is supported by a context. This is super useful for RAG.
More info at https://bespokelabs.ai/bespoke-minicheck.
There are several types of hallucinations, and the most important one for RAG is grounded factuality.
We built a model to detect this, and it does pretty well! Given a context and a claim, it tells how well the context supports the claim. You can check out a demo at https://playground.bespokelabs.ai
I am the author of the blog and I want to thank for all the feedback here. Because I now do realize the article starts with a high premise but fails short. Will do a better job next time!
What I meant to talk about is as follows: we have engineered away all sorts of discomfort from our lives and I think that's bad. So (1) be aware of this, (2) seek some discomfort, and (3) if you run into discomfort, take it in a positive way (this I didn't convey in the article).
But yeah I don't mean to say chop of your limbs! Not sure how people are reaching that conclusion.
Here's an example I can think of. Suppose you have a bunch of text documents, and you know that some documents are similar but not identical (e.g. plagiarized and slightly modified). You want to find out which documents are similar.
You can first run the contents through some sort of embedding model (e.g. the recent OpenAI embedding model [1]), and then apply LSH on those embeddings. The documents that have the same LSH value would have had very similar embeddings, and thus very similar content.
https://www.open-thoughts.ai/