I was inspired by https://slop.zackg.me/, which does the same thing for PL papers but with entire fully generated AI papers (i.e. LaTeX-rendered PDFs and all) - check it out! I thought it was fun to play, and wanted a version for ML, so made this website.
It's set up with a GitHub Actions cron job so that every day there are 5 real arXiv papers and 5 Claude-generated ones, and you try to guess which are real or fake from just the title and abstract.
Ironically, most of this was vibe coded with Claude Code. The exception is the UI, which uses Tufte CSS.
You might be aware of this, but most big tech companies (i.e. the ones with massive user counts) don't just let you roll out UI changes to everyone, because they know that this has a downstream impact on users. So they often A/B test those things, which is literally an experiment: you randomize who sees what, measure outcomes, and ship whatever wins. There are many data scientists employed in industry to set up and analyze experiments like this.
Also, it seems clear that this not harmless research. Everyone is aware of the effect social media has on our mental health (see the under-16 social media ban in Australia). Facebook definitely knows this, e.g. 2014 there was a big controversy over their News Feed “emotional contagion” study, where they altered what content people saw to measure changes in sentiment, without meaningful informed consent [1][2].
> Also I would like an example of something a social media company does that you wouldn't be able to get approval to do on animals. That claim sounds ridiculous.
This misses the main point: the issue is that for these experiments (and they are experiments) there is often no independent approval mechanism in the first place. Facebook, after receiving backlash, does have privacy/integrity/safety teams now which review these experiments, they are far from being independent third parties.
A parody of a fictional New Yorker article, in which journalist Isaac Chotiner grills the administrator from Shirley Jackson's "The Lottery" about the civic benefits of ritual stoning.
The madlib like sentences approach is actually how masked token prediction works! It was one of the pretraining tasks for BERT, but nowadays I think all (?) LLMs are trained with next token prediction instead.
It's set up with a GitHub Actions cron job so that every day there are 5 real arXiv papers and 5 Claude-generated ones, and you try to guess which are real or fake from just the title and abstract.
Ironically, most of this was vibe coded with Claude Code. The exception is the UI, which uses Tufte CSS.