Decades ago, long enough that NDAs are long expired, I analyzed ad network traffic at Google, handling big entities with involved contracts like IAC, Mozilla, AOL, Yahoo, and so on. We looked for weird traffic that might be signs of an attack, organized crime, misconfiguration, and so on.
Lots of these entities were arbitraging ad traffic, which they were mostly explicitly allowed to do. They'd buy ads on Google search engine results pages, and after a click dump the user on their own SERP, often for a different but (nominally related) keyword. The game is, change the keyword just enough that ads on your own SERP are much higher cost per click, and make your SERP's organic results shitty enough that users are highly likely to click an ad instead of a real link.
We kept finding that several of these guys had SERP ad revenue increasing by double, triple digit percentages year over year—with almost no organic traffic, and offset by commensurate lockstep increases in Google ad spend. They mostly consistently lost money, with spend, ahem, traffic acquisition costs, a few percentage points higher than revenue. There were lawyers, finance people, and analysts trying to figure out what the fuck was going on.
With Google taking a cut at every corner, actually coming out ahead at scale was a tough game to play. They couldn't make a profit, but they sure could show sustained growth... offset by TAC.
Literal billions were spent this way. Most of the ad purchases, SERP results, and SERP ads came from Google. Almost all the rest came from Microsoft.
Check out the SEC filings from AOL and IAC in 2013-2015 if you're curious. Grep fro traffic acquisition costs.
The name was given to the project when it was supposed to be a demo for nerds, not a product. They accidentally a product, and woe! Too late, the name was stuck and wouldn't come off, even if you scraped at it with your fingernail a bit.
Scaling curves don't need to be drawn at particularly enormous parameter counts to be useful! If you can do a 300M and 1.2B run (like the authors do here), then you can do 150M, 300M, 600M, and 1.2B runs with only 50% more resources, and get a much better sense for whether effects seem to amplify or diminish as scale increases.
I'm terribly sorry, but scaling curves or GTFO. Any random pile of linear algebra works fine-ish at small scales. Very few random piles of linear algebra push the Pareto envelope at large scales.
I've got 5 & 6 year old kids. They have a a VHS player / tiny CRT monitor with a few dozen tapes, a tiny janky mp3 player with all my ripped post-y2k era albums, and lots of books and art supplies.
VHS tapes are so cheap. Every thrift store has hundreds for like half a buck each. All your friends have a box in their basement they want to get rid of.