What is there to talk about the KV Cache, they’re handing off to a different model, I thought that you can’t reuse KV cache between entirely different models?
I mean the choice is: 1) we pass laws that explicitly say training models like this is legal (the original quote, 2) say it’s illegal and requires licenses for the data and ability to opt out, 3) we ignore it and continue because the companies are too big to jail.
Tesla had to build the charging infrastructure (and other fragmented companies followed), they had to educate customers, they had to fight dealership requirements, they had to build up and secure the supply chain, etc. etc. The government then comes in and gives them subsidies after they’ve made it though all those filters.
The Chinese government prioritized critical minerals and made sure there was domestic mining and processing. Then they mandated that regional electricity companies install EV chargers. Then they provided consumer incentives to buy an EV (bypass license plate lotteries). Then they didn’t play favorites; so much so that when the domestic manufacturers were crap they allowed Tesla to come in and set up production. That raised the bar on suppliers and spurred actual competition from the local brands.
They build the conditions for actual competition to occur and then are letting the companies win or fail on their own merits, someone will be bictorious and they’ll be lean and mean. A true capitalist free market compared to the sweet protectionist deals the Big Three get.
Would BYD be allowed to build a car in the USA? Even a joint venture? Of course not, Washington is mulling not allowing Chinese cars to even be driven across the boarder for those silly Mexicans and Canadians who want to buy one.
Fair use requires more than you accessing the material legally.
In the US one of the factors is “ the effect of the use upon the potential market for or value of the copyrighted work”.
If anthropic Hoovers up the world’s books and trains on them, and then spits them out verbatim on command, then it will clearly impact the value of the work; nobody will buy the original, they’ll just ask Claude.
Others also argue that even if it’s not reproducing it exactly that the training runs afoul of that factor, specifically the “market for” portion. A rights holder can no longer license their book for training of LLMs if Anthropic goes ahead and just trains on it anyway.
There was, and to a large degree Tesla had to do a lot by themselves since the US government picks winners and saves losers rather. The Chinese government prioritized EV and battery productions through onshoring, industrial policy, and blanket incentives (bypassing license plate lottery for EVs), rather than favoring a specific company.
Why does the US have 1 successful EV company while China has a dozen? Because on government cares about it and the other doesn’t. And now that they’re losing a race they couldn’t be bother to compete it they complain.
Same complaining about AI, now that the two chosen champions are facing actual competition, there’s complaints that it’s unfair. we were supposed to win, it’s unfair that they’re beating us at our own crooked game.
Yeah, in this case it's the USA side that's on the losing end. Just because the US government wants small government and no intervention (expect when it comes to the donor class, or their voting base, or their own financial interests) doesn't make it a universal truth.
We're happy to prop up companies that should have failed after they get big an dominant, but having an industrial policy to invest in a field as a whole is somehow a big problem.
You see this crying and threatening to take their toys and go home on every issue as soon as someone else is in the lead. Just look at TVs and solar panels; China invested in growing that sector since they saw it was important for the future; the USA does their best to deny climate change and demonize anything not running on fossil fuel. And now that nobody wants to buy American's overpriced and uncompetitive cars, it's the fault of other companies for planning ahead. But the same politicians complaining about it are very happy to set up their own protectionist tariffs and eventually bail out the laggards, again; all while touting the "free market"
> I have serious concerns about how these models might reflect Chinese government perspectives (try asking them about Tiananmen Square).
And I have serious concerns about the American ones. Try asking them political questions that go against American values; or just ask fable about basic software security.
> distillation: why exactly is it bad? After all, what are large language models but the distillation of all of the knowledge on the open Internet, scraped by the frontier labs and distilled into the models that are themselves being distilled? Who is exactly being wronged here? ... The U.S. should pass a law that (1) makes explicit that collecting data for training models is fair use, and (2) bars terms of service that forbid distillation
Sounds great to me; live by the sword, die by the sword.
I don’t think we need to blame that sale. The same steady decline had been going since 2016. Itms clearly there in the graph, but everyone trumpeting the ChatGPT angle conveniently ignores the preceding 5+ years of continual decline.