They've also been making things smaller. Some would argue that it's a cultural thing, but being poor also means you have to start adopting austere cultural habits as a coping mechanism.
If I understand the correctly the initial models were done by scraping the internet off public data. They now probably pay for access, especially to companies that hold the data. Even in the latter case, the content creators probably don't see anything because they signed away their rights using whatever free service they uploaded their work / comment to. In hacker news' case I'm sure some bot is scraping my prose right now and training something with it, which I'm totally fine with because the act of trying to rent seek my 0.00001 cent of value in this post is not worth the detriment to AI advancement.
They already "stole" it. They aren't giving it back and they've established their valuations based off of that. If they start paying now, it's simply going to be impossible for any more upstarts to do this or even release open-weight models because everyone with data will become rent seekers. Imagine if they started off with rent-seekers, we'd simply not have the benefit of these models at all at this point.
Yes but all the AI companies took all the public data, so when you pay for an AI model you are paying for the marginal service of building a model off that data, not for the data itself. What we should do is ensure that the data is available to more people to train AI models... but sadly this doesn't seem to be happening. Instead AI companies that were first-movers got to train off public data, and as the companies and businesses that own this data get wise they're going to start charging people to train off the data. This will make it much more difficult for anyone to train a model in the future as it will become expensive, and the companies that did happen to already train off public data will get a bit of incumbent's advantage.
I guess it doesn't matter if they allowed OpenAI to do it or not because it seems other models were allowed to train off it too. I guess we should probably be giving kudos to GitHub and Microsoft for not trying to charge for access to this data.
What exactly did they train? Copilot is powered by claude, gemini, or ChatGPT these days.
Did they train autocomplete? I mean the code is open source so anyone can scrape it and train it too. I'm kind of glad they did train it because otherwise we'd still be stuck with Apple level AI models right now.
The whole reason we have so many models, including open weight models, that are all competitive with each other is because the data is free and anyone can be training off it. If the goal was to monetize the source code I guess the authors shouldn't make it open source.
The whole concept of intellectual property rights is a social and legal construct designed to promote innovation in an economy. If you don't care about that, then there really isn't any moral or immoral aspect to it. The immorality of it and associating it with stealing was just MPAA propaganda to try to shame people into paying for stuff.
If I found some DVD lying on the ground and watched it and I didn't pay for it, it's really up to me to decide if I want to pay the creator so they can continue to produce content. If I don't pay then obviously it doesn't help them produce more content... but the consumption of the content itself neither felt nor heard by the creators.
Yesterday I was trying to figure out if my expired nacho dip would be safe to eat and wanted to know how much botulism would be toxic if I ate it and so I asked Claude. It refused to answer that question so I could see how the current safeguards can be limiting.
It beeps at you if you stop paying attention, which is superior. Hands on wheel is an arbitrary design decision more likely to placate what a layman would think is necessary to ensure safe AI steering.
This doesn't mean that it's over for SO. It just means we'll probably trend towards more quality over quantity. Measuring SO's success by measuring number of questions asked is like measuring code quality by lines of code. Eventually SO would trend down simply by advancements of search technology helping users find existing answers rather than asking new ones. It just so happened that AI advanced made it even better (in terms of not having to need to ask redundant questions).
To be fair with the languages I use there are only a finite number of ways a particular line or even function can be implemented due to high level algebraic data structures and strict type checking. Business logic is encoded as data requirements, which is encoded into types, which is enforced by the type checker. Even a non-AI based system can technically be made to fill in the code, but AI system allows this to sort of be generalized across many languages that did not implement auto-complete.
With coding agents AI almost never manually type code anymore. It would be great to have a code editor that runs on my phone so I can do voice prompts and let the coding agents type stuff for me.
I also try to do verbose type classes using Ocaml's module system and it's been handling these patterns pretty well. My guess is there is probably good documentation / training data in there for these patterns since they are well documented. I haven't actually used coding agents with Haskell yet so it's possible that Ocaml's verbosity helps the agent.
Actually I found the coding models to work really well with these languages. And the type systems are not actually complex. Ocaml's type system is actually really simple, which is probably why the compiler can be so fast. Even back in the "beta" days of Copilot, despite being marketed as Python only, I found it worked for Ocaml syntax and worked just as well.
The coding models work really well with esoteric syntaxes so if the biggest hurdle to adoption of haskell was syntax, that's definitely less of a hurdle now.
> Instead, you're writing non-trivial proofs about all possible runs of the program.
All possible runs of a program is exactly what HM type systems type check for. This fed into the coding model automatically iterates until it finds a solution that doesn't violate any possible run of the program.
We already have verification layers: high level strictly typed languages like Haskell, Ocaml, Rescript/Melange (js ecosystem), purescript (js), elm, gleam (erlang), f# (for .net ecosystem).
These aren’t just strict type systems but the language allows for algebraic data types, nominal types, etc, which allow for encoding higher level types enforced by the language compiler.
The AI essentially becomes a glorified blank filler filling in the blanks. Basic syntax errors or type errors, while common, are automatically caught by the compiler as part of the vibe coding feedback loop.
YC Winter 2011
[ my public key: https://keybase.io/notchrischen; my proof: https://keybase.io/notchrischen/sigs/h5vpKspt8FRZESGwtNyv3ZKbfddR7J5V0NDWXJistyM ]