Speaking of AI generated pages, I wonder how OpenAI filter these low quality web pages out of their training set as they continue to training.
Also, I wonder how they decide what code is worth training on. Because a lot of code is written in poor style/has technical debt, it might be the case that these LLMs in the long run lead to an increase in the technical debt in our society. Plus, eventually, and this might already be happening, the LLM are going to end up training on their own outputs, so that could lead to self immolation by the model. I am not certain RLHF completely resolves this issue.
My dad moved to the US in 99 (he grew up in a town of about 50k people at the time). He said he was shocked people would stop for him when crossing the street, and he was amazed he could by a 1/2 gallon of ice cream for a couple bucks. Previously, he might have had ice cream once a year. He was also impressed and made happy by the abundance of meat and milk.
Funnily he was quite disappointed at the fruit situation coming from a tropical country, but that was more about freshness than quantity.
I think the idea of "fully diluted market cap" is essentially bs. It is answering the question what if the supply went up but the demand went up to the same degree as to keep the price stable, what would the market cap be then?
One of the problems is price point. They are charging $10/month. While increasing the price will certainly decrease the number of users, a much higher price point could select for "power users" that will churn less and pay substantially more. In the long run that could lead to greater revenue.
Of course this depends on how the supply demand curve looks like for their specific business.
This is also about job security. Technical people often have a record of what they have done. Think commits and PRs. Managers don't have this. Their job doesn't produce anything tangible, this is not to say that it is useless. There are good managers and bad managers, with the majority falling in the latter category.
Being seen in meeting rooms filled with people, talking to them, and pointing to things on a powerpoint presentation can give other more senior managers (who determine their position and compensation) the aura that they are hard at work and making things happen.
This reminds me of a podcast Ben Orenstein was on where he talks about something similar. Made a big difference in how I build/think about side projects.
It makes the job of the person referring you so much easier. If we agree that getting a referral is the easiest way to get a job, which I think it is, then making the job of the person referring you is the most important thing.
If they can talk about you for a couple of minutes and then email their boss a link to a well written blog post, you derisk the act of referring you. You allow the boss to sell himself by providing the material, so your friend isn't on the hook as badly.
The point that the author makes is very valuable, it is important to not throw out hands in the air. If you are not moving forward, you are falling back.
Though one (perhaps nit-picky) point I'd like to make is that these dictators are not dumb. They are incredibly intelligent. They themselves are probably not hackers, but they understand people and power. They are going do what they can to get what they want. We can't ignore the factor they play in creating these problems, and we need to take it just as seriously as we would a technical security exploit.
I think it would be interesting to read case studies of different teams, their principles, how those decisions have impacted them, benefits, drawbacks from their approach, etc.
I think principles are great, but like you yourself have said, they require context. I would especially be a valuable resource for junior and mid career engineers, of which I am one.
We should investigate the benefits of sodium as well. Sodium is vitally important for many bodily function and is a very misunderstood nutrient.
Historically people consumed much more salt, and don't give me the "people used to die when they were 35" nonsense. There is a big difference between being kicked by a horse vs dying of a stroke.
I've actually used this concept at an old job. When I was given a new project the business people always wanted it done at a particular date, but it was always an unrealistic timeframe. I'd then spend sometime thinking about how long I thought it would take me, but I would always add 2 weeks or 25% to the estimated time, which ever was larger, just to deal with the human element.
This could include changing requirements, poor communication, illness, being blocked by other changes, etc.
I learned that you can get away with giving people extended deadlines as long as you hit them.
An interesting thing to think about was people used to think of getting somewhere. Now we think of things coming to us. In a way, we did achieve the "zipping around", it's just that we did it via the internet and wireless communication. Of course, it is not the same, but it is similar.
An interesting question I've been thinking about, though I haven't really gotten anywhere is: At what point does a private platform become a public utility, so that censorship is not permissible on the site?
I agree that reddit shouldn't be censoring anyone or certain topics since it feel like a public utility. But if I were to have a blog, I think I should be allowed to block someone or delete certain comments based on how I think my blog should be, but my blog isn't a public utility. Or is it?
Great post from patio11 as usual! Reminds of the notion of taking small bets that Taleb often talks about. A side project might only take 20 or 30 hours of your time over the course of a month or two, but it can give you great upside.
Very interesting. This might have some military applications as well. Could be used for recon footage in difficult environments. Such as, mountains and deserts.
Also, I wonder how they decide what code is worth training on. Because a lot of code is written in poor style/has technical debt, it might be the case that these LLMs in the long run lead to an increase in the technical debt in our society. Plus, eventually, and this might already be happening, the LLM are going to end up training on their own outputs, so that could lead to self immolation by the model. I am not certain RLHF completely resolves this issue.