IIRC training starts with the initial state and the end state, and the end state is always oriented the same way. It would be interesting to see what would happen if the end state was rotated randomly though I suspect it wouldn't work so well.
I have yes.. You need to erase a larger amount of the center, but it almost always results in a collapse wheras erasing around the center typically regrows.
There's hidden state in the model which presumably it uses to communicate position, ie there's the 3 colors but then a bunch of other channels that the model can use how it wants.
With the old model (and I suspect this one too) it's trained to generate from a single 'seed' pixel in the center of the image. If you erase the center of the image, that's when it completely collapses.
Iirc, the model is still pre-trained on modern text before being fine tuned on 1930's material, so it's possible it still has some knowledge of words that didn't exist back then. Edit: looks like they make some attempt to filter out documents from the pre-training but it's still possible some sneak in.
AI (as it currently exists) will never be "creative" in the sense that it can only imitate or interpolate between past works. At some point we'll recognize that AI generated works are boring and predictable. AI will probably never invent a new musical genre or artform since it can only reproduce or recombine works from the past. I wonder what happens when the internet is so full of AI generated slop that that the only things worth "training" on were made in the time before AI generation became a thing. Will AI generations be full of dated references to a time gone by?
People may say it increases creativity but I see it more as lowering the bar to produce things. The same could be said for a lot of inventions like photography made producing images easier and I'm sure a lot of portrait painters lost their jobs to photographers. I think the danger is that we may see a very rapid erosion of jobs in the creative space that won't be easy to transition into new fields which I feel will have a detrimental effect on our society.
I'm struggling to understand why an LLM even needs to be involved in this at all. Can't you write a script that takes the last 10 slack messages and checks the github status for any URLs and adds an emoji? It could be a script or slack bot and it would work far more reliably and cost nothing in LLM calls. IMO it seems far more efficient to have an LLM write a repeatable workflow once than calling an LLM every time.
> Valve actually tried with the first Half-Life game in a decade, and even that didn't work.
Half Life Alyx is still considered to be one of the best VR games ever made and one that is still consistently recommended to new users even years after release. IMO people buy hardware because of the exclusive content. If a standard game console came out and it only had one AAA game on it, I probably wouldn't bother buying it. But if there were 3-4 games that looked really interesting it starts to look more worth the investment. Playing VR games takes a lot of committment (time / physical space / $$$) so the payoff has to be worth it or you'll lose people. With the huge amount of money spent on R&D for new hardware I think it's a valid argument to say that maybe funding content would have been a better investment in terms of ensuring platform growth.
Also, side note but not every game requires free motion. Plenty of hits had no movement or teleport etc. A lot of these were completely new (sub-)genres that didn't exist or hit the same as they would in a traditional pancake game. Plus lots of kids seem unaffected by free movement (maybe as high as 50% of users by my rough estimate).
Another way to look at it is parallel processing vs sequential processing.. our brains can make a judgement call about a thousand subtle variables and data points that we can't exactly put our fingers on unless we really dig into it, which we usually label as 'feelings', using the parallel part of our brain. The sequential (logical) part can only consider a limited number of variables at a time. I don't think either mode of thinking is inherently worse (we need both), but in our society the feelings part has traditionally been discounted as being 'illogical' by academics.. I think AI has shown us that parallel processing is actually incredibly important to thinking.
But back to the original post, I think 'having good taste' and knowing when something feels like the right solution is one of those hard to define qualities that can make the difference between average and great products (and has far reaching effects in any business).
I haven't see any details on how OpenAI's model works, but the tokens it generates aren't directly translated into pixels - those tokens are probably fed into a diffusion process which generates the actual image.. The tokens are the latent space or conditioning for the actual image generation process.
He's not arguing that no jobs will be displaced, he's arguing that jobs will change, engineering may become more reliable, new types of software jobs may be created.
Agreed, and that is my concern as well that if people get too comfortable with it then companies will keep pushing the bounds of what is acceptable. We will need companies to be transparent about ALL the things they are using our data for.
From the wording, it sounds like they are conscious of the potential for data leakage and have taken steps to avoid it. It really depends on how they are applying AI/ML. It can be done in a private way if you are thoughtful about how you do it. For example:
Their channel recommendations:
"We use external models (not trained on Slack messages) to evaluate topic similarity, outputting numerical scores. Our global model only makes recommendations based on these numerical scores and non-Customer Data"
Meaning they use a non-slack trained model to generate embeddings for search. Then they apply a recommender system (which is mostly ML not an LLM). This sounds like it can be kept private.
Search results:
"We do this based on historical search results and previous engagements without learning from the underlying text of the search query, result, or proxy"
Again, this is probably a combination of non-slack trained embeddings with machine learning algos based on engagement. This sounds like it can be kept private and team specific.
autocomplete:
"These suggestions are local and sourced from common public message phrases in the user’s workspace."
I would be concerned about private messages being leaked via autocomplete, but if it's based on public messages specific to your team, that should be ok?
Emoji suggestions:
"using the content and sentiment of the message, the historic usage of the emoji [in your team]"
Again, it sounds like they are using models for sentiment analysis (which they probably didn't train themselves and even if they did, don't really leak any training data) and some ML or other algos to pick common emojis specific to your team.
To me these are all standard applications of NLP / ML that have been around for a long time.
To add some nuance to this conversation, what they are using this for is Channel recommendations, Search results, Autocomplete, and Emoji suggestion and the model(s) they train are specific to your workspace (not shared between workspaces). All of which seem like they could be handled fairly privately using some sort of vector (embeddings) search.
I am not defending Slack, and I can think of number of cases where training on slack messages could go very badly (ie, exposing private conversations, data leakage between workspaces, etc), but I think it helps to understand the context before reacting. Personally, I do think we need better controls over how our data is used and slack should be able to do better than "Email us to opt out".
A lot of the dialogue that OpenAI's models write is incredibly bland.. I really think we'll need less censored models trained on how to act different roles other than just 'super safe and friendly assistant'.