For a more informed opinion than folks on the internet, here's some work from Microsoft with early/internal access to gpt4: https://arxiv.org/abs/2303.12712 . I don't think people close by these systems share the same dumb parrot sentiment at all.
These projects are awesome to see, there are similar efforts for everquest. Is anyone aware of anyone trying to create different clients/renderers for these MMOs? A VR client for any of these worlds would instantly be amazing.
Just to further this, it's not just 'big names' that feel this way. Read this paper from a team at Microsoft Research: https://arxiv.org/abs/2303.12712 . These folks spent months studying properties of GPT-4, that paper is ~150 pages of examples probing the boundaries of the model's world understanding. There is obviously some emergent complexity arising from the training procedure.
This seems like (clearly?) bad journalism. RFID tags on their own are for sure not taking active sensor readings, that'd be left for something closer to TPMS style systems.
No, your OP is mistaken. The model weights have to all be accessed for the forward pass. What has happened is that using mmap changes where the memory is consumed (kernel vs process) and so it was being incorrectly interpreted. There are still 30B parameters, and you'll need that times however big your floating point representation is to use the model still.
So can we at least take a second to appreciate the orders of magnitude of growth between BBS' and tiktok? With data structures and algorithms we're very willing to accept that orders of magnitudes shifting can change our basic assumptions about things.
The point isn't that these things haven't existed in some form forever, it's that they're scaling to points where these effects are becoming increasingly dangerous, proportional to the growth of these systems.
This thread is exposing one of the dirty realities of modern SEO. Most of the time, SEO folks will measure their success based on having some way of determining their ranking for a given search query.
The problem is that in 2022, google personalizes (aka re-ranks) the SERPs on enough dimensions (device type, location, user, etc) that a single mapping of search query -> ranking fails to capture any of the nuance that google is applying to their user understanding. I'm not sure how or when the SEO space will actually reckon with this, or if they'll keep just pushing poor 'visibility' metrics that can't actually be reduced to a single dimension.
For big organizations that have the capacity and data, online advertising becomes a ROI optimization game, and one that they perform quite well at.
For a random business that wants to advertise online, without the infrastructure and data capability to back it, they will struggle to compete unless they exist in a segment full of similar peers. When the former happens, we see articles about how PPC doesn't actually work, etc.
Reality is that it takes engineering work and infrastructure, coupled with some data capabilities to unlock real value in the online advertising space.
As noted, online advertising brought all sorts of insight and visibility over traditional 'offline' marketing channels, but with that comes more savvy competitors that will do all the data things you're not.
There's nothing 1960s about it, that's just not well reasoned. These computation constructs/tools just don't currently have better abstractions while maintaining the desired computational performance.
It's a strange intersection of needs where one wants or needs something like CUDA, but doesn't care to ensure their computation is actually running optimally. If you don't want to be bothered with control and granularity, why are you trying to write high performance CUDA code in the first place?
Would you mind elaborating as to what your hobby project was?
What? CUDA is intentionally granular and low level, why do you feel that is a negative thing at this level of abstraction? Are you suggesting the tools should be better so that doesn't have to be the case? I can't figure out what you're actually trying to express here.
You're not wrong that the dataset and compute are important, and if you browse the author's previous work, you'll see there are datasets available. The reproduction of DALL-E 2 required a dataset of similar size to the one imagen was trained on (see: https://arxiv.org/abs/2111.02114).
The harder part here will be getting access to the compute required, but again, the folks involved in this project have access to lots of resources (they've already trained models of this size). We'll likely see some trained checkpoints as soon as they're done converging.
To expand a bit for the grandparent, if you check out this authors other repos you'll notice they have a thing for implementing these papers (multiple DALLE-2 implementations for instance). You should expect to see an implementation there pretty quickly I'd guess.
Why does everyone assume this guy has zero business attempting this? If you read his credentials, he should be every bit as qualified as you to attempt this kind of work while understanding the pitfalls.
According to his CV he's been active in the field for quite some time. The default assumption that he's an idiot and going to kill people just seems too cynical here.
Grandparent - you specifically mention having noted methodology problems, would you mind sharing where in the methodology you think he's gone wrong?