I agree in both parts. In particular, repeatability is only one part of it, another is in person physical presence, or a weaker form of it - on-screen presence, or at the absolute least - voice.
All of the latter plays a huge role, a lot of us don't want to be horrible to a person physically in front of us, evolutionally it makes sense - if you do, you might get a fist in the face. Or no support and help in the future, as whoever is nearby will have a high chance of meeting you again (historically, before modern technology and transportation), or even belong to your tribe.
Notice, however, that all of these factors in the end correlate with pre-existing geographical proximity to the person in question. In fact, the said proximity also maximizes the likelihood that you'll see eye to eye - will not have radical, difficult to overcome differences in world view. Yes, sure, we all "should" understand people existing in very different circumstances, but let's face it, it goes a lot better when circumstances are similar, and that once again correlates with geographical proximity.
So realistically there is a huge correlation between "connecting person A and B over the internet is good" and "person A and B already live nearby and already have a high chance of running into each other physically in the real world".
In its simplest, most primitive form what you want is to give people who have already physically met once a way of continuing the connection even if circumstances drive them apart immediately after. And if possible, keep that connection to more physical-like forms - at least audio, better yet video calls, only if absolutely necessary - text.
Wait, that's called a phone. Invented a gazillion years ago. You meet first, then you exchange the numbers. And then you call and actually talk to the person.
What social networks are doing is a fundamentally problematic way of connecting people even without any recommendation systems, rage baits, and other social and addictive engineering involved, or ads - even in the purest form it's already a problem.
Execs was indeed more of a metaphorical description.
A small subset of individuals working there.
You don't need 80k people to make policy decisions, Meta is that big not because of policy makers, but because of infrastructure people and people working on all sorts of Meta's attempts to enter other markets to expand their business, markets that have nothing to do with addictive and/or exploitative social networking products.
A president of a country also can't do anything by themselves, they need a small army of supporters in positions of influence. And still the said army is a percent of the total population.
Besides, let's be real here, yes, Meta's is one of the worst in terms of overall impact, e.g. certainly worse than Google out of the big ones, but the difference is not as big as one might imagine.
Meta also powers WhatsApp, which is basically an operating system of the entire region of Southeast Asia and India, and also a large portion of South America, which together host >2.5B people. Loosing that particular part of Meta (if Meta was to fail) would be a big loss to humanity, loss that Western dwellers refuse to know or accept, but ask anyone living in those areas they'll tell you how much of the economic boom of late in those places rides on connectivity provided by WhatsApp.
Google on the other hand owns YouTube, and in my personal anecdotal experience I get a heck of a lot more misery from YouTube and especially Shorts that I can't seem to escape or avoid, than I do from TikTok or Instagram neither of which I use at all and have no issue ensuring 0 interactions with.
Yes, in the end both do both good and bad, and yet in grand total Meta is worse than Google, but A) the difference isn't as large, and B) it's actually a lot harder to tell than it seems if the net result is even negative in the end. It probably is, but it's hard to tell definitively. As in, stating that some parts of Meta's business are better die and the humanity will be better off is undeniably correct, stating that all of it just needs to seize to exist sounds more like a hyperbolical illustration of sentiment, not an objectively sound proposition necessarily.
As far as I understand, there is plenty of research there in disciplines raging from social studies through psychology to game theory and economics, as well as informal simulations, that strongly suggest that human interactions are positive to participants pretty much if and only if those interactions are repeated, which realistically only occurs if participants are circumstantially close already - same neighborhood, same job, family, friends, same school, etc.
One-off interactions are almost invariably toxic with at least one of the participants getting cheated, bullied, or otherwise harmed.
So the whole premise of connecting people unconditionally, including anonymously, automatically, and from opposite sides of the world is inherently broken and doomed to do a lot of damage.
So even Meta's self proclaimed mission is damaging to society if followed, what could possibly at that point be expected from what they actually do, given the combination of basic facts that the primary purpose of any business is to make money, Meta's specific notoriously evident disregard towards ethics, their position as an advertisement business and entertainment provider, being deep into enshitification and market saturation, and of course actual honest mistakes to boot.
Don't confuse employees with execs. It's a gigantic company with almost 80k employees.
Most cultures around the world are acutely aware that the actions and opinions of their leaders are not a reflection of behaviors and opinions of regular citizen.
Such a fantastic paper overall, it was a pleasure to read, it's very accessible, and greatly informative. If anyone is new to the idea and is seeking a definition of AGI, reading this paper is easy and is immeasurably superior to merely googling or reading the wikipedia article.
My only criticism for the article within the particular set of goals outlined above is this:
The paper seems to be under-exploring two aspects that appear to be worth exploring explicitly and in detail:
1. Ability to rapidly learn from a very limited amount of instructory data post-deployment and substantially advance in its abilities in the domain of the learning post-deployment, as opposed to possessing certain level of professional skills immediately on-deployment.
2. Ability to invent entirely new ideas, like for instance inventing an entirely new system of numbers or another symbolic or other system on its own, all to advance its current goals.
Both in part to distinguish an AGI from a large collection of glued together Narrow AIs, each purpose-built for a specific, but entire domain of fairly loosely related tasks, and in part to ensure that a high level AGI system always appears at least as intelligent as an average human teenager across the full spectrum of all possible cognitive and metacognitive interactions with the said teenager (be those interactions initiated by another human or by the cognitive projection of the environment).
Without these abilities, there could be a system - it could be argued, I believe - that would technically (or at least arguably) satisfy the definition of an ASI level of AGI as per the paper that an average human child / teenager may appear more intelligent in comparison to, exceeding the said system in plasticity and real-time limited-input adaptability of the intellect rather than off-the-shelf proficiency in trained adult human tasks: a high level AGI system might be initially trained on trillions of tokens of input data, but once deployed, it needs to be able to acquire new skills and proficiencies from mere tens-to-thousands of input examples - such that humans do.
Perhaps the framework presented by the paper intended to silently encompass these abilities and the remarks above, but surely they deserved a separate discussion, such as other aspects of the definitions and the framework proposed by the paper are indeed explicitly discussed.
Similarly, not including "autonomy" into the "six principles" (making them seven) for composing a definition of an AGI and only discussing it briefly and on a side also appears to be a questionable choice for the same reasons.
It conceptually never admits ignorance and never asks for clarifications. It always produces something, to the best of its ability. It _seems_ to be a minor technical limitation (there is plenty of traditional ML systems producing confidence %% alongside the answer from years if not decades ago, in image recognition in particular), but most likely it's actually a very hard problem, as otherwise it would be mitigated somehow by now by OpenAI, given that they clearly agree that this is a serious problem [2] (more generally formulated as reliability [1])
[1] https://www.youtube.com/watch?v=GI4Tpi48DlA&t=1342s (22:22, "Highlights of the Fireside Chat with Ilya Sutskever & Jensen Huang: AI Today & Vision of the Future", recorded March 2023, published May 16, 2023)
Bing Chat runs on GPT-4, however [1]. And Bing gets this wrong in all 3 of its modes (Creative, Balanced, and Precise) as of time of writing.
Given this experiment and similar others presented around here, it stands to reason that GPTs(**1) often identify(**2) the problem as a "wolf, goat, and cabbage" problem and then merely guess which node of the problem is the middle node (inner node of the "danger to" graph), yielding a 1/3 chance of getting it right by pure luck, resulting in diverse reports here.
(**2) That does not always yield an adequate response beyond the mere permutation of nodes, however. I've been getting the following variants for step 1. from Bing in Precise in response to marginally slightly different rewordings of the same:
- The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because the general is still there.
- The general escorts the ambassador of Costaguana through the tunnel first. This leaves the ambassador of Atlantis and the ambassador of Buranda in the bunker, but they are not alone because they have each other.
and so on.
(**1) I also tried Bard and Llama 2 with even more disastrous results full of nonsense of (**2) kind. The earlier posted response of ChatGPT-3.5 is also prime with these as well.
Re
> By the way, as soon as these systems are able to check their reasoning (i don't think it'll be a huge leap) it's enough to solve reasoning problems with probability >0.1% for example. Because you can just have it do rollouts in its head until it's correct [2]
Mistakes of type (**2) don't seem to be fitting the target of the cyclic refinement you are proposing, as far as I can understand it. These errors aren't getting the logic wrong, but completely butcher the basic relationships of actors, like what it means to be alone, or spatial relationships between the actors and their environment.
I value being able to grasp what the code does at a glance while scrolling through. Your first snippet is entering my brain like as if without even reading at all, all at once and instantly.
Your second example absolutely does not and I'm sure you know why.
An easier to read research article that's narrower in subject and seemingly more relevant to the OP: https://research.nvidia.com/sites/default/files/pubs/2012-12... ("Unifying Primary Cache, Scratch, and Register File Memories in a Throughput Processor", 2012)
There is a fairly informative survey on the subject: https://www.osti.gov/servlets/purl/1332070 (A Survey of Techniques for Architecting and
Managing GPU Register File)
I'll leave your first question to the other comment here from frogblast, as I really battled with how to answer it well, given my limited knowledge and being an elbow deep into an analogy, after all. I got a writer's block, and frogblast actually answered something :D
> how akin GPUs "stream multiprocessors"/cores are to CPUs ones at the microarchitectural level?
I'd say, if you want to get a feel for it in a manner directly relevant to recent designs, then reading through [1], [2], subsequent conversation between the two, and documents they reference should scratch that curiosity itch well enough, from the looks of it.
If you want a much more rigorous conversation, I could recommend the GPU portion of one of the lectures from CMU: [3], it's quite great IMO. It may lack a little bit in focus on contemporary design decisions that get actually shipped by tens of millions+ in products today and stray to alternatives a bit. It's the trade-off.
> Are they out-of-order?
Short answer: no.
GPUs may strive to achieve "out of order" by picking out a different warp entirely and making progress there, completely circumventing any register data dependencies and thus any need to track them, achieving a similar end objective in a drastically more area and power efficient manner than a Tomasulo's algorithm would.
I doubt anyone will be able to answer questions this fine grained, not now (if the implementation is architecturally exposed - leaks into the ISA - and Asahi Linux group figures some of it out), or possibly not ever (if it's architecturally transparent and thus entirely micro-architectural).
> Does this mean there, effectively, are no registers?
I can only point out just for context that if by any chance you're asking whether the registers are implemented as actual hardware design "registers" - individually routed and and individually accessible small strings of flip-flops or D-latches - then the history of the question is actually "it never was registers in the first place" - architectural (ISA) registers in GPUs are implemented by a chunk of addressable ported SRAM, with an address bus, data bus, and limited number of accesses at the same time and limited b/w [1].
Not really, there is no such definition that you're referring to.
Perf of a GPU can be limited by any one of the thousand little things within the micro-architectural organization of the GPU in question, any on-chip path can become the bottleneck:
1. DRAM bandwidth
2. ALU counts
3. Occupancy
4. Instruction issue port counts
5. Quality of Warp scheduling (the scheduling problem)
6. Operand delivery
7. Any given cache bandwidth
8. Register file bandwidth (SRAM port counts)
9. Head of the line blocking in one of the many queues / paths in the design whatever that path is responsible for:
- sending memory request from the instruction processing pipelines to the memory hierarchy
- or sending the reply with the data payload back,
- or doing the same but with the texture filtering block (rather than memory H),
- or the path that parses GPU commands from command buffers created by the driver,
- or the path that subsequently processes those already decoded commands and performs on-chip resource allocation, warp creation / tear down, all of which need to be able to spawn the work further down fast enough to keep the rest of the design fed;
and so on and so on and so on.
By the time a high quality design is fully finished, matured, and successful enough on the market to show up on everyone's radar outside of the hardware design space, due to the commonly occurring ratios of costs of solutions for these various problems above, it usually ends up being 1, 2, or 3, but that's experimental data + statistics + survivorship bias, there is no "definition" that that's the case.
Further, what's "commonly occurring" is changing over time as designs drift into different operational areas in the phase space of operating modes as they pick out low-hanging fruits, science and experience behind the micro-architecture grows, common workloads change in nature, and new process nodes with new properties become the norm. Doubling up of F32 ALUs in Ampere is a good example of that, that was done in a way that changed the typical ratios substantially. And now M3 threw a giant wrench into incumbent statistics of relationships between (3) and the rest of the limiters, as well as between (3) and what's actionable for a GPU program developer to do to mend it.
You can be low on DRAM bandwidth util and ALU util at the same time. How would that be if there were no other limiters?
Generally, a component X of a computer system needs to be a limiter Y% of the time where Y equals the portion of the total cost of the system X is responsible for.
The principle is the easiest to apply in a "calculus of variations" manner: if doubling the key performance metric of X results in an increase of the cost of the entire system as a whole by 5%, but how often X is the limiter drops from 10% of the time to 5% of the time, doing the doubling would bring the design quite close to proper balance wrt. X.
Things that are cheap to beef up are relatively rare a limiter in well-designed systems as a result. Things that are expensive to beef up are often the limiter. What is and isn't expensive to do depends heavily on where in the design space the current design is at and where the technology is at, all of which is changing over time.
FP32 was cheap to double up in Ampere, so they doubled it up, even though that provided only relativelyl small performance improvement. But now as a result, FP32 is very rarely a limiter (in Ampere and Ada). That doesn't automatically mean that these designs are "gimped" in DRAM bandwidth or anything of the kind. Rather, the whole perception that a good GPU design just gotta be ALU limited all the time is nothing but a mistaken perception, just like "it's either ALU limited or DRAM bandwidth limited by definition" also is just untrue. See "occupancy limited" for a prime example.
Also, your "128 cuda cores" of Skylake variety run at higher frequencies and work off of much bigger caches, so they are faster (in serial manner)...
...until they are slower, because GPU's latency hiding mechanism (with occupancy) hides load latencies very well, while CPU just stalls the pipeline on every cache miss for ungodly amounts of time...
...until they are faster again when the shader program uses a lot of registers and GPU occupancy drops to the floor and latency hiding stops hiding that well.
"128 * the-number-of-cores" of threads can make progress truly in parallel (at the same time).
24,576 threads (or however many, I didn't validate the number and it depends on the occupancy, which depends on thread resource usage, like registers => depends on the shader program code) is how many threads can be executed concurrently (as opposed to in parallel), as in, how many of them can simultaneously reside on the GPU. A subset of those at any time are actually executed in parallel, the rest are idle.
You can think of this situation as follows using an analogy with a CPU and an OS:
1. 128 * the-number-of-cores is the number of CPU cores(*1)
2. 24,576 threads is the number of threads in the system that the OS is switching between
Major differences with the GPU:
3. On a CPU context switch (getting a thread off the core, waking up a different thread, restoring the context, and proceeding) takes about 2,000 cycles. On a GPU _from the analogy_ that kind of thread switching takes ~1-10 cycles depending on the exact GPU design and various other details.
4. In CPU/OS world the context switching and scheduling on the OS side is done mostly in software, as the OS is indeed software. In GPU's case the scheduler and all the switching is implemented as fixed function hardware finely permeating the GPU design.
5. In CPU/OS world those 2,000 cycles per context switch is so much larger than a roundtrip to DRAM while executing a load instruction that happened to miss in all caches - which is about 400-800 cycles or so depending on the design - that OS never switches threads to hide latencies of loads, it's pointless. As far as performance is concerned (as opposed to maintaining the illusion of parallel execution of all programs on the computer), the thread switching is used to hide the latency of IO - non-volatile storage access, network access, user input, etc. (which takes millions of cycles or more - so it makes sense).
In the GPU world the switching is so fast, that the hardware scheduler absolutely does switch from thread to thread to hide latencies of loads (even the ones hitting in half of the caches, if that happens), in fact, hiding these latencies and thus keeping ALUs fed is the whole point of this basic design of pretty much all programmable GPUs that there ever were.
6. In real world CPU/OS, the threads that aren't running at the time reside (their local variables, etc) in the memory hierarchy, technically, some of it ends up in caches, but ultimately, the bulk of it on a loaded system is in system DRAM. On a GPU, or I suppose by now we have to say, on a traditional GPU, these resident threads (their local variables, etc) reside in on-chip SRAM that is a part of the GPU cores (not even in a chunk on a side, but close to execution units in many small chunks, one per core). While the amount of DRAM (CPU/OS) is a) huge, gigabytes, and b) easily configurable, the amount of thread state the GPU scheduler is shuffling around is measured typically in hundreds of KBs per GPU Core (so on the order of about "a few MBs" per GPU), and the equally sized SRAM storing this state is completely hardwired in the silicon design of the GPU and not configurable at all.
Hope that helps!
footnotes
(*1) a better analogy would be not "number of CPU cores", but "number-of-CPU-cores * SMT(HT) * number-of-lanes-in-AVX-registers", where number-of-lanes-in-AVX-registers is basically "AVX-register-width / 32" for FP32 processing which (the latter) yields about ~8 give or take 2x depending on the processor model. Whether to include SMT(HT) multiplier (2) in this analogy is also murky, there is an argument to be made for yes, and an argument to be made for no, and depends on the exact GPU design in question.
Okay, I went through the other video they reference ("Discover new Metal profiling tools for M3 and A17 Pro" [1]), and there is actually a whole bunch of extra very relevant (IMO) information on the subject, starting about 13:30 or so.
Judging by the output/GUI of their GPU profiler, "complex" there is more like "complex instructions", think f32 (floating point) ops that aren't additions and multiplications (and FMAs), but trigonometry, square roots, that sort of thing.
All of the latter plays a huge role, a lot of us don't want to be horrible to a person physically in front of us, evolutionally it makes sense - if you do, you might get a fist in the face. Or no support and help in the future, as whoever is nearby will have a high chance of meeting you again (historically, before modern technology and transportation), or even belong to your tribe.
Notice, however, that all of these factors in the end correlate with pre-existing geographical proximity to the person in question. In fact, the said proximity also maximizes the likelihood that you'll see eye to eye - will not have radical, difficult to overcome differences in world view. Yes, sure, we all "should" understand people existing in very different circumstances, but let's face it, it goes a lot better when circumstances are similar, and that once again correlates with geographical proximity.
So realistically there is a huge correlation between "connecting person A and B over the internet is good" and "person A and B already live nearby and already have a high chance of running into each other physically in the real world".
In its simplest, most primitive form what you want is to give people who have already physically met once a way of continuing the connection even if circumstances drive them apart immediately after. And if possible, keep that connection to more physical-like forms - at least audio, better yet video calls, only if absolutely necessary - text.
Wait, that's called a phone. Invented a gazillion years ago. You meet first, then you exchange the numbers. And then you call and actually talk to the person.
What social networks are doing is a fundamentally problematic way of connecting people even without any recommendation systems, rage baits, and other social and addictive engineering involved, or ads - even in the purest form it's already a problem.