My "When I met Vint" story is less exciting then some here. He attended the "Disability Support Group" at Google regularly. He made a point of just being there to listen and support others.
We have long passed at least my owner personal "threshold of evidence" to assume that Mars had life.
That bar is pretty low. I am not hard to convince.
We don't have a baseline by which to contextualize the situation statistically. There is no p-value we can meaningfully satisfy. Worse than that, evidence required for a belief derives from the risk of being wrong.
Is there a discussion of what evidence WOULD be enough for the scientific community to say loudly "There was life on mars". Because right now it seems all risk and no utility, so the evidence bar would be really high.
This is all built on a premise that kinda scares me. Its not a new idea, it's one I have participated in my entire career. That concentration of power at the top labeled "why". It doesn't actually make sense, it just gives the people with power a narrative of security. "Why" should be a conversation up and down the org chart. If "what" doesn't come with why, you make the wrong "what". If "how" doesn't come with "why" you sabotage yourself. If your "why" can't survive effective communication and context, it was probably a bad call. If you feel like you "need to go fast" you are just gambling and looking at the world with survivorship bias.
Every engineering decision and debate I have seen has always boiled down to 2 things "people don't know the motivation for an action" and "people have incomplete knowledge of the situation". Normally both. Every single conversation, when poised in a "this is the high level goal" context, went from a debate to a constructive design session in seconds.
Failure to "propagate the why" is a tar-pit for execution and decision making.
The basic concept, "put a new compound in everybody's bodies" because it makes them better suited for thier environment is entirely reasonable.
Humans didn't evolve in the context of plentiful food. It's one of the ways our success took us to a place our bodies are maladapted to. Genetic pressures will solve the problem eventually. Memetic measures have seen some success.
Like "flouride in the water", it is going to require a lot of work to justify that "exposing everyone is univerally good" but it is a possible reasonable and safe path eventually.
I think you are taking an entire space of "intellectual immune system" out of consideration. More than that, you are ignoring the core reason I think they are bottle-necked. They might have more access to compute. But to compute what? The bottleneck of intelligent behavior isn't compute it is experience. We have a lot of "text encoded experience" to feed it, through our collective corpus of writing, but ultimately potential behaviors can only be tested by active experimentation. No amount of observation can discern correlation from causality. Only active experimentation. The "train on itself" only works in a "toy universe" where the model of consequences are trivial to "test"
So in order to scale, AI doesn't need compute. It needs "engagement with reality and agency". Which is STILL might do better than us, but is happening in the real world, with real competition over resources. As long as we don't do something dumb like enthusiastically give it control over our major economic actors. I don't think we need to worry.
On the "intellectual immune system" side, I would argue that language's limitations are themselves fitness. We are already in danger of memetic hijacking. All those points you make about multiple instances of an AI cooperating, don't take malice and memetic attacks into account. It goes back to "why I am not afraid of grey goo". I trust yeast to find a way to metabolize basically everything. We have memetic attacks too.
As a distributed systems engineer, we are a LONG way from "magical scalable ai".
The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.
We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.
I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.
Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.
AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.
AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me.
We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.
We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.
What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
This is not how I wanted intellectual property to die. A special "IP Laundering" loophole for big tech companies is WORSE than the intellectual property issues we had before.
Reputation systems have been a theoretical idea for a while, but we haven't come up with anything sybil-proof without centralized identity management. "we have a menu" sounds a lot like "we don't actually have any viable plan" in this case.
Don't get me wrong, this is awesome. I think it is built on a subtlety bad premise. I think it is time to start build organizational nomic games on this sort of contract system, literal organization governance, for systems like this to thrive.
Gods I love and loath Tiddlywiki. It has some of the most convoluted javascript written before javascript ever actually got all the features that made javascript convoluted. But it did the job!
I felt like this paper nailed it years a go, and nobody has followed up properly.
The metric involved is basically impossible to compute fully, but easy to approximate. Any online approximation will model everything it can see have changes until it is satisfied.
I feel like nobody actually understands the real reason security through obscurity is so bad. It results in dead cryptographers. The implementor becomes the weakest link in the chain, and entities with a tolerance for violence can fix that problem.
Every time Idiocracy comes up, I feel obligated to point out that it is WILDLY optimistic. The people are dumb, not evil. They struggle to adapt and learn, but are willing to try and willing to accept new information with evidence.
"Finish the sequence" with 4 options and "no pattern" as the choices.
It becomes "what does the moderately intelligent person who wrote the test thinks counts as a pattern" not the intended exercise at all. There was never enough samples to even guess at a real pattern in them.
Nobody does "face to face" key exchange like I imagine. Just two phones facing each other spamming QR codes for the other to read.
What I REALLY want is an app that builds a big bank of nonces between you and your peers over short range radio or QR codes and then lets you use a 1-time pad.
Ultimately, I'm only offering criticism because I have spent a lot of time working on exactly this problem, but I am not in a position to actually implement it. This is awesome and you should be proud of it.
distributed hash tables, p2p systems, supply chain and network security.