The comments here indicate a surprisingly, in my view, mixed reception, for such an unambiguously good talk?
The main thrust is simple:
- Technology is not good or bad, it is an amplifier of human values and our collective ambitions.
- Those of us who are terrified of where technology is going are actually terrified that humans are, on the net, bad, and technology will amplify that, bringing about dystopia.
- Believing this this is self-fulfilling. By embracing such beliefs you surrender your agency, which is what actually brings about dystopia. If you, instead, have a positive vision of humanity, then you are one of the people that drives society forwards rather than backwards.
We are currently at a moment in history where this message bears repeating.
A lot of things feel unstable, and as a result we have an opportunity to redefine the world in good ways and bad ones. Those who choose to redefine it in ways that are intractably bad will often do so from the point of view of a system observer, not a system participant. This is cope, because in reality everyone, whether they choose to accept it or not, is a system participant.
You're surprised they didn't eat the tokens to churn on lots of open problems instead of asking others to pay for those tokens? They're in the token business. If they're eating the tokens, it's in support of a marketing effort, not in support of innovation across the frontier of all the other academic disciplines. The collective frontier is way too big for them to just "solve it" without asking society to at least help them break even on such an enormous public good.
I suspect this article got upvoted partly for the headline & content, and partly because of who said it. Lots of people already have a good feel for the AI/journalism niche Simon Willison occupies, and can use that background knowledge to read into his words more productively.
I think it lands differently for different kinds of prose (e.g. informational, persuasive, etc.)
Persuasive prose, in particular, which is probably the majority of the things that get posted here, is less persuasive to the extent to which it includes obvious "AI tells". Even in the cases where AI is more articulate, (1) the emotional weight of the text feels manipulative when it is clear that the emotions were, at best, "vetted" by a human as opposed to having been produced by one, and (2) I think we have a built-in "bullshit-proximity" sensor, and have recently been trained to expect AI to be more willing to engage in bullshit than humans due to their obsequious disposition. (For the record, I think humans are also full of bullshit, so this one's more of a toss-up.)
Low-tech users don't give a damn if something has the guts of an "app" or not, they care about having a thing on the home screen they can click.
Businesses have the incentive to give their users that low friction experience (at the point of need) using already familiar rails (i.e. "install app from app store").
The makers of both iOS and Android treat the ability to "bookmark" a web URL onto your home screen as a power user feature that requires navigating through complex, technical-sounding menus. Does it have to be like that? Of course not. They just have a business interest in pushing users away from the open web and towards their walled gardens.
--
Mind you, I'm not saying, "advertising doesn't play a role in this". A clump of well aligned motivations is obviously going to be more powerful than a single isolated motivation. But let's not forget that apps built for non-technical users, which—I cannot stress this enough—IS MOST USERS, benefit greatly from lowest common denominator solutions where they never feel like they have to learn anything to get going.
We only come to understand idea quality via evolutionary fitness tests, which means we have to MAKE things in order to find out if they're any good. Everything you make is gonna do hill-climbing in a [potentially unstable] environment in search of an ecological niche. This is true of species a la Darwin, but it's also true of products & ideas.
Anything we make gets to participate in this evolutionary process, even if it sounded dumb at the outset. In fact, doing dumb things is how we get "unstuck" from outdated ways of thinking.
The way I understand doom scrolling (and all other forms of passive engagement, e.g. youtube, reddit, hacker news, etc.) is that when we dip below "baseline" mood (be it through stress or any kind of adversity), quick hits of stimuli momentarily bring us back up to that baseline.
At the same time, access to abundant entertainment establishes in us an unrealistically high baseline that we're destined to fall below whenever we're not actively "plugged in". When you fall below baseline, unrealistic though it may be, you feel lacking, like an addict, and you respond to that feeling exactly like an addict.
It's all too easy nowadays to forget that boredom is good. Adversity is good. Having the time to sit with your thoughts is good. We grow stronger through any form of perseverance, and weaker through any form of surrender.
Think of the entirety of the context (the full thread of conversation, all tool call output, etc.) as one message that's been submitted to the LLM all at once just so it can generate the next token (which is just the next word or even syllable.) Once that token is generated it is appended to the context, and the entire context is once again used to generate the next token. Keep doing that until the entire response is generated. The larger the context, the more stuff the model has to pay attention to as it generates each next token.
To use an analogy: imagine your friend is the author of an unfinished book. They die with 19 chapters written, and on their death bed ask you to write the 20th chapter. Assuming you're up to the task, you can only do this well if you take the time to absorb the entirety of what's been written so far.
This is how LLMs work. Context caching is an optimization on top of this, but it has its limits.
At scale, specs can only be vital to the degree to which their conformance testing is automated. Good specs should use a formal, runnable verification language. Otherwise you'll accumulate specs that are right when they ship, wrong in subtle ways 3 months in, and wrong in glaring ways 6 months in. AI doesn't change this dynamic, it amplifies it.
There's a subgenre of dystopian sci-fi where the premise is that reality in general is destined to be eclipsed by matrix-like hyper-realities, and that people will vastly prefer those and cede reality to whoever's left.
I guess the way this could work itself out is that if you prefer a hyper-reality, your genes do not pass on, and someone else's do, and within some number of generations we bounce back in response to evolutionary pressure.
I learned a fun fact in a recent interview of David Reich (by Dwarkesh):
> Every mutation that can occur does occur. There are eight billion people in the world. There are maybe 30 new mutations every generation, so that’s 240 billion new point mutations every generation. There are only three billion DNA bases in the genome, so every mutation that can occur does occur about 100 times every generation. We’re not mutation-limited anymore.
We can talk to each other like adults, the key is understanding what the goal of any given conversation is. Truth-seeking is just one possible goal among many.
Part of becoming an adult is learning how little most people care about that particular goal, and how big the buffet of alternatives is: creating shared meaning, understanding each other's values, building trust, giving/receiving emotional support, processing grief, etc. (Think of this as an upfront taxonomic exercise, followed by lots of in situ calibration exercises.)
Even for something like "decision making", which, naively, one might assume should be grounded in facts, a lot of the "facts" wind up being fuzzy and subjective. This is baked into the social fabric.
Agreed. Change makes people uncomfortable. The nature of the change doesn't matter; the transition itself is the root of the discomfort.
When things are stagnant, we gradually optimize our lives towards a low energy state and overfit to our exact circumstances. When a change in circumstances reveals past optimizations to be wasted work, it kick-starts the four stages of grief over the loss of that low energy state.
Don't let your guard down. Tricking Opus 4.6 is not impossible, it's just still an active research frontier. Once the right incantation for any specific model is known, it'll be weaponized.
There was an excellent article on the front page recently about role confusion, which highlights just how just far models have to go on this: https://role-confusion.github.io/
Except that is not what's happening. The article clarifies something that is misleading if you interpret the headline in isolation: "high-end M6" means "the high-end variants of the M6 line", not "the entire M6 line".
My favorite of these has always been the USA PATRIOT Act, or the "Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act". And that one predates LLMs, so… sheesh.
While I'm sold on the fact that modern nuclear can be built & administered safely in the face of natural disasters, and is a net good environmentally, I'm worried about corporate cronyism's corrosive effects on safety (a la Fukushima) and future instability in the form of cornered animals (e.g. Putin, Trump) acting erratically by bombing civilian infrastructure.
http://github.com/staticshock
https://www.linkedin.com/in/backer
[ my public key: https://keybase.io/staticshock; my proof: https://keybase.io/staticshock/sigs/RmmrDvMdW0vtp1Y3DMef9vq4Q7hWg8iWHLtEo8E8xOY ]