Fair enough. I really like the tarpit analogy, wasn't familiar with it. You can keep pulling your feet out faster than the tar rises, as long as you're willing to keep spending the energy, possibly with diminishing returns over time.
I think we're basically agreeing here. Your point (if I'm reading it right) is that taste and discernment do scale, but the gains come through pretraining/parameter scaling, which is slow and expensive compared to the fast, cheap wins in math/coding from smaller models. So taste is more of a lagging indicator of scale. it improves, but it's the last thing people notice because the benchmarkable stuff races ahead. Which also means taste isn't really a moat, just late to get commoditized.
If you're properly bitter-lesson-pilled then why wouldn't better models continue to develop and improve taste and discernment when it comes to design, development, and just better thinking overall?
What if the slowdown isn't a bug but a feature? What if AI tools are forcing developers to think more carefully about their code, making them slower but potentially producing better results? AFAIK the study measured speed, not quality, maintainability, or correctness.
The developers might feel more productive because they're engaging with their code at a higher level of abstraction, even if it takes longer. This would be consistent with why they maintained positive perceptions despite the slowdown.
> This is also an uncomfortable direction. All investors have been betting on the application layer. In the next stage of AI evolution, the application layer is likely to be the first to be automated and disrupted.
i think it says, amongst other things, that there is a salient difference between competitive programming like codeforce and real-world programming. u can train a model to hillclimb elo ratings on codeforce, but that won't necessarily directly translate to working on a prod javascript codebase.
anthropic figured out something about real world coding that openai is still trying to catch up to, o3-mini-high notwithstanding.
The "second year university student" analogy is interesting, but might not fully capture what's unique about LLMs in strategic analysis. Unlike students, LLMs can simultaneously process and synthesize insights from thousands of historical conflicts, military doctrines, and real-time data points without human cognitive limitations or biases.
The paper actually makes a stronger case for using LLMs to enhance rather than replace human strategists - imagine a military commander with instant access to an aide that has deeply analyzed every military campaign in history and can spot relevant patterns. The question isn't about putting LLMs "in charge," but whether we're fully leveraging their unique capabilities for strategic innovation while maintaining human oversight.
The paper argues against using LLMs for military strategy, claiming "no textbook contains the right answers" and strategy can't be learned from text alone (the "Virtual Clausewitz" Problem). But this seems to underestimate LLMs' demonstrated ability to reason through novel situations. Rather than just pattern-matching historical examples, modern LLMs can synthesize insights across domains, identify non-obvious patterns, and generate novel strategic approaches. The real question isn't whether perfect answers exist in training data, but whether LLMs can engage in effective strategic reasoning—which increasingly appears to be the case, especially with reasoning models like o1.
How generalizable are these findings given the rapid pace of AI advancement? The paper studies a snapshot in time with current AI capabilities, but the relationship between human expertise and AI could look very different with more advanced models. I would love to have seen the paper:
- Examine how the human-AI relationship evolved as the AI system improved during the study period
- Theorize more explicitly about which aspects of human judgment might be more vs less persistent
- Consider how their findings might change with more capable AI systems
I imagine it's difficult to be a good teacher and find effective ways to encourage students to rigorously think about things they care about in spite of the discomfort it might cause.
I also believe increasingly capable and sophisticated AI systems will play a formative role in transforming education, not as the current chatbots that are disrupting education as mentioned in the article, but as active participants in the reimagined classrooms of the future. The transition will probably be rough, but it has the potential to bring about a better future and more fruitful learning and writing.
With more capable models, reliable test time compute, and more sophisticated RAG (https://openai.com/index/openai-acquires-rockset/) I genuinely struggle to see meaningful use cases for traditional user interfaces.
The biggest hurdle I'm personally struggling with now as a software engineer trying to find the motivation to continue writing code and developing products isn't simply that AI is getting better and better at writing more complex code and doing more of my job for me (though it certainly doesn't help), but more importantly that it will soon do away with entire classes and categories of desktop, web, and mobile applications as the human interface evolves towards conversational, intent-driven interactions with AI.
Vast majority of all apps are just tables, forms, and JSON over the wire—I don't see that continuing to be the case for much longer.
Relevant to this discussion is the fact that if an LLM can’t come up with that, it wouldn’t be due to the inability to mix and match to form novel ideas, but something else, and that something else hasn’t been clearly articulated yet.
I'm happy to leave the conversation here. I don't necessarily disagree with what you're saying, but we appear to be making different points, or at least at different levels of description, and it's not really productive anymore.