> I don’t see how it could be that bad (incorrect, specifically), considering bun is probably the most widely-used production use case of zig.
That may be the case, but the bun project only needs zig to correctly compile bun. The zig project needs to be able to correctly compile all existing and possible zig programs.
I haven't reviewed things, but it's possible and even likely (at least based on my own experience with LLMs) that the validation is mostly focused on bun compilation.
Title is annoying and the article doesn't bear it out. This is not "reverse" game theory. It's just game theory and incentives: something you'd learn in any course of study of economics.
But yes, if you change incentives, you can change behavior. And if you can find a way to create and enforce incentives that push toward an outcome you want, then you get more of that outcome. This is a good lesson to remind people of: incentives matter. So often---especially in discussion of public policy---we see conflation of stated desires with incentives, and of incentives with "cash paid to someone". The former is fallacy, and the latter myopic.
> ALL OF IT is meaningless. It's a pointless discussion.
That is a nihilistic take. That it's pointless and everything about the domain (measurement of productivity) is meaningless.
"Productivity" is no more made up of a concept than "good work". The fact that many attempts to measure it are quite dumb (e.g., the idea of using LOC as a metric) does not mean the whole idea is not worth considering or discussing.
I've had a large project recently which has biased my view on unit testing from LLMs. It includes a lot of parsing and other workflows that require character-specific placement in strings for a lot of tests. Due to how tokenization works, that is a gnarly use case for LLMs. I am trying not to form too many strong opinions about LLM-driven-TDD based on it. My forays into other domains show better results for unit tests, but the weight of my experience is in this particular low point lately.
Yes, but it comes up in conversations of LLMs a lot. Thus, the rant in question. I think we are in agreement, or at least we lack disagreement, because that is the only stance I endeavored to take in the post.
Measuring productivity in software development is a hard problem, beyond the typical categorizations used in computer science. Unfortunately, I think my best answer is to go read the book I linked in the conclusion: https://link.springer.com/chapter/10.1007/978-1-4842-4221-6_...
That is an unsatisfying answer. I can point to anecdotes that suggest AI is hurting productivity or improving it, but those don't make an argument. And the extremes on either side make it very difficult to consider. How do you weigh "An LLM deleted my production database" against "I built a business on the back of AI-assisted software"?
I think we have to wait and see. And we should revisit questions of cost and value continuously, not just about LLMs, but generally in life. Most of my motivation (though not an overwhelming majority) around using LLMs right now is a mix of curiosity and wanting to avoid the fate of the steam shovel.
The post is about using LOC as a metric when making any sort of point about AI. Nowhere do I suggest someone shouldn't use it, nor that they should expect negative results if they opt to.
The full PDF is available for download. It's mostly a series of essays, so you can pick and choose and read nonlinearly. It's worth thinking about beyond nihilistic takes.
As stated in the article, I have unlimited access to multiple frontier models and I use Claude Code, among other harnesses. The rest of your list is not directly addressed in the post, because it is irrelevant to the point being made, but I do all of those things and more. You will note that in the appendix on LLM usage, some of the things I constantly have to correct in LLM-generated code are testing mistakes. And if you care to ask, yes I have context files to address these mistakes, and I iterate them to try to improve the experience.
I would honestly appreciate constructive feedback on LLM usage, because, as I stated, I am constantly having to rework code that LLMs generate for me. The value I get from LLMs is not in code generation.
> The complaints about LLM's that lack any information about the domains being worked in, the means of integration (deep in your IDE vs cut and paste into vim) and what your asking it to do (in a very literal sense) are the critical factors that remain "un aired" in these sorts of laments.
I'm not sure if this is a direct response to the article or a general point. The article includes an appendix about my use of LLMs and the domains I have used them in.
I didn't set out to teach you anything, change your behavior, or give you practical takeaways, so it's a rant (: Emotions can be expressed with citations.
I am fully on board with gen AI representing a paradigm shift in software development. I tried to be careful not to take a stance on other debates in the larger conversation. I just saw too many people talking about how much code they're generating as proof statements when discussing LLMs. I think that, specifically---i.e., using LOC generated as the basis of any meaningful argument about effectiveness or productivity---is a silly thing to do. There are plenty of other things we should discuss besides LOC.
I actually consider that the claim is not that bold, and in fact has been common in our industry for most of the short time it has been around. I included a few articles and studies with time breakdowns of developer activity that I think help to illustrate this.
If an activity (getting code into source files) used to take up <50% of the time of programmers, then removing that bottleneck cannot even double the throughput of the process. This is not taking into account non-programmer roles involved in software development. This is akin to Amdahl's law when we talk about the benefits of parallelism.
I made no argument with regard to threat to the profession, and I make none here.
I would imagine most large-scale data center construction projects will include electrical engineers to design the electrical subsystem. A rack's floor footprint is a few square feet. You can put several million dollars of hardware into that rack. A data center will have at least a few racks. It's a very reasonable investment to bring someone in to do electrical design.
I've had the experience of getting to sit beside several categories of people across my career and watch them attempt to do something which is causing issues or errors. The pattern I have seen the most is what I can only describe as speedrunning the error. People will try to do the thing they (think they) know how to do. When information or error comes available on the screen, they completely ignore it; if it is a popup, it is closed as quickly as possible and if it is shown somewhere on the screen that doesn't interrupt their flow, then it is completely ignored.
I have given instructions to repeat, but more slowly, and people will still click through errors without a chance to read. I have asked people to go step by step and pause after every step so we can look at what's going on, and they will treat "do thing and close resulting error" as a single step, pausing only after having closed the error.
The only explanation I have that I can understand is that closing errors and popups is a reflex for many people, such that they don't even register doing it. I don't know if this is true or if people would agree with it.
I've seen this with programmers at all levels of seniority. I've seen it with technically capable non-programmers. I've seen it with non-technical people just trying to use some piece of software.
The only thing that's ever been effective for me is to coach people to copy all text and take screenshots of literally everything that is happening on their screen (too many narrow screenshots that obscure useful context, so I ask for whole-screen screenshots only). Some people do well with this. Some never seem to put any effort into the communication.
That may be the case, but the bun project only needs zig to correctly compile bun. The zig project needs to be able to correctly compile all existing and possible zig programs.
I haven't reviewed things, but it's possible and even likely (at least based on my own experience with LLMs) that the validation is mostly focused on bun compilation.