If you've ever undertaken the task of documenting entire workflows, then you know that you quickly put up the white flag at the word "entire".
When you actually talk to people about what they do there are often many, many nuances, micro-events, micro-decisions and micro-actions in their work. This is why it can take days/weeks/months to completely train a new person for a job.
This level of detail is barely documented - anywhere. There is a huge amount of information buried in workflows that AI has barely had access to for training. A lot of this is more in the realm of world models, rather than LLMs.
So imagine trying to use AI to improve these workflows it knows so little about. Then imagine AI trying to reinvent them across an organization.
We find these use cases where AI provides great value - totally true - but these barely scratch the surface of what goes on.
This strikes me as a very solid methodology for improving the results of all AI coding tools. I hope Anthropic, etc take this up.
Rather than converging on optimal code (Occam's Razor for both maintainability and performance) they are just spewing code all over the scene. I've noticed that myself, of course, but this technique helps to magnify and highlight the problem areas.
It makes you wonder how much training material was/is available for code optimization relative to training material for just coding to meet functional requirements. And therefore, what's the relative weight of optimizing code baked into the LLMs.
"According to the study, AI still struggles with several crucial facets of coding: sweeping scopes involving huge codebases, the extended context lengths of millions of lines of code, higher levels of logical complexity, and long-horizon or long-term planning about the structure and design of code to maintain code quality."
uhhh, not sure even the best people or teams are very good at this either. Condemning AI for not being capable of something we're not capable of, ok...
“If it takes longer to explain to the system all the things you want to do and all the details of what you want to do, then all you have is just programming by another name.”
This is called the specification process, which hopefully is already occurring today.
There's so much self-serving bias in articles like this, as well as the comments on HN, Reddit, etc. It's good to critique AI, but that self-serving line is frequently crossed by many people.
It's a good idea to think of a converse situation, but this is a bad example. The constraint was not about technology but about budget, perceived benefits and political will.
Let's just imagine we're critiquing cars or planes in about 1910. They're so flawed. No one can say with certainty whether or how they will improve.
Side note of interest, from Wikipedia:
"Flying Machines Which Do Not Fly" is an editorial published in the New York Times on October 9, 1903. The article incorrectly predicted it would take one to ten million years for humanity to develop an operating flying machine.
This subject always reminds me of something someone said, possibly Professor Alain April, "software is the only system where maintenance tends to degrade it."
But - why is the most important information, the title, in such a light, hard to read, font? The title should stand out, not the comments count etc. See... Hackernews! :-)
I don't understand the glee so many people have over this. I love being able to use Generative AI tools. How is it different than if I asked a person to draw these pictures for me? I know someone will gleefully clobber this question with a legal answer, but God, let's move forward, hunh?
I gave a presentation many years ago to a software process group in Silicon Valley entitled, "Unsolved Problems of Software Maintenance". The problems are all still there.
Having done lots of both greenfield and maintenance work over four decades, I can unquestionably say that software maintenance is way, way more difficult.
I heard this first from Joel Spolsky, "it's easier to write a program than read it."
What this completely overlooks is the time developers spend trying to understand existing code. For greenfield development that is close to zero. For maintenance work it is over half of their time and arguably the most difficult work in software engineering. Maintenance work in complex systems is far more challenging than developing new complex systems.
Some people, not many, take software maintenance seriously and conduct studies on it - in the study at the link below they point out that comprehending existing code took on average 57% of developer's time. Actual editing consumed 5%.
And software maintenance is no different - think again.