Fascinating! I spent the past two days at a software conference in the Barbican. (The SDD conf). The place is truly beautiful in a brutalistic way. Had lunch in the greenhouse.
Agree -- maintainable code was always important, but even more so in the age of AI. Unless we introduce proper guardrails via reliable code quality metrics, all these AI assistants will serve more as legacy code generators than genuine help.
That paper benchmarked the performance of the most popular LLMs on refactoring tasks on real-world code. The study found that the AI only delivered functionally correct refactorings in 37% of the cases.
AI-assisted coding is genuinely useful, but we (of course) need to keep skilled humans in the loop and set realistic expectations beyond any marketing hype.
There are two books which I keep coming back to learn from:
* Structure and Interpretation of Computer Programs: SICP changed the way I view programming. It's an excellent intro to the mindset of solving problems with code and how to iteratively build up a software design. You need to be comfortable with Scheme, though, in order to get the most out of the book.
* The Mythical Man-Month: Not strictly about programming but more on how to succeed with software projects. Hence, The Mythical Man-Month is more about people than code, which is why it has aged so well: people is a fairly stable construct.
I'm not John, but I'm one of the authors of the research paper. Indeed, one goal with this work is to complement DORA’s delivery metrics with similar correlations between code quality and its business impact.
Hopefully, this work can help developers when communicating with the product/business side. The sad reality is that without quantifiable metrics, short term targets will win over the long-term maintainability of the codebase. Only a minority of companies actively manage technical debt.
This is indeed something that the academic research paper (see https://arxiv.org/abs/2203.04374) identifies as Future Work. There could be a code familiarity component that explains the large risks and variations.
From the paper:
"Could it be that changes done by the main author are faster and more predictable than corresponding changes made by a minor contributor? We suspect that author experience could be a factor that impacts the predictability of changes to low quality code."
If that turns out to be the case, then it would highlight organizational dimensions of low quality code: key personnel dependencies and on-boarding challenges.
Yes, cyclomatic complexity might be a popular metric but it was never a good one. At least not for its original purpose.
That said, I occasionally find cyclomatic complexity useful: cyclomatic complexity is a pretty good estimate on the number of unit tests a piece of code needs.
I started to code 35 years ago on a Commodore 64. The beauty of those machines was that you literally had to write code to even load a game (LOAD, RUN, etc.).
As such, the barriers to start exploring programming on your own were low: the development environment was already there, you were familiar with the interface, and it booted in an instant. I haven't seen any modern day technology replicate that ease of access for beginning coders.
That's a great visual presentation, but not really an innovation. The CodeScene tool has that built in together with a set of deep analyses on top (see https://codescene.com/).
CodeScene isn't built on the open source tool. CodeScene's engine is implemented from scratch in order to deal with large-scale codebases. The basic metrics are the same, but CodeScene adds plenty of information and pattern detection on top of them. If you're interested, the story of CodeScene is written down here: https://empear.com/blog/happy-birthday-3-years/
Cool - thanks! While the measures a simple in theory, there are some practical challenges; git repositories tend to be messy. So part of the practical challenge is to clean the input data (e.g. filter out auto-generated content, checked in third party libraries).
Another challenge is that version-control data is quite file centric, while many actionable insights are on a higher architectural level. In CodeScene we solve this by aggregating files into logical components that can then be presented and scored.
A large codebase under active development presents a moving target; Even if you knew how something worked last week, that code might have changed twice since then. Detailed knowledge in the solution domain gets outdated fast.
To address this issues, I work with something I call a behavioral code analysis. In a behavioral code analysis, you prioritize the code based on its relative importance and the likelihood that you will have to work with it and, hence, needs to understand that part. Behavioral code analysis is based on data from how the organization works with the code, and I use version-control data (e.g. Git) as the primary data source. More specifically, I look to identify hotspots. A hotspot is complicated code that the organization has to work with often. So it's a combination of static properties of the code (complexity, dependencies, abstraction levels, etc) and -- more important -- a temporal dimension like change frequency (how often do you need to modify the code?) and evolutionary trends.
I have found that identifying and visualizing hotspots speeds up my on-boarding time significantly as I can focus my learning on the parts of the code that are likely to be central to the solution. In addition, a hotspot visualization provides a mental map that makes it easier to mentally fit the codebase into our head.
I also have a book, Software Design X-Rays: Fix Technical Debt with Behavioral Code Analysis, that goes into more details and use cases that you might find useful for working with large codebases: https://pragprog.com/book/atevol/software-design-x-rays