A Field Study on Technical Debt(blog.sei.cmu.edu)
blog.sei.cmu.edu
A Field Study on Technical Debt
http://blog.sei.cmu.edu/post.cfm/field-study-technical-debt-208
8 comments
In most places I have been, the infrastructure DOES suck.
First question: "Backups?" followed by "When was the last time you did a restore?"
Second question: "What's your revision control?" For all of my slagging on git and the people who use it, I am thrilled if I hear "CVS" or "Subversion" because it means they're using version control. "git" or "mercurial" tells me I have an team that has some level of clue.
Third question: "Build system?" Good luck. Never yet seen it.
First question: "Backups?" followed by "When was the last time you did a restore?"
Second question: "What's your revision control?" For all of my slagging on git and the people who use it, I am thrilled if I hear "CVS" or "Subversion" because it means they're using version control. "git" or "mercurial" tells me I have an team that has some level of clue.
Third question: "Build system?" Good luck. Never yet seen it.
I took a development job once, where I learned on the first day:
1. No backups. Emergency plan = several older copies of the product's source spread among hard drives of various former employees' systems.
2. No source/revision control. Current version of the product's source code was literally whatever was on the lead developer's hard drive at the moment.
3. No deployment process. When a release had to happen, above mentioned developer would build an EXE out of whatever he had (if it built) and directly deploy it to customers.
4. No dependency management. What versions of vendor-provided libraries do we use? Whatever is on the guy's hard drive. Oh, and they're all binaries because we either lost or can't build the source anymore.
5. No bug tracker. The list of stuff that needed to get worked on was whatever the CEO complained about last. There was no concept of a backlog of existing bugs or technical debt.
6. No documentation. But are you surprised at this point?
7. Setting up an environment to build the code was a manual process that relied on searching through past E-mails for lost tribal knowledge, which basically amounted to: Try different include and library paths until there are no errors. Once built I was greeted with 2500+ compiler warnings... (out of 200 or so C++ files).
Oh, and this software ran in airplanes.
1. No backups. Emergency plan = several older copies of the product's source spread among hard drives of various former employees' systems.
2. No source/revision control. Current version of the product's source code was literally whatever was on the lead developer's hard drive at the moment.
3. No deployment process. When a release had to happen, above mentioned developer would build an EXE out of whatever he had (if it built) and directly deploy it to customers.
4. No dependency management. What versions of vendor-provided libraries do we use? Whatever is on the guy's hard drive. Oh, and they're all binaries because we either lost or can't build the source anymore.
5. No bug tracker. The list of stuff that needed to get worked on was whatever the CEO complained about last. There was no concept of a backlog of existing bugs or technical debt.
6. No documentation. But are you surprised at this point?
7. Setting up an environment to build the code was a manual process that relied on searching through past E-mails for lost tribal knowledge, which basically amounted to: Try different include and library paths until there are no errors. Once built I was greeted with 2500+ compiler warnings... (out of 200 or so C++ files).
Oh, and this software ran in airplanes.
>took a development job once, where I learned on the first day: No backups, No source/revision control, No deployment process, No dependency management, No bug tracker.
And that is why one should always ask about these important things before the first day, i.e. during the interviews, before you decide if you want the job or not. Fortunately I'm in a position that the above would be a near-automatic "no" from me.
edit "Near" automatic as there might be a way out if the company knows that they need to improve drastically and has the will to do it. The other response "what were you able to do to improve the situation?" is key.
And that is why one should always ask about these important things before the first day, i.e. during the interviews, before you decide if you want the job or not. Fortunately I'm in a position that the above would be a near-automatic "no" from me.
edit "Near" automatic as there might be a way out if the company knows that they need to improve drastically and has the will to do it. The other response "what were you able to do to improve the situation?" is key.
Unfortunately, often the answers you get when you ask in an interview bear very little relation to the reality you find when you start a job.
Sounds like a small company making a niche product.
What were you able to do to improve the situation?
Which issue did you tackle first (documentation of processes backwards or source control forwards)?
What were you able to do to improve the situation?
Which issue did you tackle first (documentation of processes backwards or source control forwards)?
I'd like to think I left the situation far better than I found it.
Tackled source control and issue tracking first since they were low hanging fruit. Next came basic (manual) backups. My background is not in putting together general office-wide IT solutions, so I did not feel qualified to recommend any specific package. After that, it was on to a sane build and deployment process that involved a dedicated build machine and included QA. After a long time I could start safely making code changes to clean the actual codebase up.
I think you'll see this a lot in manufacturing companies who sell physical hardware. The embedded software is a critical part of the product, yet it's not treated with much more care or seriousness than anything else on the BOM. Night and day difference between working at a company where software _is the product_ vs where software is just one of the many components that goes inside the product.
Tackled source control and issue tracking first since they were low hanging fruit. Next came basic (manual) backups. My background is not in putting together general office-wide IT solutions, so I did not feel qualified to recommend any specific package. After that, it was on to a sane build and deployment process that involved a dedicated build machine and included QA. After a long time I could start safely making code changes to clean the actual codebase up.
I think you'll see this a lot in manufacturing companies who sell physical hardware. The embedded software is a critical part of the product, yet it's not treated with much more care or seriousness than anything else on the BOM. Night and day difference between working at a company where software _is the product_ vs where software is just one of the many components that goes inside the product.
Thanks for reply. Software for manufactured systems is an area that can only grow so I hope more get involved in the way you did.
Are you a consultant? It would make sense that companies that need to hire a consultant for these types of things are the companies that aren't currently doing them.
My favorite right now is pom.xml files declaring dependencies that don't exist in the corporate Maven repository, nor in any public one. Kinda defeats the whole purpose of using Maven if I have to go download jars manually.
It can go too far the other way :)
"What's your revision control?" "Git." fine "And Mercurial." Okay "And SVN. And CVS. Some are hosted locally, some are accounts on Bitbucket and GitHub".
"Build system?" "Cmake. And Scons. Depending on which version of the software you're building. And Make. And some bash scripts."
"What's your revision control?" "Git." fine "And Mercurial." Okay "And SVN. And CVS. Some are hosted locally, some are accounts on Bitbucket and GitHub".
"Build system?" "Cmake. And Scons. Depending on which version of the software you're building. And Make. And some bash scripts."
> First question: "Backups?" followed by "When was the last time you did a restore?"
KDE almost lost hundreds of their git repositories in 2013 because they were not doing backups ( https://news.ycombinator.com/item?id=5432760 ).
KDE almost lost hundreds of their git repositories in 2013 because they were not doing backups ( https://news.ycombinator.com/item?id=5432760 ).
>"git" or "mercurial" tells me I have an team that has some level of clue.
If you're judging a team being clued in by the SCM, you're doing it wrong. There are great teams that use SVN, and know-nothing teams that use git.
The measure of clued-in should be how well does a technology match the organization's needs?
If you're judging a team being clued in by the SCM, you're doing it wrong. There are great teams that use SVN, and know-nothing teams that use git.
The measure of clued-in should be how well does a technology match the organization's needs?
A few years ago we introduced svn in our company, to replace PVCS. After 4 years, we managed to hunt most of our source code down and get it commited somewhere. Now, we have loads of contractors, some good, some bad, most cheap, and almost nobody longer than a few months. After seeing what insane things some of these guys managed to do in subversion, I simply don't dare introducing git: The history rewriting features like rebase in their hands scare the hell out of me. At least with subversion, i know that what's in there stays in there. git is great if you can trust your people, but svn is a better choice in a large political company that does not care about what happens to its code.
And dont get me started on the abuse of backups or build systems.
And dont get me started on the abuse of backups or build systems.
You need to look harder then, we tick all three and even use Gitflow. With back ups used all the time for testing live state locally and a CI 1-click (well 5 or 6 actually...) deploy to Test/Stage/Prod.
Congratulations. You're the 1%, you don't need my services. :)
I came from an awful IT environment and even my team met your criteria. I think you might be primarily exposed, due to the nature of your work, to the worst of the worst. We had/have a lot of technical debt, but the architectural ones are the hardest to fix (because they cost real capital). You wouldn't believe how hard it is to get sign off on new servers (at some companies). My team nearly made an executive decision to switch to GCP or AWS just to escape our own internal infrastructure group.
Believe me, there are 10.000 million dollar non-software companies that barely use version control, and with extremely awful IT departments. And there are companies making a killing selling software to said companies.
I'm very glad I don't and all power to you spreading the good word :)
[deleted]
where are you working or interviewing?
I think, at least in sfbay, it's safe to assume git. And probably some CI tool. This is true across my personal sample of perhaps 15 companies that I know well enough.
I think, at least in sfbay, it's safe to assume git. And probably some CI tool. This is true across my personal sample of perhaps 15 companies that I know well enough.
You would be surprised. Even in the Bay Area.
The Bay Area is better than most places, but the moment you aren't dealing with a purely software shop, the probability they don't have this stuff goes through the roof.
The PCB design tool Altium only has provision for using Subversion (which is fine, thanks), but the number of people who actually USE it is miniscule.
The Bay Area is better than most places, but the moment you aren't dealing with a purely software shop, the probability they don't have this stuff goes through the roof.
The PCB design tool Altium only has provision for using Subversion (which is fine, thanks), but the number of people who actually USE it is miniscule.
Sorry about the downvote! accidental keypress!
As a first-year programmer a project manager said to me, "I've never heard a programmer praise another's code. It's always, 'This is crap!'" So I always try to be slow to criticize. There is a lot of crap, but things might not be as simple as you think. I really believe we should take more time to read and understand before we write. Or think about G.K. Chesterton's fence:
https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence
You should resist someone who says, "I don't know why this is here, let's get rid of it!" Only get rid of something if you do know why it's there!
https://en.wikipedia.org/wiki/Wikipedia:Chesterton%27s_fence
You should resist someone who says, "I don't know why this is here, let's get rid of it!" Only get rid of something if you do know why it's there!
Oh no, I always get rid of stuff I don't know why it's there.
Because if it does not break when I run "make check", it's not covered by the testing and it shouldn't have been there in the first place anyway.
And if it had a use for someone, but nobody knows, removing it is also a very efficient way to figure out :)
This all depends on the nature of the business, the app, codebase, customers, processes of dev/QA/ops, etc. For example, this cavalier attitude might work at a social media startup but not at a bank.
[deleted]
> You should resist someone who says, "I don't know why this is here, let's get rid of it!" Only get rid of something if you do know why it's there!
I know this is orthogonal, but please try to avoid these situations by commenting code which people might wonder exactly that of, in the future. :(
I know this is orthogonal, but please try to avoid these situations by commenting code which people might wonder exactly that of, in the future. :(
Even better, write a test with a name that describes its functionality.
[deleted]
Thanks for posting - I hadn't really considered Chesterton's Fence as a technical metaphor but as soon as you posted I immediately thought of a person who has a habit of this very act. I'm adding this term to my lexicon.
The reason I bring this up is I am surprised that architectural choice is overwhelmingly the top technical debt concern. That doesn't seem right. I would expect it to be poor code due to time constraints.
1. The system grew alongside the devs' understanding of the problem space. There was never time to change decisions that were later found to be not the best (and the older such a decision is, the harder it is to fix).
2. The problem space changes over time. Either because the outside world changes, or because of scope or scale increases. This means that even good architectural decisions can become bad over time.
Isolated pieces of bad code can be fixes when/if they need to be touched for some other reason. Fixing currently-inappropriate architecture takes a bit more effort.
1. The system grew alongside the devs' understanding of the problem space. There was never time to change decisions that were later found to be not the best (and the older such a decision is, the harder it is to fix).
2. The problem space changes over time. Either because the outside world changes, or because of scope or scale increases. This means that even good architectural decisions can become bad over time.
Isolated pieces of bad code can be fixes when/if they need to be touched for some other reason. Fixing currently-inappropriate architecture takes a bit more effort.
This is very wise.
You try to make the best architectural decision that you can at the time, with the knowledge and resources you have available. Time passes and you learn new things -- maybe the problem changes, or your understanding of it improves, or your understanding of alternative implementation strategies improves. For whatever reason, you can now imagine a new architecture that would be superior, if only it were implemented to replace the old architecture. Now you have technical debt. This doesn't necessarily mean that the best decision now is to pay off the debt by reworking the architecture -- that depends upon a cost/benefit/opportunity cost analysis.
edit:
Tangentially, there's a pretty interesting presentation by Kevlin Henney titled "The Architecture of Uncertainty" [1]. My poor summary: When designing the initial architecture of a system, Kevlin suggests that the team brainstorm to identify which parts of the system have a lot of uncertainty. Each region of uncertainty then becomes a subsystem. Put interfaces between the subsystems that need to be connected. Hopefully you now have an architecture with stable interfaces, even if individual subsystems need to be completely rewritten during the course of the project.
[1] http://www.infoq.com/presentations/Architecture-Uncertainty
You try to make the best architectural decision that you can at the time, with the knowledge and resources you have available. Time passes and you learn new things -- maybe the problem changes, or your understanding of it improves, or your understanding of alternative implementation strategies improves. For whatever reason, you can now imagine a new architecture that would be superior, if only it were implemented to replace the old architecture. Now you have technical debt. This doesn't necessarily mean that the best decision now is to pay off the debt by reworking the architecture -- that depends upon a cost/benefit/opportunity cost analysis.
edit:
Tangentially, there's a pretty interesting presentation by Kevlin Henney titled "The Architecture of Uncertainty" [1]. My poor summary: When designing the initial architecture of a system, Kevlin suggests that the team brainstorm to identify which parts of the system have a lot of uncertainty. Each region of uncertainty then becomes a subsystem. Put interfaces between the subsystems that need to be connected. Hopefully you now have an architecture with stable interfaces, even if individual subsystems need to be completely rewritten during the course of the project.
[1] http://www.infoq.com/presentations/Architecture-Uncertainty
Disclaimer I haven't watched the presentation, though I do have first hand experience refactoring large systems that grew slowly over time.
Based off of your edit's description, this technique feels like a patch that inevitably fall apart. It relies on the assumption that your team can correctly identify the centers of uncertainty, and that that uncertainty model will continue to apply. Thinking about such things is an excellent idea, but it is not sufficient - uncertainty is, after all, uncertain.
I think in many cases the more important thing is to create an abstraction that allows you to perfectly represent your existing business logic in the most concise way possible. This should let you cut down on the number of edge cases outside the model, and generally simplify the system. Simplicity in specification is important, because it will allows newcomers to quickly understand the inner workings of your code, quickly correlating business logic with real code - if they can understand it, and can work within it, then they will not be tempted to hack around it (which is the root of code deterioration). I strongly believe that human friendliness and understandability should be key design goals in ANY new system, not an after thought.
So long as no one breaks the abstraction, the 99%, day to day changes should be easy. When you finally do hit a case that requires a significant abstraction change, then your concise code will make it obvious that it's outside of your abstraction model, and can evaluate options at that point.
Based off of your edit's description, this technique feels like a patch that inevitably fall apart. It relies on the assumption that your team can correctly identify the centers of uncertainty, and that that uncertainty model will continue to apply. Thinking about such things is an excellent idea, but it is not sufficient - uncertainty is, after all, uncertain.
I think in many cases the more important thing is to create an abstraction that allows you to perfectly represent your existing business logic in the most concise way possible. This should let you cut down on the number of edge cases outside the model, and generally simplify the system. Simplicity in specification is important, because it will allows newcomers to quickly understand the inner workings of your code, quickly correlating business logic with real code - if they can understand it, and can work within it, then they will not be tempted to hack around it (which is the root of code deterioration). I strongly believe that human friendliness and understandability should be key design goals in ANY new system, not an after thought.
So long as no one breaks the abstraction, the 99%, day to day changes should be easy. When you finally do hit a case that requires a significant abstraction change, then your concise code will make it obvious that it's outside of your abstraction model, and can evaluate options at that point.
I have found the #1 source of architecture/reality mismatches over time being that "business" has deliberately kept developers only partially informed on a "need to know" basis.
To be clear, there was nothing malicious about it, it's just that many stakeholders only give their short term needs as input, not the long term strategy that's discussed behind closed doors in the board room. The big problem here is that non-engineers don't get that some strategies don't simply add to the problem space, but fundamentally change it.
"Why the fuck didn't you tell us sooner?" is one of the most common phrases in software development.
To be clear, there was nothing malicious about it, it's just that many stakeholders only give their short term needs as input, not the long term strategy that's discussed behind closed doors in the board room. The big problem here is that non-engineers don't get that some strategies don't simply add to the problem space, but fundamentally change it.
"Why the fuck didn't you tell us sooner?" is one of the most common phrases in software development.
Quite a lot of the Scrum process is the valiant attempt to extract coherent requirements from an end user. You should be so lucky as for them to have a coherent plan at all, let alone a secret one. Insofar as they do have one, in my experience there's always a few bits with magical flying unicorn ponies as a requirement they'll get to in time.
The thing that is very frustrating in the "There was never time to change decisions..." bit is that there often is time.
If you are curious about what I mean, I invite you to try a little experiment. For a few weeks (2-4 should be sufficient) tell people, "If you see something wrong with the design, refactor it as soon as you see it. If you need some extra time on your story to compensate for that work, just bring it up at standup and we will modify the sprint commitment".
If your experience is like mine, you will find that the vast majority of people will not refactor the design at all. About 10% of the people will try to refactor something, and will end up trying to rewrite the entire app. They will do more damage than good and will probably give up half way through. If you are lucky, maybe 5% will actually refactor something and be successful.
Because I have tried this many times, I've interviewed people and asked them why they do what they do. For the people who don't refactor, the reason they usually give is: "There is no time to refactor". Which is really odd because they have explicitly been given time. What I have come to realize, though, is that people do not want time; they want absolution of responsibility. If you ask them to make the judgement call, they do the math in their head (unconsciously) and determine that they will be rewarded more and criticised less if they do feature work without refactoring. This forces the decision up to the PM/PGM/BM/BA/Whatever, for whom refactoring has no direct benefit. The result is that refactoring is rarely done, and if it is done it is the result of a large political process.
For the people who try to rewrite everything, training seems to be the overwhelming issue. They pull on a thread and the whole sweater comes apart. For some people in this category, though, giving them carte blanche to decide what to do means that they feel they can finally "do it right". Doing it right in this context means that they can replace all the code that they didn't personally write and therefore don't like. Since everyone on the team only writes a small portion of the code, it means that rewriting everybody else's portion is basically rewriting the app. Again, training seems to help because even if a person's goal is to replace everything, if they learn how to do it piece by piece they can be successful. Also if they do that, they will be required to have many coversations and may eventually learn how to work with others.
Finally, you may get one or two people who naturally know how to refactor well. It is useful to find out who these people are and to encourage them. Unfortunately, this often enrages the "my way or the highway" people. The people who are good at refactoring, if encouraged, will naturally dominate the design of the application. Often these people are suppressed by political means because they are so effective at driving the design. In order to enable these people you will need to make some tough decisions on the business end of things.
If you are curious about what I mean, I invite you to try a little experiment. For a few weeks (2-4 should be sufficient) tell people, "If you see something wrong with the design, refactor it as soon as you see it. If you need some extra time on your story to compensate for that work, just bring it up at standup and we will modify the sprint commitment".
If your experience is like mine, you will find that the vast majority of people will not refactor the design at all. About 10% of the people will try to refactor something, and will end up trying to rewrite the entire app. They will do more damage than good and will probably give up half way through. If you are lucky, maybe 5% will actually refactor something and be successful.
Because I have tried this many times, I've interviewed people and asked them why they do what they do. For the people who don't refactor, the reason they usually give is: "There is no time to refactor". Which is really odd because they have explicitly been given time. What I have come to realize, though, is that people do not want time; they want absolution of responsibility. If you ask them to make the judgement call, they do the math in their head (unconsciously) and determine that they will be rewarded more and criticised less if they do feature work without refactoring. This forces the decision up to the PM/PGM/BM/BA/Whatever, for whom refactoring has no direct benefit. The result is that refactoring is rarely done, and if it is done it is the result of a large political process.
For the people who try to rewrite everything, training seems to be the overwhelming issue. They pull on a thread and the whole sweater comes apart. For some people in this category, though, giving them carte blanche to decide what to do means that they feel they can finally "do it right". Doing it right in this context means that they can replace all the code that they didn't personally write and therefore don't like. Since everyone on the team only writes a small portion of the code, it means that rewriting everybody else's portion is basically rewriting the app. Again, training seems to help because even if a person's goal is to replace everything, if they learn how to do it piece by piece they can be successful. Also if they do that, they will be required to have many coversations and may eventually learn how to work with others.
Finally, you may get one or two people who naturally know how to refactor well. It is useful to find out who these people are and to encourage them. Unfortunately, this often enrages the "my way or the highway" people. The people who are good at refactoring, if encouraged, will naturally dominate the design of the application. Often these people are suppressed by political means because they are so effective at driving the design. In order to enable these people you will need to make some tough decisions on the business end of things.
For the people who don't refactor, the reason they usually give is: "There is no time to refactor".
My #1 reason to avoid refactoring code is because that code has been there, is battle tested and hasn't had a bug filed against it in months. That is not my first instinct, either. There are lots of things I see that I had written months ago that I badly want to rewrite every time I see it. I have to restrain myself because it's not just the time spent refactoring. It's writing tests (if you're into that sort of thing). It's getting QA to hammer on it some more. It's fixing all the little bugs that you thought you had fixed that you re-broke. It's all the little bugs you've never seen before because this is a new design and you're not perfect.
Most of the time, it's not worth the refactor, even if it does slow down adding future features to that particular area. There are, of course, two exceptions: 1) if you're constantly playing bug whack-a-mole on a particular section and 2) if a section of the code you're working on is constantly changing. I grasp every opportunity to hold a meeting, stand on the nearest chair, strike a dramatic pose and shout "We are rewriting the loading system... FROM SCRATCH!" My project manager then tells me to get down and lay off the coffee, but that's ok, I've had my moment.
My #1 reason to avoid refactoring code is because that code has been there, is battle tested and hasn't had a bug filed against it in months. That is not my first instinct, either. There are lots of things I see that I had written months ago that I badly want to rewrite every time I see it. I have to restrain myself because it's not just the time spent refactoring. It's writing tests (if you're into that sort of thing). It's getting QA to hammer on it some more. It's fixing all the little bugs that you thought you had fixed that you re-broke. It's all the little bugs you've never seen before because this is a new design and you're not perfect.
Most of the time, it's not worth the refactor, even if it does slow down adding future features to that particular area. There are, of course, two exceptions: 1) if you're constantly playing bug whack-a-mole on a particular section and 2) if a section of the code you're working on is constantly changing. I grasp every opportunity to hold a meeting, stand on the nearest chair, strike a dramatic pose and shout "We are rewriting the loading system... FROM SCRATCH!" My project manager then tells me to get down and lay off the coffee, but that's ok, I've had my moment.
Certainly it is always a judgement call. I was just saying to my colleague yesterday that where a developer really makes a difference on a project is by consistently being able to make the right judgement calls. One can say there is no silver bullet, and while it is impossible to make a project go faster than it can, it's very possible to make it go orders of magnitude slower ;-)
On that note, a few things I try to keep in mind: As you say, don't gratuitously change code. You may hate the design, but if it ain't broke, don't fix it. This is probably the biggest mistake that "change the world" programmers make.
Second, try not to rewrite code -- ever. Usually there is no business case (see "don't gratuitously change code"). Even if you think there is a business case, it dramatically increases risk. My rule of thumb: anything that lasts longer than 2 weeks has a very likely chance of being cancelled. If you must rewrite, it has to take much less time than that.
Finally, keep in mind that refactoring is not rewriting, nor redesigning (though it is closer to the latter than the former). Refactoring is transforming the code so that it performs exactly the same function (bugs and all!!!) with a different "shape". Ideally you will have tools to help you refactor in such a way so that you can prove that the resultant code executes in exactly the same way as the original code. With or without the tools, you should have a suite of unit/integration tests that will alert you when you have made a mistake.
Refactoring allows you to slowly migrate code from one "shape" to another over time while not breaking it. People who are skilled at refactoring can evolve efficient design even starting with really badly written code.
Why do you want to do refactoring? While, as you point out, the cost of doing a work-around from a sub-optimal design is low, the cumulative cost of these work-arounds over time can be quite substantial. A work-around introduces complexity to the code. This complexity makes everything slightly more difficult and slightly more risky. It also makes further work-arounds more likely. These work-arounds compound the problem. Because poorly designed code usually has high coupling, problems in one area of the code can manifest in other areas without warning. As the work-arounds increase, the complexity can increase exponentially. As an example of a worst case scenario, I once worked on a project where the programmers averaged 1 line of code per day (Yes, LOC is a poor measure of productivity, but no matter how you slice it, that's just incredibly bad).
My experience has been that teams which refactor effectively outperform similarly skilled teams who don't refactor by a very large margin. Although productivity metrics are impossible, the difference is quite startling. Interestingly, I have also had some experience with teams that have very high test coverage, but which don't refactor consistently (or effectively). These teams also do not seem to benefit from dramatically improved performance. My current theory is to write tests to support refactoring and don't worry about any of the other benefits.
YMMV :-)
On that note, a few things I try to keep in mind: As you say, don't gratuitously change code. You may hate the design, but if it ain't broke, don't fix it. This is probably the biggest mistake that "change the world" programmers make.
Second, try not to rewrite code -- ever. Usually there is no business case (see "don't gratuitously change code"). Even if you think there is a business case, it dramatically increases risk. My rule of thumb: anything that lasts longer than 2 weeks has a very likely chance of being cancelled. If you must rewrite, it has to take much less time than that.
Finally, keep in mind that refactoring is not rewriting, nor redesigning (though it is closer to the latter than the former). Refactoring is transforming the code so that it performs exactly the same function (bugs and all!!!) with a different "shape". Ideally you will have tools to help you refactor in such a way so that you can prove that the resultant code executes in exactly the same way as the original code. With or without the tools, you should have a suite of unit/integration tests that will alert you when you have made a mistake.
Refactoring allows you to slowly migrate code from one "shape" to another over time while not breaking it. People who are skilled at refactoring can evolve efficient design even starting with really badly written code.
Why do you want to do refactoring? While, as you point out, the cost of doing a work-around from a sub-optimal design is low, the cumulative cost of these work-arounds over time can be quite substantial. A work-around introduces complexity to the code. This complexity makes everything slightly more difficult and slightly more risky. It also makes further work-arounds more likely. These work-arounds compound the problem. Because poorly designed code usually has high coupling, problems in one area of the code can manifest in other areas without warning. As the work-arounds increase, the complexity can increase exponentially. As an example of a worst case scenario, I once worked on a project where the programmers averaged 1 line of code per day (Yes, LOC is a poor measure of productivity, but no matter how you slice it, that's just incredibly bad).
My experience has been that teams which refactor effectively outperform similarly skilled teams who don't refactor by a very large margin. Although productivity metrics are impossible, the difference is quite startling. Interestingly, I have also had some experience with teams that have very high test coverage, but which don't refactor consistently (or effectively). These teams also do not seem to benefit from dramatically improved performance. My current theory is to write tests to support refactoring and don't worry about any of the other benefits.
YMMV :-)
We're saying the same things, just in different ways. I was attempting to be humorous in my previous comment, preferring comedy to precision. :)
Interestingly, I have also had some experience with teams that have very high test coverage, but which don't refactor consistently (or effectively). These teams also do not seem to benefit from dramatically improved performance. My current theory is to write tests to support refactoring and don't worry about any of the other benefits.
That's something I had never considered. Do you have any general guidelines for testing for refactoring?
Interestingly, I have also had some experience with teams that have very high test coverage, but which don't refactor consistently (or effectively). These teams also do not seem to benefit from dramatically improved performance. My current theory is to write tests to support refactoring and don't worry about any of the other benefits.
That's something I had never considered. Do you have any general guidelines for testing for refactoring?
As a developer I've had some very good and very bad experiences... usually centered around two issues.. 1. I am not a morning person and am consistently late, I get work done, but not at 8/9am on the dot. And, 2. because I will take the time to understand what I am working on... this usually results in some refactoring and generally less code in the end. I once saved enough code in size to include lodash, and eventemitter into the client side of a project, by refactoring out a piece using those two libraries. I was of course chastised because it took longer than expected. I left when the writing on the wall was I would be fired anyway... That was over a year ago, and my understanding is the project I was working on with a 3 month delivery time still isn't done because no effort was made to resolve technical debt and they kept throwing more people/teams at it.
From the outside, it's rather hard to tell the difference between someone getting problems that are "inherently simple" and someone spending time finding simpler solutions to problems.
That's actually an interesting question. Aside from problems that are easily observable as difficult because lots of smart people have tried and failed, how can one objectively know whether it was difficult or easy?
Even within "just a web app" one can run into some gnarly issues or odd old code that renders otherwise simple tasks difficult, but since it's only one person working on it you can't really test whether or not it's actually difficult.
This might impact both the dev/manager disconnect as well as imposter syndrome.
Even within "just a web app" one can run into some gnarly issues or odd old code that renders otherwise simple tasks difficult, but since it's only one person working on it you can't really test whether or not it's actually difficult.
This might impact both the dev/manager disconnect as well as imposter syndrome.
[deleted]
> they do the math in their head (unconsciously) and determine that they will be rewarded more and criticised less if they do feature work without refactoring
Great point. They know they've been told they have time but they don't actually believe they have time.
Great point. They know they've been told they have time but they don't actually believe they have time.
Half of the time, the build system of companies I join is unnecessarily complicated. I often end up simplifying things once I get a handle for how it works, creating nice one command deploys or workflows that don't require chaining multiple things or memorizing a gazillion abstractions. Meanwhile, people often waste lots of times on inefficient build tooling and multiple steps to get to the end goal.
None of the companies I have been at created a good model system (although one came close, the employee responsible left due to the startup acting like a consultancy instead of focusing on a quality product) - this has often bit companies hard as data structures start mutating due to changing API & changing requirements, making certain things such as offline caching more difficult.
Architectural choice tends to roll into poor time constraints since poor time constraints puts even more pressure on the code base when poor architecture is involved.
None of the companies I have been at created a good model system (although one came close, the employee responsible left due to the startup acting like a consultancy instead of focusing on a quality product) - this has often bit companies hard as data structures start mutating due to changing API & changing requirements, making certain things such as offline caching more difficult.
Architectural choice tends to roll into poor time constraints since poor time constraints puts even more pressure on the code base when poor architecture is involved.
It does suck. Most code sucks. We just get used to it. Doesn't mean it doesn't work, doesn't mean it's not valuable. But it still sucks.
As for architecture being the biggest technical debt problem that should be obvious. Architecture is often the most difficult thing to change after the fact.
As for architecture being the biggest technical debt problem that should be obvious. Architecture is often the most difficult thing to change after the fact.
"Good code" is often a euphemism for understandable code.
Understandability is an important metric for good code (within reason).
It just shows how completely subjective it is, and that "technical debt" is becoming an excuse for anything and everything.
I know someone who says all the time that what they want is "The cheapest, fastest (least time to complete), most predictable thing to get something done" and "that if it needs fixing, it can be done when the customer pays for it"
Even green field projects there wind up with significant technical debt in under a year or two.
Even green field projects there wind up with significant technical debt in under a year or two.
Notice that nowhere in the article is there any mention of what a good architectural choice is, nor any mention of the perception of debt in relation to the size of the system[0]. My personal experience with architectural choices is that all of them are great early on when the code base is small. As that code base grows, they all suffer under the weight.
[0]: 100k-1M is a very large range. Kudos to them for filtering out small code bases, but a graph of these perceptions measured against code size would have been helpful.
[0]: 100k-1M is a very large range. Kudos to them for filtering out small code bases, but a graph of these perceptions measured against code size would have been helpful.
(author here)
We cross-tabbed system age vs perception of amount of TD. There was a moderate association between older systems (> 6 yrs) and more perceived debt.
I did not explicitly look at size as this was not one of the original research questions, but a good point. I suspect older systems will tend to be larger (in the domains we studied, anyway). And your point about arch choices being great "early on" can, I think, be captured in the "system age" variable. I guess I'm trying to think of a system that might be young and yet quite large in LOC.. would be an interesting outlier to look at.
I did not explicitly look at size as this was not one of the original research questions, but a good point. I suspect older systems will tend to be larger (in the domains we studied, anyway). And your point about arch choices being great "early on" can, I think, be captured in the "system age" variable. I guess I'm trying to think of a system that might be young and yet quite large in LOC.. would be an interesting outlier to look at.
I guess I'm trying to think of a system that might be young and yet quite large in LOC.. would be an interesting outlier to look at.
Take a look at video game companies. They tend to build up a large infrastructure early in the development of a game.
Keep up the great work! I love to see scientific research being done on software development. :)
Take a look at video game companies. They tend to build up a large infrastructure early in the development of a game.
Keep up the great work! I love to see scientific research being done on software development. :)
Part of this is the fresh-eyes effect. Just having someone new look at things should result in good stuff happening in the first three months.
Shameless plug: I didn't know about their SonarQube project until I'd built debt ceiling, but while ruby only at the moment, it is an open source attempt to provide some mechanics for defining and tracking some quantified approximation of technical debt. No claims it's perfect of course, but would love more real world feedback and/or feature requests! Also would like to extend it to handle JavaScript analysis as well, ideally through a plug-in architecture that could easily handle arbitrary other languages as well.
https://github.com/bglusman/debt_ceiling
https://github.com/bglusman/debt_ceiling
I like the tool. As for the extensions, I would suggest that in my experience SonarQube has the same ability for extension, and does most of its work in the open. So it might be worth considering turning your project into a plugin (or plugins) for SonarQube?
Thanks for sharing this. I'm not working on a ruby project, but if I was, I'd give this a try.
I like how `debt_ceiling` supports `TODO` or `# TECHDEBT` tags.
When I think of technical debt, there are some kinds that don't seem easy to automatically measure: "we made an assumption 6 months ago that turned out to be quite wrong, but we haven't had enough time to go back and rework it yet". This could manifest as poor choices of data structures, or an entire system/subsystem that turns out to be nonsensical given increased understanding of requirements. People who understand the problems can manually tag the code, so it's good to see tool support for these kinds of annotations.
I like how `debt_ceiling` supports `TODO` or `# TECHDEBT` tags.
When I think of technical debt, there are some kinds that don't seem easy to automatically measure: "we made an assumption 6 months ago that turned out to be quite wrong, but we haven't had enough time to go back and rework it yet". This could manifest as poor choices of data structures, or an entire system/subsystem that turns out to be nonsensical given increased understanding of requirements. People who understand the problems can manually tag the code, so it's good to see tool support for these kinds of annotations.
Oh very cool! I will check it out.
There were 39 separate business units represented among the three companies
The context in which this study was done is quite different from an early stage startup. This is not surprising since the SEI tends to focus on big engineering solutions to big problems [which is not to say that its insights and guidance aren't applicable]. But it is important to realize that they focus on engineering in situations that are relatively stable over the long term (29% of systems studied over 10 years old), and almost 1/3 of the systems studied were embedded where changing the software is particularly difficult and throwing more hardware at a problem is not cost effective.
The context in which this study was done is quite different from an early stage startup. This is not surprising since the SEI tends to focus on big engineering solutions to big problems [which is not to say that its insights and guidance aren't applicable]. But it is important to realize that they focus on engineering in situations that are relatively stable over the long term (29% of systems studied over 10 years old), and almost 1/3 of the systems studied were embedded where changing the software is particularly difficult and throwing more hardware at a problem is not cost effective.
There are some grievances in codebases that do not come with a huge expense. For example, duplicated code. For another example, overly verbose code. I've seen a lot of new codebases and these are minor nuisances and are pretty easy (i.e., take little time) to cope with. I don't classify these as technical debt because if they cost anything it's always so little.
What costs the most is architecturally unsound code. It's actually not really the code per se, it's the abstractions and how they fit together--very specifically, to what extent the abstractions cohere, to what extent they are coupled, and how narrow are the interfaces. Give me a code base that nails these qualities (narrow interfaces, minimal dependencies) and all else (duplicate code, inelegant code) is in comparison NOTHING to deal with.
So the cost of code is almost entirely related to its _architectural soundness_ (to what degree it's coupled, its interfaces are too wide, etc.)
The biggest and prevelant fallacy of our industry is this notion of technical debt. Because the notion implies that it's somehow necessary. It's not necessary.
The ability to build and evolve architecturally sound code is a function of skillset. If the skillset is there, there is no necessary extra cost. In fact, for people that have this skillset it costs MORE to depart from instincts and couple code, create wide interfaces, etc.
When a developer says 'It will take too long to do the right thing,' (i.e., build architecturally sound code) he/she is saying, 'I don't have the skillset.'
I know all the objections here (it takes time to understand your use cases, etc.), but the objections nearly always miss the fact that architectural soundness and things like code scope/flexibility are ORTHOGONAL properties. You can have very specific code that does very specific minimal-viable product type things ... but this has nothing to do with whether the code is architectural sound or not.
As long as we continue to think of techincal debt as something that's somehow necessary and something to be negotiated we will always be missing the point. The point is that we need to produce workers with the right skillset. There are so few people that have it--which is entirely a failure in our academies, educators, training systems... To keep talking about technical debt and not about how to acquire the architectural skillset is to keep chasing our tails.
What costs the most is architecturally unsound code. It's actually not really the code per se, it's the abstractions and how they fit together--very specifically, to what extent the abstractions cohere, to what extent they are coupled, and how narrow are the interfaces. Give me a code base that nails these qualities (narrow interfaces, minimal dependencies) and all else (duplicate code, inelegant code) is in comparison NOTHING to deal with.
So the cost of code is almost entirely related to its _architectural soundness_ (to what degree it's coupled, its interfaces are too wide, etc.)
The biggest and prevelant fallacy of our industry is this notion of technical debt. Because the notion implies that it's somehow necessary. It's not necessary.
The ability to build and evolve architecturally sound code is a function of skillset. If the skillset is there, there is no necessary extra cost. In fact, for people that have this skillset it costs MORE to depart from instincts and couple code, create wide interfaces, etc.
When a developer says 'It will take too long to do the right thing,' (i.e., build architecturally sound code) he/she is saying, 'I don't have the skillset.'
I know all the objections here (it takes time to understand your use cases, etc.), but the objections nearly always miss the fact that architectural soundness and things like code scope/flexibility are ORTHOGONAL properties. You can have very specific code that does very specific minimal-viable product type things ... but this has nothing to do with whether the code is architectural sound or not.
As long as we continue to think of techincal debt as something that's somehow necessary and something to be negotiated we will always be missing the point. The point is that we need to produce workers with the right skillset. There are so few people that have it--which is entirely a failure in our academies, educators, training systems... To keep talking about technical debt and not about how to acquire the architectural skillset is to keep chasing our tails.
> When a developer says 'It will take too long to do the right thing,' (i.e., build architecturally sound code) he/she is saying, 'I don't have the skillset.'
I'll be charitable and say this may be the case part of the time. But as a general argument it is bollocks. In the vast majority of production systems that survive for any amount of time there will be requirements changes, and sooner or later an engineer will face a choice of shoehorning an attribute in somewhere it doesn't fit naturally, versus refactoring to more appropriately model the new business universe. Saying that this won't happen if your skills are sufficient is a no-true-scotsman argument—it's all tradeoffs, and no matter how good you are at anticipating change, sooner or later even the best architect will be blindsided by some out-of-the-blue business requirement.
I'll be charitable and say this may be the case part of the time. But as a general argument it is bollocks. In the vast majority of production systems that survive for any amount of time there will be requirements changes, and sooner or later an engineer will face a choice of shoehorning an attribute in somewhere it doesn't fit naturally, versus refactoring to more appropriately model the new business universe. Saying that this won't happen if your skills are sufficient is a no-true-scotsman argument—it's all tradeoffs, and no matter how good you are at anticipating change, sooner or later even the best architect will be blindsided by some out-of-the-blue business requirement.
This is the fallacy I was referring to--architectural soundness has nothing to do with anticipating change.
The "choice" your engineer faces to shoehorn a feature in [1] versus implement it in a way that optimizes for soundness (decoupled code w/ narrow interfaces) is not a choice if the skill set is there--i.e., if he/she has trained and practiced the right skill set.
[1] (By 'shoehorn' I assume you mean implement it in a way that leaves the code base in an unsound state.)
The "choice" your engineer faces to shoehorn a feature in [1] versus implement it in a way that optimizes for soundness (decoupled code w/ narrow interfaces) is not a choice if the skill set is there--i.e., if he/she has trained and practiced the right skill set.
[1] (By 'shoehorn' I assume you mean implement it in a way that leaves the code base in an unsound state.)
You can't just reaffirm that without evidence and an argument. It is self-evident to me (and I presume every other software engineer here) that different architectures lend themselves to different types of change. If you want to make that claim than you probably should write a dissertation about it because it's quite a controversial claim. But simply stated as if it were unassailable fact is pushing you right into crackpot territory.
> It is self-evident to me that different architectures lend themselves to different types of change.
This is such a vague statement, I have no idea what you mean or how it relates to our discussion.
My argument is this:
(1) The most costly issue with a code base has to do with its architectural properties. How architecturally sound it is [1]. Most other issues that we tend to categorize as technical debt (like code duplication or verbose code, for examples) pose such tiny cost in comparison to the cost of poorly architected code [2].
(2) Architectural soundness is an objective property of code independent of the code's generality, flexibility, etc. Again, it's the degree to which its components and its interfaces narrow.
(3) There exists a skill set that understands, recognizes and produces architecturally sound code. [3]
(4) Given an architecturally sound code base and a set of features, a skilled person can implement the features without violating the architectural soundness of the code base AND do so without having to pay for it in time-to-delivery. [4]
I think I'm being fairly clear in my assertions here. They are based on my long experience. To disagree is to pose a different theory. Do you care to elaborate on what you mean by 'crackpot territory'?
---
[1] More specifically, the degree to which its components (modules, classes, functions) are coupled and their interfaces not as narrow as they could be. This is an objective property of a code that has clearly delineated components (i.e., you can quantify dependencies, interface sizes, module sizes.)
[2] Yes, this is a claim based on my 20 years of experience working in myriad code bases. How to distill and prove this, formalize the argument, etc is probably work to be done.
[3] This skill set doesn't come cheap and requires proper training, and currently our industry/academies aren't producing it.
[4] Again, this claim based on my experience and yes there is probably some good research opportunity here to formalize it.
This is such a vague statement, I have no idea what you mean or how it relates to our discussion.
My argument is this:
(1) The most costly issue with a code base has to do with its architectural properties. How architecturally sound it is [1]. Most other issues that we tend to categorize as technical debt (like code duplication or verbose code, for examples) pose such tiny cost in comparison to the cost of poorly architected code [2].
(2) Architectural soundness is an objective property of code independent of the code's generality, flexibility, etc. Again, it's the degree to which its components and its interfaces narrow.
(3) There exists a skill set that understands, recognizes and produces architecturally sound code. [3]
(4) Given an architecturally sound code base and a set of features, a skilled person can implement the features without violating the architectural soundness of the code base AND do so without having to pay for it in time-to-delivery. [4]
I think I'm being fairly clear in my assertions here. They are based on my long experience. To disagree is to pose a different theory. Do you care to elaborate on what you mean by 'crackpot territory'?
---
[1] More specifically, the degree to which its components (modules, classes, functions) are coupled and their interfaces not as narrow as they could be. This is an objective property of a code that has clearly delineated components (i.e., you can quantify dependencies, interface sizes, module sizes.)
[2] Yes, this is a claim based on my 20 years of experience working in myriad code bases. How to distill and prove this, formalize the argument, etc is probably work to be done.
[3] This skill set doesn't come cheap and requires proper training, and currently our industry/academies aren't producing it.
[4] Again, this claim based on my experience and yes there is probably some good research opportunity here to formalize it.
> Architectural soundness is an objective property of code independent of the code's generality, flexibility, etc. Again, it's the degree to which its components and its interfaces narrow.
I like this definition, and I agree you could quantify this into a useful measure. However the optimal architecture according to this measure is a function of the problem domain. You split the modules and define the interfaces to maximize the encapsulation and minimize the interfaces.
So what happens when the business rules introduces a new cross-cutting concept? The basis of the architecture soundness is now in question. Do you define a new module and add n new interfaces? Or do you split the concern into the existing modules, fattening them in the process, and widening existing interfaces as well?
I like this definition, and I agree you could quantify this into a useful measure. However the optimal architecture according to this measure is a function of the problem domain. You split the modules and define the interfaces to maximize the encapsulation and minimize the interfaces.
So what happens when the business rules introduces a new cross-cutting concept? The basis of the architecture soundness is now in question. Do you define a new module and add n new interfaces? Or do you split the concern into the existing modules, fattening them in the process, and widening existing interfaces as well?
> However the optimal architecture according to this measure is a function of the problem domain.
I agree with this. But domains typically evolve incrementally -- and using the skillsets I mention you can evolve your code accordingly. [1]
There is then the question of these cross-cutting, earth-shattering domain changes you mention. I'm not convinced my argument doesn't have a play here but regardless if these kinds of major domain changes are happening frequently enough to matter in this discussion then I think there's a different business-level problem going on.
It's hard to talk abstractly about this for so long--until of course someone gets to building out these ideas more formally. Getting into concrete cases would be more fruitful.
---
[1] That you can without cost tradeoff is my argument. Today these skills are so rare so you don't find orgs/codebases that do this properly--and therefore we have this fallacy that because it's prevalent that it's somehow necessary. We don't realize it's really a skills problem.
I agree with this. But domains typically evolve incrementally -- and using the skillsets I mention you can evolve your code accordingly. [1]
There is then the question of these cross-cutting, earth-shattering domain changes you mention. I'm not convinced my argument doesn't have a play here but regardless if these kinds of major domain changes are happening frequently enough to matter in this discussion then I think there's a different business-level problem going on.
It's hard to talk abstractly about this for so long--until of course someone gets to building out these ideas more formally. Getting into concrete cases would be more fruitful.
---
[1] That you can without cost tradeoff is my argument. Today these skills are so rare so you don't find orgs/codebases that do this properly--and therefore we have this fallacy that because it's prevalent that it's somehow necessary. We don't realize it's really a skills problem.
> I don't classify these as technical debt because
Apologies if this is too meta, but that is the most interesting part of your comment for me. It conceals a massive assumption which all devs make, resulting in very different conclusions.
What IS technical debt?
My own preference for the answer is Uncle Bob's description, which is this: technical debt is any production code that does not have (good) tests.
But regardless, any attempts to address technical debt must necessarily begin with a common understanding of what technical debt actually is.
I've rarely seen teams agree that understanding and I think that contributes to their problems.
Apologies if this is too meta, but that is the most interesting part of your comment for me. It conceals a massive assumption which all devs make, resulting in very different conclusions.
What IS technical debt?
My own preference for the answer is Uncle Bob's description, which is this: technical debt is any production code that does not have (good) tests.
But regardless, any attempts to address technical debt must necessarily begin with a common understanding of what technical debt actually is.
I've rarely seen teams agree that understanding and I think that contributes to their problems.
> What IS technical debt?
Any feature of the code or tech stack that increases maintenance/operations cost compared to alternatives.
> My own preference for the answer is Uncle Bob's description, which is this: technical debt is any production code that does not have (good) tests.
That's certainly an example of technical debt.
Any feature of the code or tech stack that increases maintenance/operations cost compared to alternatives.
> My own preference for the answer is Uncle Bob's description, which is this: technical debt is any production code that does not have (good) tests.
That's certainly an example of technical debt.
> > My own preference for the answer is Uncle Bob's description, which is this: technical debt is any production code that does not have (good) tests.
> That's certainly an example of technical debt.
Agreed, it is not the only example, but perhaps it is a good one, as that is a particularly important form of debt that makes the code harder to safely change. I.e. it is a form of technical debt that makes it more expensive to pay off other kinds of technical debt.
Curiously, Michael Feathers has a similar definition of legacy code [1]:
> To me, legacy code is simply code without tests.
[1] http://www.amazon.com/dp/0131177052
> That's certainly an example of technical debt.
Agreed, it is not the only example, but perhaps it is a good one, as that is a particularly important form of debt that makes the code harder to safely change. I.e. it is a form of technical debt that makes it more expensive to pay off other kinds of technical debt.
Curiously, Michael Feathers has a similar definition of legacy code [1]:
> To me, legacy code is simply code without tests.
[1] http://www.amazon.com/dp/0131177052
I think "code without tests" is a fine example of technical debt or legacy code, but using it as the definition of either disregards the importance of everything other than tests that goes into making systems maintainable.
Tests are necessary but not sufficient. (Particularly, tests embody
Tests are necessary but not sufficient. (Particularly, tests embody
I remember reading this http://higherorderlogic.com/2010/07/bad-code-isnt-technical-...
According to the author call options are a better metaphor for what we call technical debt but admittedly a more difficult one to understand: anyone who knows what a call option is raise hand.
According to the author call options are a better metaphor for what we call technical debt but admittedly a more difficult one to understand: anyone who knows what a call option is raise hand.
I work in an organisation that has a large technical debt. The organisation manages at least $A60b of member's assets, and is overseen by the Australian Tax Office. It also is very visible politically. These three factors make it extremely conservative in terms of change.
However, change is imposed on it inevitably from the outside, so instead of making large-scale changes to its architecture, it tries to keep up with the market by making thousands of small hacks. The original architecture was not designed to cope with this, so it has become hack upon hack upon hack. I should be employed (and paid above-average) in this jobs for decades to come.
However, change is imposed on it inevitably from the outside, so instead of making large-scale changes to its architecture, it tries to keep up with the market by making thousands of small hacks. The original architecture was not designed to cope with this, so it has become hack upon hack upon hack. I should be employed (and paid above-average) in this jobs for decades to come.
Sounds like a system I worked on for an investment bank. Sure you will be employed for years fixing hack upon hack, but its not very rewarding work (in the non financial sense).
Don't underestimate the value/reward/fun/etc that you can get from a fresh project. New problems, new development process, the chance to re-iterate past designs and architectures, integration with existing/old/legacy systems, and a host of other things.
That's my bit of advice to you on that, as it reads a bit like you're very comfortable and expecting it to stay that way for "decades to come".
That's my bit of advice to you on that, as it reads a bit like you're very comfortable and expecting it to stay that way for "decades to come".
I do work on my own projects that are fun and interesting. But nothing beats a regular, permanent high-earning job, no matter how boring it is. I personally know a couple of developers who are stressed financially after working for "fun" companies.
I am not sure I see the advantage in taking a metaphor that is a little off but understandable and replacing it with one that is more accurate, but very few people understand.
I can't make heads or tails of the first figure in the post.
The reason I bring this up is I am surprised that architectural choice is overwhelmingly the top technical debt concern. That doesn't seem right. I would expect it to be poor code due to time constraints.