Software Development as an Emergent System(blinkingcaret.com)
blinkingcaret.com
Software Development as an Emergent System
https://www.blinkingcaret.com/2018/08/22/software-development-emergent-system-ndepend/
4 comments
The one company that I worked for that did everything “right” - unit tests, paying down technical debt, code reviews, etc. was led by a manager who was originally the founder of the product that got acquired by what was then a Fortune 10 company.
After two years working on the new well tested, well written product, no one wanted the new .Net product, they stopped working on it and put everyone on the old PHP based product where there was a demand. Within 6 months after people’s vesting period elapsed, everyone (14+) developers left because no one wanted to do PHP.
Also, when he was a young hungry startup, he didn’t worry about “good practices”, but now that he was flush with cash and didn’t have to worry about going out of business, he could do things by the book.
I learned a few lessons over the years after that product and seeing how things really work - especially at smaller VC funded companies. No one cares about code quality, just growing features, acquiring customers, and the exit strategy.
It worked for Twitter....
After two years working on the new well tested, well written product, no one wanted the new .Net product, they stopped working on it and put everyone on the old PHP based product where there was a demand. Within 6 months after people’s vesting period elapsed, everyone (14+) developers left because no one wanted to do PHP.
Also, when he was a young hungry startup, he didn’t worry about “good practices”, but now that he was flush with cash and didn’t have to worry about going out of business, he could do things by the book.
I learned a few lessons over the years after that product and seeing how things really work - especially at smaller VC funded companies. No one cares about code quality, just growing features, acquiring customers, and the exit strategy.
It worked for Twitter....
Most care about velocity, and doing work to support that is usually defensible.
You can keep velocity high on a badly run project for a couple years, but if your roadmap or business plan is long term, you will find that 5ings calcify before you’re ready for them. I always try to remind people that this is about the long term.
It’s also easier to grow and shrink a team when the code is Clean. But some of your coworkers won’t like the idea that they can be gotten rid of so easily. At the heart of its that’s the struggle. One or two experts on the system is enough to keep those people and their bosses fat and happy for quite a while.
You can keep velocity high on a badly run project for a couple years, but if your roadmap or business plan is long term, you will find that 5ings calcify before you’re ready for them. I always try to remind people that this is about the long term.
It’s also easier to grow and shrink a team when the code is Clean. But some of your coworkers won’t like the idea that they can be gotten rid of so easily. At the heart of its that’s the struggle. One or two experts on the system is enough to keep those people and their bosses fat and happy for quite a while.
Did the products do the same thing, or was the new product something entirely different that didn't have any market fit?
In other words, what's the lesson here? Speed at all costs because you quickly figure out if the product is wanted? Or speed at all costs because bugs just don't matter?
In other words, what's the lesson here? Speed at all costs because you quickly figure out if the product is wanted? Or speed at all costs because bugs just don't matter?
Without going into too much detail, the initial product was for the B2B market with buyers and sellers where the buyers could also be sellers. They had near total penetration on the seller’s side but they wanted to make a similar product that was buyer focused.
The lesson, for new markers that aren’t heavily regulated and the cost of bugs isn’t high (like healthcare and banking) is speed at all cost, and then once you have penetration, then start worrying about technical debt.
The lesson, for new markers that aren’t heavily regulated and the cost of bugs isn’t high (like healthcare and banking) is speed at all cost, and then once you have penetration, then start worrying about technical debt.
>The lesson, for new markers that aren’t heavily regulated and the cost of bugs isn’t high (like healthcare and banking) is speed at all cost, and then once you have penetration, then start worrying about technical debt.
Sounds like wisdom. I've been thinking along those lines myself.
I wonder what this means for correctness-focused languages (functional, with static and strong typing). Personally I'm a fan of these, but it could be that they're just not desirable in most industries. I'm assuming that these languages slow you down, but I hope they don't all do that.
Sounds like wisdom. I've been thinking along those lines myself.
I wonder what this means for correctness-focused languages (functional, with static and strong typing). Personally I'm a fan of these, but it could be that they're just not desirable in most industries. I'm assuming that these languages slow you down, but I hope they don't all do that.
Static and strong typing not only helps eliminate a whole class of bugs, there are also automated, guaranteed safe types of refactors that you can do with static, strongly typed languages.
I think it's a matter of attitude. Either you accept that there is value in keeping a code base reasonable or not. Maybe some metrics can help to make a point but any metric will be abused over time if used too religiously.
You are making a good point about people not being around long term. In my company there are departments with high turnover and they tend to go for quick victories that are often costly in the long term. At the time this becomes clear the manager who made the previous has already been promoted somewhere else and the next guy will make another quick decision to fix the previous problem. Then this guy gets promoted too and the cycle continues.
Technical debt is really only a problem if you think long term.
You are making a good point about people not being around long term. In my company there are departments with high turnover and they tend to go for quick victories that are often costly in the long term. At the time this becomes clear the manager who made the previous has already been promoted somewhere else and the next guy will make another quick decision to fix the previous problem. Then this guy gets promoted too and the cycle continues.
Technical debt is really only a problem if you think long term.
A good analogy for older management, which is disappearing as an extant object in the office, is that of the file room. Sometimes, you'll see a seeming "whiz kid" file clerk who seems to be far faster than normal, but it turns out that he's "faster" because he doesn't actually re-file and reorganize.
Refactoring is quite literally the same thing as re-filing and reorganizing physical files. Under the academic definition of refactoring, the "contents" (the functionality) is not changing. Only the organization is changing.
Refactoring is quite literally the same thing as re-filing and reorganizing physical files. Under the academic definition of refactoring, the "contents" (the functionality) is not changing. Only the organization is changing.
A colleague and I developed a good routine for a while. He's a much faster programmer than me. But his code quality is usually much lower (not wrong, but often poorly structured and hard to maintain). He'd code fast, I'd code slow. Then I'd clean up his stuff while we waited for feedback on the system and he did other things. Refactoring work and the like, no meaningful (to an end user) changes.
This was effective. Except that management saw him as fast and so he kept getting more work, and completing it. But the quality wasn't there because no one went in behind him to keep it clean and tidy. Management never really saw the issue, they just thought the rest of us were bad programmers because we were so slow. We were slow because: 1) some of us (like me) work slowly; 2) the fast guy introduced a lot of technical debt (or we introduced it because we weren't given the cleanup time I used to have).
He's still a good programmer (in general), but without the support structure his products are far less useful than they same at first glance (due to the introduced technical debt).
This was effective. Except that management saw him as fast and so he kept getting more work, and completing it. But the quality wasn't there because no one went in behind him to keep it clean and tidy. Management never really saw the issue, they just thought the rest of us were bad programmers because we were so slow. We were slow because: 1) some of us (like me) work slowly; 2) the fast guy introduced a lot of technical debt (or we introduced it because we weren't given the cleanup time I used to have).
He's still a good programmer (in general), but without the support structure his products are far less useful than they same at first glance (due to the introduced technical debt).
Having to work in and around very bad code slows everybody else down, we have a very prolific guy who creates his own jargon (inconsistently) and enjoys convolution like it’s an art form. Everybody else has to spend twice as long doing the most rudimentary features.
But management loves their heroes, so the people with high tolerance for complexity get promoted, and all the people I trust keep leaving...
But management loves their heroes, so the people with high tolerance for complexity get promoted, and all the people I trust keep leaving...
I have worked with people like that. They are really good for prototyping new stuff. But you definitely don't want their code go straight into a long term code base.
Indeed, sort of like how not ever bathing is only a problem if you think long term.
To paraphrase Quentin Crisp, after the first few years the smell doesn't get any worse.
In software it actually gets worse and worse and worse....
Oh I know. It's Quentin Crisp's joke, if you know who that is.
I worked for two years at a big corporation (mostly as an experiment, but also to prove that I could) on a huge and somewhat important internal project (meaning that we're were never going to be killed because we represented a thing that would save millions of dollars when it finally worked AND we had no external users that could abandon us, ours was a captive audience.)
> At the time this becomes clear the manager who made the previous has already been promoted somewhere else and the next guy will make another quick decision to fix the previous problem. Then this guy gets promoted too and the cycle continues.
Not only did I see that happen two or three times while I was there, but the project itself had been crapped out by some cowboy coder who stood up a demo, got some traction for it, and moved on to greater things. He had a rep as a Rockstar because of this project but if you actually looked at what he did (as I had to every day) it was a rickety un-designed P.O.S. that could barely function. It was a glorified spreadsheet basically, but of course not implemented on top of the in-house spreadsheet product. We rolled our own TableView... It also crashed on importing more than 25K (not M, K) records, which was one of the main reasons it had been started in the first place: the existing system it was meant to replace was slow on large ("large") datasets. The new system wasn't slow, it just crashed. This was progress.
But the money was good. Food too.
-----
edit, I started getting flashbacks to that job after writing the above. Hoo boy was that ever a shocking place to work. I feel like I was kidnapped by aliens for two years.
The software was a textbook case of the "Lava Flow" anti-pattern: http://antipatterns.com/lavaflow.htm It had at least five distinct waves of rewrites, all of which were left petrified in the code.
At one point the dispatch system was rewritten in this weird not-OOP, not-Aspect, not-Actor, style, during a wave of the "composition-not-inheritance" fad that comes and goes. I remember opening up one of the affect modules and thinking, "This is like Romulan code." It was so strange.
I worked for two years at a big corporation (mostly as an experiment, but also to prove that I could) on a huge and somewhat important internal project (meaning that we're were never going to be killed because we represented a thing that would save millions of dollars when it finally worked AND we had no external users that could abandon us, ours was a captive audience.)
> At the time this becomes clear the manager who made the previous has already been promoted somewhere else and the next guy will make another quick decision to fix the previous problem. Then this guy gets promoted too and the cycle continues.
Not only did I see that happen two or three times while I was there, but the project itself had been crapped out by some cowboy coder who stood up a demo, got some traction for it, and moved on to greater things. He had a rep as a Rockstar because of this project but if you actually looked at what he did (as I had to every day) it was a rickety un-designed P.O.S. that could barely function. It was a glorified spreadsheet basically, but of course not implemented on top of the in-house spreadsheet product. We rolled our own TableView... It also crashed on importing more than 25K (not M, K) records, which was one of the main reasons it had been started in the first place: the existing system it was meant to replace was slow on large ("large") datasets. The new system wasn't slow, it just crashed. This was progress.
But the money was good. Food too.
-----
edit, I started getting flashbacks to that job after writing the above. Hoo boy was that ever a shocking place to work. I feel like I was kidnapped by aliens for two years.
The software was a textbook case of the "Lava Flow" anti-pattern: http://antipatterns.com/lavaflow.htm It had at least five distinct waves of rewrites, all of which were left petrified in the code.
At one point the dispatch system was rewritten in this weird not-OOP, not-Aspect, not-Actor, style, during a wave of the "composition-not-inheritance" fad that comes and goes. I remember opening up one of the affect modules and thinking, "This is like Romulan code." It was so strange.
I'm working on a couple code bases that are 10+ years old in a mature organization, but for internal customers only. In those cases there's definitely awareness that paying down technical debt is worthwhile since everyone can remember times where the debt ballooned and it was painful.
Other projects not so much, especially in this day and age where rapid delivery and throwaway is actually an accepted form of SW development.
The fact that Agile and its ilk sprung up and mostly failed to address these issues shows how hard a problem this is.
Other projects not so much, especially in this day and age where rapid delivery and throwaway is actually an accepted form of SW development.
The fact that Agile and its ilk sprung up and mostly failed to address these issues shows how hard a problem this is.
Where I work, the basic doctrine is that Engineering can schedule "chores" -- refactoring, tech debt paydown etc -- according to its best judgement. We don't interrupt Product's priorities willy-nilly but depending on the cadence you'll find a certain amount of tidying working going on all the time.
There's also a fairly high bar for what gets marked as "Done", so the rework is a bit lower. I've been on stories where we spent as much time dealing with the architectural consequences of a new feature as we did on the visible bits.
Sometimes it can be hard. Especially in the anchorship role. I've once had to tie up an entire track of work to fix the tech debt equivalent of a toxic spill for weeks and weeks. It sucked, there was zero user-visible value gained, but it simply had to be done or we would suffer more and more.
I dunno if that's considered agile or not. At some level it doesn't bother me.
There's also a fairly high bar for what gets marked as "Done", so the rework is a bit lower. I've been on stories where we spent as much time dealing with the architectural consequences of a new feature as we did on the visible bits.
Sometimes it can be hard. Especially in the anchorship role. I've once had to tie up an entire track of work to fix the tech debt equivalent of a toxic spill for weeks and weeks. It sucked, there was zero user-visible value gained, but it simply had to be done or we would suffer more and more.
I dunno if that's considered agile or not. At some level it doesn't bother me.
I don't think the author understands what "emergence" is. The concept of emergence is that a complex system might have properties that CANNOT be explained by studying the parts.
The talk about boids made me think this would go in a different direction.
Programmers and our employers exhibit flocking behavior. See how Ruby on Rails brought so many people to the Ruby language. How Python had a similar surge earlier in the 00s, and later with the explosion of use (numpy and similar) in data science applications. Additionally, things like Ruby on Rails helped make the web application commonplace. See JavaScript and AJAX for that as well.
Perl established an effective code repository (CPAN) that enabled their userbase to grow via a similar flocking mechanism. Go has no central repository but has an easy to use mechanism for adding dependencies based on git repositories. That ease of access to published modules allows people to adopt the technology once a peer (someone working in a similar domain) publishes something. And it encourages people to contribute back in some fashion (publish new modules or update the existing ones). In Perl's case you saw a lot of people from outside IT making use of it (the same is now true for Python with data science) because the heavy lifting was done and they could focus on gluing things together.
Exploiting this flocking behavior, then, becomes important to anyone wanting to push a new technology out. Even if it's the best thing and would produce the best software ever, it won't always succeed on its own merits.
(random thoughts, no real thesis here)
Programmers and our employers exhibit flocking behavior. See how Ruby on Rails brought so many people to the Ruby language. How Python had a similar surge earlier in the 00s, and later with the explosion of use (numpy and similar) in data science applications. Additionally, things like Ruby on Rails helped make the web application commonplace. See JavaScript and AJAX for that as well.
Perl established an effective code repository (CPAN) that enabled their userbase to grow via a similar flocking mechanism. Go has no central repository but has an easy to use mechanism for adding dependencies based on git repositories. That ease of access to published modules allows people to adopt the technology once a peer (someone working in a similar domain) publishes something. And it encourages people to contribute back in some fashion (publish new modules or update the existing ones). In Perl's case you saw a lot of people from outside IT making use of it (the same is now true for Python with data science) because the heavy lifting was done and they could focus on gluing things together.
Exploiting this flocking behavior, then, becomes important to anyone wanting to push a new technology out. Even if it's the best thing and would produce the best software ever, it won't always succeed on its own merits.
(random thoughts, no real thesis here)
The simple model for this kind of thing is Polya's Urn.
Take an urn with green and red balls (or blue and orange, if you're colour-blind).
Pick out a ball. If it is red, put it back and replace a green ball with another red ball. If green, put it back and replace a red ball with a green.
Even starting with a perfect 50:50, and choosing completely randomly, a Polya process will eventually show that your probability of picking one colour becomes more or less certain. The urn "locks" into a particular path. While in theory it could return to 50:50 or swing back to the other colour, the odds of doing so rapidly become astronomical.
John Sterman's Business Dynamics: Systems Thinking and Modeling for a Complex World has a really excellent discussion. Worth looking for.
Take an urn with green and red balls (or blue and orange, if you're colour-blind).
Pick out a ball. If it is red, put it back and replace a green ball with another red ball. If green, put it back and replace a red ball with a green.
Even starting with a perfect 50:50, and choosing completely randomly, a Polya process will eventually show that your probability of picking one colour becomes more or less certain. The urn "locks" into a particular path. While in theory it could return to 50:50 or swing back to the other colour, the odds of doing so rapidly become astronomical.
John Sterman's Business Dynamics: Systems Thinking and Modeling for a Complex World has a really excellent discussion. Worth looking for.
That book actually just made it onto my Amazon wishlist (for other reasons). I know I'm unlikely to succeed in my current company, but I'm trying to increase my understanding/fluency in systems thinking in order to improve the company's understanding. Realistically, this means I'm building a skillset for my next job.
That particular model is interesting for what I described. Essentially, if someone wants to program they'll be drawn to one of the projects/people already out there which are already using some language. They'll then use the same language and become a person or produce a project that the next person may be drawn to. The original thing that drew them to the language is still there, and they're another point to draw someone else to the same language.
Thanks.
That particular model is interesting for what I described. Essentially, if someone wants to program they'll be drawn to one of the projects/people already out there which are already using some language. They'll then use the same language and become a person or produce a project that the next person may be drawn to. The original thing that drew them to the language is still there, and they're another point to draw someone else to the same language.
Thanks.
Exactly, this is path dependency in a case where value comes from network effects. I have occasion to apply this material in thinking about my work and I'm hoping to get to apply more of it as time goes.
Sterman's is the best all-round book on systems thinking I've read. Not too hand-wavy, not swimming in calculus, plenty of interesting case studies and practical advice that show deep experience. It's in a sweet spot for me.
Edit: as a note, it always seems to be substantially cheaper on Book Depository than on Amazon. I suspect that bots are, hilariously, in a feedback loop bidding up a book about feedback loops.
Sterman's is the best all-round book on systems thinking I've read. Not too hand-wavy, not swimming in calculus, plenty of interesting case studies and practical advice that show deep experience. It's in a sweet spot for me.
Edit: as a note, it always seems to be substantially cheaper on Book Depository than on Amazon. I suspect that bots are, hilariously, in a feedback loop bidding up a book about feedback loops.
This [0] is something I came across while looking up information on systems dynamics/thinking in general. That's how the book ended up on my list. I haven't gone through it yet but I probably will over the next few weeks now that my life/schedule has settled down a bit.
I've also been reading Weinberg's General Systems Thinking (bundle with it and some other books at Leanpub[1]). I can't say I've seen anything new in it yet, but it (like many books) is connecting dots for me that I hadn't connected before. Reading through it, I wish I'd seen some of its content when I was younger, it could've made some things easier over the years.
[0] https://ocw.mit.edu/courses/sloan-school-of-management/15-87...
[1] https://leanpub.com/b/generalsystemsthinkerbundle
I've also been reading Weinberg's General Systems Thinking (bundle with it and some other books at Leanpub[1]). I can't say I've seen anything new in it yet, but it (like many books) is connecting dots for me that I hadn't connected before. Reading through it, I wish I'd seen some of its content when I was younger, it could've made some things easier over the years.
[0] https://ocw.mit.edu/courses/sloan-school-of-management/15-87...
[1] https://leanpub.com/b/generalsystemsthinkerbundle
I write down every issue - everything from user interface issues to bugs, and then delete them as they get fixed. I can then use that file to measure code debt. The file is saved in version control, so no fancy stuff, but I can run statistics and compare code growth and growth of issues.
In my experience you want to keep this file small. On a current project I've estimated that it will take five years to clear this file <grin>. This file seem to grow exponentially with new features. And I think at some point there's a point of no return - where you can't possible fix all issues ...
Of course, having a measurement might help shine a spotlight on how things are getting worse, but I've never actually seen this happen. Perhaps others have a different experience?