Does Software Understand Complexity?(michaelfeathers.silvrback.com)
michaelfeathers.silvrback.com
Does Software Understand Complexity?
https://michaelfeathers.silvrback.com/does-software-understand-complexity
7 comments
Do you have a link to your thesis?
Here's the ref[0], I lost both the tex source and pdf output in a double crash (backup and laptop basically simultaneously) and don't even have access to the full text anymore (oh the irony). My work on execution flow modelling wasn't referenced and is forever gone in that crash though. Bummer.
It's only a master of advanced studies (Diplôme d'Etudes Approfondies [1]) so it's way shorter than a doctoral thesis. Also, the thing is probably half-assed in some way, as there was some, dare I say, "lack of support" on challenging ideas which was highly non-motivational on getting a proper paper out (The moment I talked about leveraging AI/NN in component supervision I was outcast as a sci-fi author). It doesn't cover half of my work (had to cut it down to only the core part, enough to get the gist of things, but that's only the tip of the iceberg). And it's in French.
If you (or anyone else) can ever get the full text please forward it back to me so I can laugh at my arrogant young self!
[0]: https://hal.inria.fr/inria-00000787
[1]: https://en.wikipedia.org/wiki/Master_of_Advanced_Studies#Fra...
It's only a master of advanced studies (Diplôme d'Etudes Approfondies [1]) so it's way shorter than a doctoral thesis. Also, the thing is probably half-assed in some way, as there was some, dare I say, "lack of support" on challenging ideas which was highly non-motivational on getting a proper paper out (The moment I talked about leveraging AI/NN in component supervision I was outcast as a sci-fi author). It doesn't cover half of my work (had to cut it down to only the core part, enough to get the gist of things, but that's only the tip of the iceberg). And it's in French.
If you (or anyone else) can ever get the full text please forward it back to me so I can laugh at my arrogant young self!
[0]: https://hal.inria.fr/inria-00000787
[1]: https://en.wikipedia.org/wiki/Master_of_Advanced_Studies#Fra...
>I’ve long thought that as we add code and dependencies to systems we approach biological levels of complexity.
According to Taleb: complexity is good for system reliability in the long term. (As well as redundancy in every aspect like head count, components count, technology stacks used in the same system) to mimic biological complexity.
But I strive to make everything as simple as possible, remove unnecessary components, etc.
I still cant understand contexts where complex is better than simple
According to Taleb: complexity is good for system reliability in the long term. (As well as redundancy in every aspect like head count, components count, technology stacks used in the same system) to mimic biological complexity.
But I strive to make everything as simple as possible, remove unnecessary components, etc.
I still cant understand contexts where complex is better than simple
Simple is better where you need to maintain control, because then a human being can understand the entire system and its behavior becomes deterministic from our perspective. Complexity--practically by definition--means that the system's behavior becomes harder to predict, when software development is (in most cases) explicitly focused on delivering expected behavior. So for most practical software development purposes, simplicity is better.
But complexity can also be emergent from the aggregation of many small, simple components. E.g. the internet, which grew out of some relatively simple protocols connecting a handful of computers into the globe-spanning organism that it is today. So you may not design a complex system as a software developer, but your simple system may become a component in a larger, complex system. And that system will be more robust than the individual, simple components are. Such as the replacement of HTTP with HTTPS at the protocol level or fiber replacing copper in the physical system of the internet. Those simple systems have (or rather will soon) perish, while the more complex system that emerged from them survives.
But complexity can also be emergent from the aggregation of many small, simple components. E.g. the internet, which grew out of some relatively simple protocols connecting a handful of computers into the globe-spanning organism that it is today. So you may not design a complex system as a software developer, but your simple system may become a component in a larger, complex system. And that system will be more robust than the individual, simple components are. Such as the replacement of HTTP with HTTPS at the protocol level or fiber replacing copper in the physical system of the internet. Those simple systems have (or rather will soon) perish, while the more complex system that emerged from them survives.
> Such as the replacement of HTTP with HTTPS at the protocol level or fiber replacing copper in the physical system of the internet. Those simple systems have (or rather will soon) perish, while the more complex system that emerged from them survives.
Funny you should say that w.r.t. to reliability.
I just got back from vacation today and have just had 2 SSL-related issues I needed to solve, not to mention one related with dependencies on smaller components, which when updated broke my chain of software.
All these would have worked more reliably 1. without HTTPS and 2. if everything were packaged together :)
In short, the simpler solution would clearly be more reliable here.
Funny you should say that w.r.t. to reliability.
I just got back from vacation today and have just had 2 SSL-related issues I needed to solve, not to mention one related with dependencies on smaller components, which when updated broke my chain of software.
All these would have worked more reliably 1. without HTTPS and 2. if everything were packaged together :)
In short, the simpler solution would clearly be more reliable here.
Sure, but that's your experience as an individual developer on a small component of the system. If we're looking at the health of the system _as a whole_ (of which you, and the work you're doing to solve these issues, are a part), it's completely resilient to this change.
> Those simple systems have (or rather will soon) perish, while the more complex system that emerged from them survives.
I've wondered whether that is why we find ourselves surrounded by complex software. Legacy code is code that provides more value than the cost of its replacement. Bad code endures when that cost goes up.
I've wondered whether that is why we find ourselves surrounded by complex software. Legacy code is code that provides more value than the cost of its replacement. Bad code endures when that cost goes up.
> But I strive to make everything as simple as possible, remove unnecessary components, etc. I still cant understand contexts where complex is better than simple
If there are some, I'm not really aware of them. Systems we create should be human-manageable. The thing I find interesting is that, even with decades of attempts at better practice, we still end up with complexity that makes our work difficult.
If there are some, I'm not really aware of them. Systems we create should be human-manageable. The thing I find interesting is that, even with decades of attempts at better practice, we still end up with complexity that makes our work difficult.
Simplicity is especially important for developers because it is perhaps the most important determinant for the cost of reviewing, understanding and maintaining code.
But this cost association might be sort of special for systems built with code. Or systems that are built for handling discrete use cases rather than for continuous or approximate use cases.
This is quite speculative and a bit out of depth but: For some systems it might make more sense to duplicate a lot of functionality and use redundancy to achieve stability. Thinking of evolutionary systems for example.
But this cost association might be sort of special for systems built with code. Or systems that are built for handling discrete use cases rather than for continuous or approximate use cases.
This is quite speculative and a bit out of depth but: For some systems it might make more sense to duplicate a lot of functionality and use redundancy to achieve stability. Thinking of evolutionary systems for example.
I had a similar experience this past week at the ISSS conference in Corvallis.
I'm applying to the systems science PhD program at PSU this year and was amazed at the both the depth and breadth of the systems research and discussion to be had, but also noted the lack of software developers or computer/information systems discussion.
I'm looking forward to being able to bring some of my expertise to this field, while also mining the knowledge and concepts for use in better software systems design.
I'm going to definitely look into this conference for next year, though hopefully they won't be running the same week as the ISSS conference again.
I'm applying to the systems science PhD program at PSU this year and was amazed at the both the depth and breadth of the systems research and discussion to be had, but also noted the lack of software developers or computer/information systems discussion.
I'm looking forward to being able to bring some of my expertise to this field, while also mining the knowledge and concepts for use in better software systems design.
I'm going to definitely look into this conference for next year, though hopefully they won't be running the same week as the ISSS conference again.
Yeah, I would imagine that at least some subset of artificial intelligence theorists would be throwing anything they could at the problem of the management of complex systems. Even if they didn't make immediate progress, it would yield a benchmark to judge future effort by.
I'm not sure whether it is finalized yet but the conference may go biennial, skipping next year. That's just what I've heard indirectly. If you're interested, definitely watch for it.
I used to walk past the Harder House on SW 10th and Market every day, where I think the systems PhD program sits. They have these fascinating network diagrams stuck up in the windows, and I was always curious what they were...
Yep, that's the place... if it's the one I'm thinking of, then I think it's a diagram of a search through a set of potential models during a RA (Reconstructability Analysis) of 3 independent variables with 1 dependent variable.
It's Dr. Marty Zwick's favorite picture since he's the one who teaches DMM (Discrete Multivariate Modeling) which deals with RA. I haven't taken it yet, but he was at ISSS and gave a plenary on RA, as well as chaired a full day of RA related presentations.
Here is some more information, which also has the diagram I think you're referencing:
https://www.pdx.edu/sysc/research-discrete-multivariate-mode...
It's Dr. Marty Zwick's favorite picture since he's the one who teaches DMM (Discrete Multivariate Modeling) which deals with RA. I haven't taken it yet, but he was at ISSS and gave a plenary on RA, as well as chaired a full day of RA related presentations.
Here is some more information, which also has the diagram I think you're referencing:
https://www.pdx.edu/sysc/research-discrete-multivariate-mode...
Software systems are the perfect data generating processes. They are often compositional and yet complex because of simple interactions blowing things up. And we have effectively infinite data so it's a good place to get started. It would be amazing if an AI could reverse engineer a backend of an app just based on using the product and maybe making some scaling assumptions. But we're not that good with loss functions yet.
Maybe someday we can have an AI watch our log streams and tell us the ideal architecture for our systems. Then the trick will be to generate fake log data for what we want to happen in general. Perhaps we can just keep it coarse grained and say the AI can fill in the rest. The future of systems architecture!
Even more interesting is Conway's law. What if an AI could tell you how best to organize your company?
Maybe someday we can have an AI watch our log streams and tell us the ideal architecture for our systems. Then the trick will be to generate fake log data for what we want to happen in general. Perhaps we can just keep it coarse grained and say the AI can fill in the rest. The future of systems architecture!
Even more interesting is Conway's law. What if an AI could tell you how best to organize your company?
What if an AI could tell you how best to organize your company?
Companies organize around office politics, in my experience efficiency is quite low on their priority list, especially in big enterprises.
Companies organize around office politics, in my experience efficiency is quite low on their priority list, especially in big enterprises.
Increased efficiency comes at the cost of lowered control. It took me awhile to wrap my brain around the mindset. If the company takes me up on my offer to rebuild their infrastructure at my current pay rate rather than hire a consultant to do it for $X00,000, note that hundreds of thousands of dollars is quite cheap for this kind of consultation, that presents an efficiency improvement at the cost of control. The CFO wants Magento and I'm not interested in coding Magento.
It's more rational from a business standpoint for it to introduce often-extreme efficiency penalties in order to achieve the CFO's will. This is an actual scenario that happened to me by the way. I couldn't analyze it with quite this amount of granularity until recently.
Everybody in the company that could have formed an opinion agreed with me about the efficiency gains of doing it my way. But what I was wanting was politically impossible. So me and the company parted ways.
Since then Magento's new version provides an API layer that makes it way way easier for companies with competent technical leadership to have their programmers use disparate skillsets and still get along. I can learn Magento well enough to work on it on the backend, and maintain the Rails frontend that I would have wanted to work on. If I had saw this coming at the time, I'd have stuck around and helped manage the replatforming. But all I was looking at was the extra $X0,000 I was going to make elsewhere.
It's more rational from a business standpoint for it to introduce often-extreme efficiency penalties in order to achieve the CFO's will. This is an actual scenario that happened to me by the way. I couldn't analyze it with quite this amount of granularity until recently.
Everybody in the company that could have formed an opinion agreed with me about the efficiency gains of doing it my way. But what I was wanting was politically impossible. So me and the company parted ways.
Since then Magento's new version provides an API layer that makes it way way easier for companies with competent technical leadership to have their programmers use disparate skillsets and still get along. I can learn Magento well enough to work on it on the backend, and maintain the Rails frontend that I would have wanted to work on. If I had saw this coming at the time, I'd have stuck around and helped manage the replatforming. But all I was looking at was the extra $X0,000 I was going to make elsewhere.
That's what we're aiming to do at https://matrix.ai
super interesting topic. I'm currently also looking more and more into how UNIX and the network layers manage complexity in abstract terms. Would've loved if the article would've been a bit more specific in terms of what takeaways from the conference might be of interest for software.
I may get a chance to write up more over the next week. If I do, it will be on the same blog.
Can anyone here recommend a good introductory textbook for control theory?
1st sentence: “This past week I was at a ¿conference?...”
(Stories about the mystery conference)
2nd sentence of 2nd paragraph: “I was at ICCS2018, a complex systems conference.“
Why is this normal now? It’s ruining the internet. Everything I read lately is wound up into a desperate suspense-arc.
I understand not everybody has liberal arts degrees (and definitely not this person) but that might be all the more reason to not make such an attempt.
Introductions come first. It’s a lot like meeting a person for the first time. Creativity is not a bad thing, but if you’re sacrificing respect in the process, you’re blowing it.
(Stories about the mystery conference)
2nd sentence of 2nd paragraph: “I was at ICCS2018, a complex systems conference.“
Why is this normal now? It’s ruining the internet. Everything I read lately is wound up into a desperate suspense-arc.
I understand not everybody has liberal arts degrees (and definitely not this person) but that might be all the more reason to not make such an attempt.
Introductions come first. It’s a lot like meeting a person for the first time. Creativity is not a bad thing, but if you’re sacrificing respect in the process, you’re blowing it.
This was all over the place behind the core theme of my CS master's degree, for which I happened to also have background in both mechanical and electrical engineering. There are a lot of things that felt quite natural to me but that were proving elusive and challenging to people I worked with who lacked an equally diverse background by being exclusively focused on CS.
Two points I raised were designing with, leveraging, and modelling feedback loops in component-based software using control theory (application was on auto-adaptative AV streaming across a network of varying quality† but the overarching theme was largely generalisable); and modelling execution flow using laws from electrical engineering (using high level concepts such as power potential and current intensity as well as qualifying code paths as being resistive, capacitive, and inductive) as an alternative approach to complexity and optimisation.
The resistance to cross-domain mixing was surprisingly high, entrenched thought processes are a remarkably deceitful thing, even when faced with provable results.
† Something I would find out to be implemented a couple of years later with zero-delay Xbox Live movie streaming.