Software Maintenance by Skeleton Crew Is an Anti-Pattern(18f.gsa.gov)
18f.gsa.gov
Software Maintenance by Skeleton Crew Is an Anti-Pattern
https://18f.gsa.gov/2016/02/23/software-maintenance-is-an-anti-pattern/
12 comments
Steven -- I applaud your team's approach. From my two and a half years in government I have rarely run into projects like you describe. We have lots of legacy systems, typically built without automated tests. Often a vendor is hired to build a "complete" piece of software, then given a meager budget to "maintain" it.
Thanks.
We have a lot of that issue of "hire contractors to build, then use FTEs to maintain". I try to push back where I can because contractors that know up-front that they won't have to support the system are often not inclined to either follow standards (which are less exciting, I know, but make maintenance more predictable) and/or not be concerned about avoiding excessive technical debt up front.
We have a lot of that issue of "hire contractors to build, then use FTEs to maintain". I try to push back where I can because contractors that know up-front that they won't have to support the system are often not inclined to either follow standards (which are less exciting, I know, but make maintenance more predictable) and/or not be concerned about avoiding excessive technical debt up front.
It may be worth noting that some government-sponsored software is intended to get dropped as soon as version 1 gets released. Of course, this is a small fraction of government-produced software.
This happens for a variety of reasons. Often funds are allocated specifically to develop what industry calls a minimum viable product-- with the understanding that if the project is deemed successful, the research group can seek further funding to improve the tool.
There is also a model, in government research funding, that says the government's role is mainly to point the way, or to open up markets. In this model, if version 1 of the software proves itself useful, then an industry partner can step in and it take over. However, if nobody shows interest, then the government should _not_ sustain the effort, because the lack of interest implies a lack of utility to the public.
For example, the OpenEIS project [0] was funded specifically to produce "reference code" that a company can use to check the output of a professional implementation. The reference code is not at all intended to be fast or robust. It's intended to demonstrate the algorithms, and to document the expected output for a variety of reasonable inputs. As a programmer on the project, I knew from the start that my work would end with the first deliverable.
[0] http://eis.lbl.gov/openeis.html
This happens for a variety of reasons. Often funds are allocated specifically to develop what industry calls a minimum viable product-- with the understanding that if the project is deemed successful, the research group can seek further funding to improve the tool.
There is also a model, in government research funding, that says the government's role is mainly to point the way, or to open up markets. In this model, if version 1 of the software proves itself useful, then an industry partner can step in and it take over. However, if nobody shows interest, then the government should _not_ sustain the effort, because the lack of interest implies a lack of utility to the public.
For example, the OpenEIS project [0] was funded specifically to produce "reference code" that a company can use to check the output of a professional implementation. The reference code is not at all intended to be fast or robust. It's intended to demonstrate the algorithms, and to document the expected output for a variety of reasonable inputs. As a programmer on the project, I knew from the start that my work would end with the first deliverable.
[0] http://eis.lbl.gov/openeis.html
Without any concrete evidence or cases, the general description of a problem in this article is not very convincing. Software maintenance itself is not an anti-pattern. Instead, what is described is a situation of product/project mismanagement.
But I think this article is really addressing public facing web platforms instead of all "software," which the US government has only really experimented with during the last administration, so some of the sentiments of this article, in the context of government, may be true even though it reads false.
Beginning with the analogy to building construction: A skyscraper isn't complete when just the floors and frame are put in place, and the building team doesn't let the builders go at that point. When it is opened up, the maintenance staff have their own job, but it's nothing like what renovators might have to do years later.
> ... software is not considered complete when it's first released.
> Governments often use two anti-patterns... (2) moving to reduce
sustaining staff too early;
> To keep teams from constantly plugging leaks while missing
the reason the boat is sinking, we need to apply the
principles of agile development and user centered design
But I think this article is really addressing public facing web platforms instead of all "software," which the US government has only really experimented with during the last administration, so some of the sentiments of this article, in the context of government, may be true even though it reads false.
Beginning with the analogy to building construction: A skyscraper isn't complete when just the floors and frame are put in place, and the building team doesn't let the builders go at that point. When it is opened up, the maintenance staff have their own job, but it's nothing like what renovators might have to do years later.
> ... software is not considered complete when it's first released.
I don't know where this assumption comes from.
My anecdata, doing IT support for about 1,000 small-business sites: it's customary to only adopt new Microsoft releases after SP1 has been released. Our customers sometimes even explicitly instruct us so.
I don't know where this assumption comes from.
My anecdata, doing IT support for about 1,000 small-business sites: it's customary to only adopt new Microsoft releases after SP1 has been released. Our customers sometimes even explicitly instruct us so.
While this is changing, in game development the finished product ships on read-only media and without patches or online support, that is all the customer is going to get.
Now there's a prevalence of day one patches but it's pretty lame to see developers waiting till the weeks between pressing the disc and it appearing on shelves to fix standing issues.
In the case of M$ software that's a security or support issue not the absence of core features.
Now there's a prevalence of day one patches but it's pretty lame to see developers waiting till the weeks between pressing the disc and it appearing on shelves to fix standing issues.
In the case of M$ software that's a security or support issue not the absence of core features.
Already in 2013 92% of PC game sales involved no physical media [http://gamerant.com/pc-digital-physical-sales-comparison-201...]. This enables the day-1-habit, of course, but it also mitigates the problems - there aren't any shelves on which unpatched versions of the game keep sitting after launch.
Console is lagging behind for this.
In fact, the reason Server 2003 got an extra year of support is that they waited until after Vista SP1 to do Server 2008.
"... software is not considered complete when it's first released."
The author was probably meaning "finished" instead of "complete", which gives a different meaning more in line with the article.
The author was probably meaning "finished" instead of "complete", which gives a different meaning more in line with the article.
Indeed, what is "complete" anyway? Is it the opposite of "incomplete" (all requirements are met)? Or does it mean "not requiring further extension" (no new requirements will emerge)?
This seems to assume that all "private industry" does things in a way different than how government does.
Different kinds of projects have different needs for upfront design. One can quickly ship and iterate on an online dating app. One cannot launch a new space shuttle every two weeks and keep trying until the crew survives the mission.
Different kinds of projects have different needs for upfront design. One can quickly ship and iterate on an online dating app. One cannot launch a new space shuttle every two weeks and keep trying until the crew survives the mission.
>This seems to assume that all "private industry" does things in a way different than how government does.
It does. A skeleton crew would be abandonware.
It does. A skeleton crew would be abandonware.
I don't understand your reply. I'm familiar with private industry projects producing high-risk projects requiring lots of long term up front investment in capital and time and private industry projects working low-risk efforts where using the customers as testers is perfectly fine. Likewise there are small units of government entities that can iterate on low-risk projects in addition to the big infrastructure projects that larger government organizations work on.
I think the OP is painting with too broad a brush when talking about "private industry".
I think the OP is painting with too broad a brush when talking about "private industry".
I responded to this question, but now I see that I was missing your point. (It doesn't help that the HN title changed to something other than the article title after it was posted.)
There are plenty of private industry shops maintaining software on a skeleton crew that needs no more features or a bigger market. Again, this article is assuming that all private industry only serves a growing market. There's plenty of software already written and being profitably maintained by skeleton crews to happy customers.
There are plenty of private industry shops maintaining software on a skeleton crew that needs no more features or a bigger market. Again, this article is assuming that all private industry only serves a growing market. There's plenty of software already written and being profitably maintained by skeleton crews to happy customers.
Or a startup.
I've worked O&M on software in the defense industry. Never again.
On my current project, when development is done and we're officially transitioned over to "O&M", my employer either gives me a new project to work on or I'm leaving to do something else.
Oh, and I've come to the conclusion that civil servants are not capable of grasping any development methodology other than 'waterfall.' Trying to do 'agile' is an exercise in frustration. The bureaucrats say they want agile (buzzword!), but they really don't. Eventually you settle on doing 5-week 'mini-waterfalls' and calling it 'agile'. :/
On my current project, when development is done and we're officially transitioned over to "O&M", my employer either gives me a new project to work on or I'm leaving to do something else.
Oh, and I've come to the conclusion that civil servants are not capable of grasping any development methodology other than 'waterfall.' Trying to do 'agile' is an exercise in frustration. The bureaucrats say they want agile (buzzword!), but they really don't. Eventually you settle on doing 5-week 'mini-waterfalls' and calling it 'agile'. :/
> Oh, and I've come to the conclusion that civil servants are not capable of grasping any development methodology other than 'waterfall.'
Civil servants are quite capable of understanding other development methodology. Civil servants actually involved in software development are often constrained by legal and regulatory regimes which no one in a substantively technical role has significant input into, which mandate procurement, risk-management, and approval processes which essentially mandate a high-level flow very similar to a completely non-iterative, cartoonish mockery of waterfall, and to the extent that they can attempt to implement more effective processes, they must do so within those constraints.
Civil servants are quite capable of understanding other development methodology. Civil servants actually involved in software development are often constrained by legal and regulatory regimes which no one in a substantively technical role has significant input into, which mandate procurement, risk-management, and approval processes which essentially mandate a high-level flow very similar to a completely non-iterative, cartoonish mockery of waterfall, and to the extent that they can attempt to implement more effective processes, they must do so within those constraints.
I see it in private industry too, I just worked in 2-week sprints on something only to discover that all our "feedback" for the past six months was actually from (surprise!) the managers in the pipeline, who hadn't actually enabled the feature to be seen by real end-users.
Among many possible causes, two that come to mind:
1. Managers who (despite lip-service to the idea) refuse to take any incremental risks looking bad with negative feedback each sprint, and would rather risk one big failure-event.
2. Companies which have either very-convoluted procedures or who try to treat end-users as cheap interchangeable human-robots... Which means no change can ever be rolled out without exhaustive documentation and training plans. Therefore all changes have to be "big enough" to be worth it.
Among many possible causes, two that come to mind:
1. Managers who (despite lip-service to the idea) refuse to take any incremental risks looking bad with negative feedback each sprint, and would rather risk one big failure-event.
2. Companies which have either very-convoluted procedures or who try to treat end-users as cheap interchangeable human-robots... Which means no change can ever be rolled out without exhaustive documentation and training plans. Therefore all changes have to be "big enough" to be worth it.
It's not just civil servants who can't understand agile. Most non-software people can't. Which is totally normal and expected if you think about it: the agile movement started in software. You can't build a house or a plane using agile. It has to be waterfall.
> It's not just civil servants who can't understand agile. Most non-software people can't. Which is totally normal and expected if you think about it: the agile movement started in software.
Agile is (to the extent it has concrete substance) very similar to Lean, which (while it has come to software) actually started in manufacturing and engineering of tangible products. The difference between the two is mainly that Lean has a much stronger culture (and has thus also produced a lot more tools) around validation of methods through measurement, and thus is a lot more woo-resistant than Agile, which has proven to be decidedly prone to devolving in exactly the same kind of one-size fits all, consultant-pushed, top-down methodologies without good feedback on what works in the particular environment that the Agile Manifesto was a response to.
> You can't build a house or a plane using agile. It has to be waterfall.
"Agile" and "waterfall" aren't opposed, agile is opposed to the idea that any one methodology -- Scrum as much as Waterfall -- can be selected as right for a team without reference to the particular team and context in which they are working. To the extent that it works in a particular context, waterfall can be Agile.
Agile is (to the extent it has concrete substance) very similar to Lean, which (while it has come to software) actually started in manufacturing and engineering of tangible products. The difference between the two is mainly that Lean has a much stronger culture (and has thus also produced a lot more tools) around validation of methods through measurement, and thus is a lot more woo-resistant than Agile, which has proven to be decidedly prone to devolving in exactly the same kind of one-size fits all, consultant-pushed, top-down methodologies without good feedback on what works in the particular environment that the Agile Manifesto was a response to.
> You can't build a house or a plane using agile. It has to be waterfall.
"Agile" and "waterfall" aren't opposed, agile is opposed to the idea that any one methodology -- Scrum as much as Waterfall -- can be selected as right for a team without reference to the particular team and context in which they are working. To the extent that it works in a particular context, waterfall can be Agile.
Agile is the idea that requirements are always in flux, and therefore it's pointless to spend time upfront gathering requirements and writing specs and following those specs since there's a good chance they will have become irrelevant by the time the product ships. So agile says "just get something, anything, out the door as soon as possible, receive feedback, then keep revising."
Waterfall is the opposite approach, where you have a formal requirements gathering phase, then you commit everything into design docs, and follow those docs with as few changes as possible. It's similar to how a house is built: first you check your local zoning laws, then have an architect draw the plans, then dig a hole and build a foundation, then build the skeleton, the walls, etc. You don't tell the client, "hey, we just dug the hole for the foundation, come live in it for a few weeks and give us feedback so we know what part to build next."
Waterfall is the opposite approach, where you have a formal requirements gathering phase, then you commit everything into design docs, and follow those docs with as few changes as possible. It's similar to how a house is built: first you check your local zoning laws, then have an architect draw the plans, then dig a hole and build a foundation, then build the skeleton, the walls, etc. You don't tell the client, "hey, we just dug the hole for the foundation, come live in it for a few weeks and give us feedback so we know what part to build next."
> You can't build a house or a plane using agile. It has to be waterfall.
If you can build a car with it,[0] I'm sure you can build a plane.
[0] http://wikispeed.org/
If you can build a car with it,[0] I'm sure you can build a plane.
[0] http://wikispeed.org/
This could be applied to any software that is developed as a product (in the sense of a piece of software to sell to multiple players in a market).
The article talks about the demand/market for the software increasing after the release, until it reaches a peak, then tails off.
Agile uses the term 'Product Owner', but if you're developing an intranet for Acme Inc, is it really a product? Once the new intranet site is released, is Acme Inc suddenly going to hire a lot more people, increasing the "market"? Will new features generate more "profit" for the intranet?
I wouldn't describe this as Gov vs Private Industry, but as software products vs bespoke software development. And for bespoke software development, private industry often has the same approach to maintenance: once it's done, it's time to wind down the team.
The article talks about the demand/market for the software increasing after the release, until it reaches a peak, then tails off.
Agile uses the term 'Product Owner', but if you're developing an intranet for Acme Inc, is it really a product? Once the new intranet site is released, is Acme Inc suddenly going to hire a lot more people, increasing the "market"? Will new features generate more "profit" for the intranet?
I wouldn't describe this as Gov vs Private Industry, but as software products vs bespoke software development. And for bespoke software development, private industry often has the same approach to maintenance: once it's done, it's time to wind down the team.
> Agile uses the term 'Product Owner', but if you're developing an intranet for Acme Inc, is it really a product?
Approaching this from a purely semantic perspective: yes, a product is something that is produced as a result of a process. Not "in the sense of a piece of software to sell to multiple players in a market", but that's a completely arbitrary distinction that you drew yourself, so it isn't really an argument against calling someone a "product owner" of a single customer product.
> Once the new intranet site is released, is Acme Inc suddenly going to hire a lot more people, increasing the "market"? Will new features generate more "profit" for the intranet?
Maybe Acme Inc find that the product that they have paid for have benefits to productivity, but are willing to pay for continued development to refine it further?
Having worked on a lot of "one-off" type projects I think that it is important to maintain software gracefully. Existing customers will ask us to roll out new features on short notice, and having built the product in a way that lets us adjust and add to it easily enables us to sign deals at competitive prices.
That's not to say that this always happens, but I am less suspicious of this as an ideal.
Approaching this from a purely semantic perspective: yes, a product is something that is produced as a result of a process. Not "in the sense of a piece of software to sell to multiple players in a market", but that's a completely arbitrary distinction that you drew yourself, so it isn't really an argument against calling someone a "product owner" of a single customer product.
> Once the new intranet site is released, is Acme Inc suddenly going to hire a lot more people, increasing the "market"? Will new features generate more "profit" for the intranet?
Maybe Acme Inc find that the product that they have paid for have benefits to productivity, but are willing to pay for continued development to refine it further?
Having worked on a lot of "one-off" type projects I think that it is important to maintain software gracefully. Existing customers will ask us to roll out new features on short notice, and having built the product in a way that lets us adjust and add to it easily enables us to sign deals at competitive prices.
That's not to say that this always happens, but I am less suspicious of this as an ideal.
Maybe the problem is that development is decentralized and done off-site? For each each website, or component is built by separate, often lucrative contract. On one hand developers re-invent the wheel knowing that they financially benifit from the bloat, on the other hand they feel little responsibility for their often weighty work.
Can we do something crazy like centralize our software development or have these administrative positions electable? If you can run for town dog catcher, maybe that asshole who made the WordPress site should also stand for office!
Can we do something crazy like centralize our software development or have these administrative positions electable? If you can run for town dog catcher, maybe that asshole who made the WordPress site should also stand for office!
I don't see the point of shoehorning technical concepts onto managerial problems. All it's doing is confusing the point. It's like the author just learned about the concept and now wants to re-imagine everything in terms of that concept.
Peculiar that the author makes very strong statements without any proof of research findings.
If users have put up with bugs for this long, why should the system be improved?
Either fix the bugs quickly, or don't fix them at all, and move on.
Either fix the bugs quickly, or don't fix them at all, and move on.
Or the "continuous release" model of browser-based and app software. Tehy seem to updated automatically nearly every week.
This title is misleading, should be closer to "Government frequently adopts anti-patterns when maintaining software" Something more constrained to what the article is actually about rather that the global catch all "software maintenance".
I spent over 20 years in the software industry and I've never seen anyone describe software as in "maintenance" except when staffed by a skeleton crew who does not do user research or anything forward thinking. That's ok, I guess, in industry, not ok in government when a fixed size market is not a business indicator, but a fact-of-life.
Indeed, that's the headline on the article itself, but it does seem misleading. I think "Software Maintenance by Skeleton Crew is an Anti-pattern" would be a better but still concise headline.
Updated
I agree that the title is misleading, but I'd clarify it to "Software development as typically practiced in industry is an anti-pattern." (Maybe it also applies to government; I don't know.) It seems fairly typical in my experience for the old thing to be continually patched over instead of fixed, awaiting replacement by the new thing that is never quite ready.
In general, "the new thing" doesn't solve problems that patches fix. For a web platform especially, it sounds like expecting a new engine from a new coat of paint.
A lot of it is that consistently people design things on the assumption that 80% of labor will be in the first release, rather than the 80% going into maintaiance which is what happens most of the time.
This is closely linked to the idea that least cost development is a matter of doing every individual action 'cheaply' and pretending you are not going to screw things up. If you want least cost actually least risk is a good approximation.
This is closely linked to the idea that least cost development is a matter of doing every individual action 'cheaply' and pretending you are not going to screw things up. If you want least cost actually least risk is a good approximation.
I think the article is using software maintenance as a kind of inside-government term-of-art for the maintenance portion of a waterfall-style project.
Undoubtedly, the audience of this article is software developers and managers of software projects within governemnt. Therefore, although the title doesn't convey the article's message to outsiders, it might not actually be misleading.
Undoubtedly, the audience of this article is software developers and managers of software projects within governemnt. Therefore, although the title doesn't convey the article's message to outsiders, it might not actually be misleading.
[deleted]
At some point though, the team will decrease in size and the software will be considered "complete for now". Generally, that software will continue to be "on life support" (I guess you could say that?) for the next 5 - 20 years. That isn't a bad thing. Some of the projects I maintain have code from the mid-90s that I'm still making incremental updates to. Sure, I'd like to rewrite it in something newer, but it isn't about me, it isn't about fancy shiny. It's about the fact the system as it exists is proven and solid and meets 90% of the customers stated need. "Whack-a-mole" changes are just a sign of poor testing, poor architecture, or lazy developers. All of our teams here make incremental changes on aging software and generally don't experience major regressions except when it's time to do the big rewrites (which are usually customer-initiated when they bring a raft of changes and some money). When existing technologies are so obsolete as to be unsupportable, then we (IT) will initiate a rewrite as well, but hopefully with incremental upgrades we can keep a product running for as long as the customer likes without spending any more money than they want to spend.