A $300M IT flop(fcw.com)
fcw.com
A $300M IT flop
http://fcw.com/articles/2014/07/24/it-flop-at-ssa.aspx
17 comments
>'(a) obviously tailored to the capabilities of a single firm so there was no way to win unless you were them'
For what it's worth, this isn't necessarily corruption at work.
As I've related on HN before, the procurement arms of these agencies are often woefully unequipped for handling anything IT related. It's typically a process made for sourcing completely interchangeable, relatively simple supplies or services seeing terribly complex software or systems dumped into them - not 'shoe-horned' as that would imply an attempt at fitting - simply dumped so that requirements spill on the floor and everything gets jumbled up onto the existing conventions.
Given that, providing an agnostic SOW is a great way to end up with stuck with a lowball bid which makes the most self-serving, word-splitting assumptions possible to stick you with something utterly apart from what you need.
>'(b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time.'
Related to what I said above - these requirements are not as hard and fast as they might seem. Writing a defensively worded response, getting your foot in the door to talk with the people who actual request and implement is not a terrible way to start doing real work.
>'Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow.'
Right.
It's not unlike any sort of courtship, business or otherwise. You can't just ask outright, you have to make friends and hear the information gush over lunch or volunteer some no-fee consulting and take notes.
Big vendors and VARs have people who do this nearly full-time so it's certainly much harder for a small company, but not impossible.
>'We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.'
Something to be aware of, when dealing with a Lockheed terms like 'adjustments, feature additions' are euphemisms for 'one million dollars'. If you're pitching these possibilities and expecting/willing to make them to earn the business then it might pay to be explicit about it.
For what it's worth, this isn't necessarily corruption at work.
As I've related on HN before, the procurement arms of these agencies are often woefully unequipped for handling anything IT related. It's typically a process made for sourcing completely interchangeable, relatively simple supplies or services seeing terribly complex software or systems dumped into them - not 'shoe-horned' as that would imply an attempt at fitting - simply dumped so that requirements spill on the floor and everything gets jumbled up onto the existing conventions.
Given that, providing an agnostic SOW is a great way to end up with stuck with a lowball bid which makes the most self-serving, word-splitting assumptions possible to stick you with something utterly apart from what you need.
>'(b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time.'
Related to what I said above - these requirements are not as hard and fast as they might seem. Writing a defensively worded response, getting your foot in the door to talk with the people who actual request and implement is not a terrible way to start doing real work.
>'Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow.'
Right.
It's not unlike any sort of courtship, business or otherwise. You can't just ask outright, you have to make friends and hear the information gush over lunch or volunteer some no-fee consulting and take notes.
Big vendors and VARs have people who do this nearly full-time so it's certainly much harder for a small company, but not impossible.
>'We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.'
Something to be aware of, when dealing with a Lockheed terms like 'adjustments, feature additions' are euphemisms for 'one million dollars'. If you're pitching these possibilities and expecting/willing to make them to earn the business then it might pay to be explicit about it.
That's exactly right. The staff organizing the project for the government gets advice (bad advice) on the technologies that it should use from approved development firms then make using that technology a strict requirement. These firms are some of the few that have developers that use that technology and win the bid, everybody except the hired firm loses.
Any small competent development team can built almost anything imaginable with 6 years of time, proper funding, and without unnecessary restrictions.
$300 million to develop anything is insane. I can't imagine what happens when the check is cut, "let's hire 100 programmers, 10 managers, 5 accountants, 10 support staff and they'll figure it out."
Maybe there needs to be some kind of "insurance" company that oversees the project and has to pay back a portion of the money if the project fails. I'm not sure if that's reasonable, just an idea. If the project starts going off-track then they start letting people go and find competent replacements.
Plan projects out, have milestones, goals, backup plans. If they spend 6 years and $300 million dollars on a computer system and fail then they were just winging it from the beginning. The managers responsible should be identified and go on a blacklist. Right now it's either "oh well, hopefully we'll do better next time".
Any small competent development team can built almost anything imaginable with 6 years of time, proper funding, and without unnecessary restrictions.
$300 million to develop anything is insane. I can't imagine what happens when the check is cut, "let's hire 100 programmers, 10 managers, 5 accountants, 10 support staff and they'll figure it out."
Maybe there needs to be some kind of "insurance" company that oversees the project and has to pay back a portion of the money if the project fails. I'm not sure if that's reasonable, just an idea. If the project starts going off-track then they start letting people go and find competent replacements.
Plan projects out, have milestones, goals, backup plans. If they spend 6 years and $300 million dollars on a computer system and fail then they were just winging it from the beginning. The managers responsible should be identified and go on a blacklist. Right now it's either "oh well, hopefully we'll do better next time".
Startups certainly don't help themselves in this regard, though. The sheer number that shut down, get acquired etc. means that I don't totally blame government bodies that choose large, established companies.
Maybe there is an opportunity for a private company to be the middle-man between government projects and small competent development teams. To ensure they meet the requirements and to provide real advice to the government agencies that need projects built.
Your sort of talking about FFRDC's. Part of their role as a non-profit , non-government agency is to "Consult" with the government.
I worked at one for a while, we were held in very high regard amongst the Government. When they doled out new contracts they basically asked us our opinion for which company to choose, based on past performance. We would also review other contractors work and give it the thumbs up / thumbs down.
This was on a much smaller scale.
http://en.wikipedia.org/wiki/List_of_federally_funded_resear...
I worked at one for a while, we were held in very high regard amongst the Government. When they doled out new contracts they basically asked us our opinion for which company to choose, based on past performance. We would also review other contractors work and give it the thumbs up / thumbs down.
This was on a much smaller scale.
http://en.wikipedia.org/wiki/List_of_federally_funded_resear...
That's what's really happening behind the scenes with 99% of this stuff.
You have a prime contractor that then subcontracts the actual work to small companies and then the prime just handles project management and the government interface.
However, this ends up causing problems with the motivations of the individual actors not lining up.
You have a prime contractor that then subcontracts the actual work to small companies and then the prime just handles project management and the government interface.
However, this ends up causing problems with the motivations of the individual actors not lining up.
I worked for a company who provided a SaaS to a few private entities but also some very large state agencies, including an entire branch of the government. I remember sitting in a meeting with my boss and the owner of the company, while we sat on the phone with someone from the procurement department of another agency while they crafted the bid to only allow us to compete. It was disgusting and they did nothing to hide it. The state employee (who knew the company owner socially) even mentioned the competition by name, along the lines of "if we require ______ then ______ won't be able to bid, right?"
Competitors can bid, but will have generally worse answers for the specially crafted requirements. The competitors can either say "no, we don't do that" and hope that the committee doesn't mind, say "yes, we could do that" and see if the agency will pay extra for that special req, or say "yes, we do that" and build out that feature if they win the bid.
In this case, the bid specified company requirements (e.g. number of employees, past projects, etc) that immediately narrowed the field to 2 or 3 firms.
Add in special consideration (the owner of the company I worked at was a woman and my state gives preference points to woman-, minority- and veteran-owned businesses) and it was about as close to a no-bid contract as you can get.
Add in special consideration (the owner of the company I worked at was a woman and my state gives preference points to woman-, minority- and veteran-owned businesses) and it was about as close to a no-bid contract as you can get.
Eh, once i was at a startup where we were selling our mostly COTS software to a government agency, but they had to put it out for a bid. They basically lifted our marketing material into the req and said "software must do all <X> and cost $<Y>."
We had a competitor who had copied a bunch of our stuff, including our typos, but they were missing one essential feature. Yet they still got the bid somehow. A mystery for the ages.
We had a competitor who had copied a bunch of our stuff, including our typos, but they were missing one essential feature. Yet they still got the bid somehow. A mystery for the ages.
Now consider the situation that the government official knows your company can deliver on time and in budget, with little fuss, and your competitors are known to underbid, over-promise, deliver very late or never and discuss endlessly over contractual fine language when they start failing.
It's absolutely disgusting that he'd craft his bid to the supplier he knows isn't taking shit, right?
It's absolutely disgusting that he'd craft his bid to the supplier he knows isn't taking shit, right?
The road to hell is paved with good intentions. Thinking it will turn out better is not a good justification of corruption, specially because in the end, it doesnt turn out better.
The lack of accountability and transparency in these kind of expenditures are common in all governments around the world and its the classic way to stiff taxpayers out of their money.
because in the end, it doesnt turn out better.
<citation needed>
So what's an administrator to do in that situation? Take the "best" bid, see the project go over time, see the contract and specification disputes start, see the costs go up, and end up with a 300M failure after 6 years?
What you call "corruption" and "a lack of accountability or transparency" I would call "exercise best judgment and experience" to save the taxpayers their money. Basically, a procurer doing his job.
You would have some point if software were more amendable to requirements specification and bidding, but the entire industry is still terrible at it, so it's a bad process, because it doesn't actually promote "accountability and transparency" at all.
<citation needed>
So what's an administrator to do in that situation? Take the "best" bid, see the project go over time, see the contract and specification disputes start, see the costs go up, and end up with a 300M failure after 6 years?
What you call "corruption" and "a lack of accountability or transparency" I would call "exercise best judgment and experience" to save the taxpayers their money. Basically, a procurer doing his job.
You would have some point if software were more amendable to requirements specification and bidding, but the entire industry is still terrible at it, so it's a bad process, because it doesn't actually promote "accountability and transparency" at all.
I think we have different ideas of what a government procurer's job is. It's not a government employees job to craft a bid in order to prevent specific private companies from being able to apply. Saying you want a staff of 10 FTEs because $ILLOGICAL_REASON is still illogical but certainly not the least illogical thing in the government procurement process, for sure. Saying you want a staff of 10 FTEs because $FAVORED_COMPANY has 12 and $COMPETITOR has 8 is absolutely corruption. Short of envelopes full of cash I don't know how you get more corrupt than that.
And to your earlier comment, you are assuming both
> your company can deliver on time and in budget, with little fuss
and
> your competitors are known to underbid, over-promise, deliver very late or never and discuss endlessly over contractual fine language when they start failing.
This may very well be the case in other examples of this going on, but I can assure you neither point was true in my case (but is obviously specific to me, the plural of anecdote is not data, etc).
And to your earlier comment, you are assuming both
> your company can deliver on time and in budget, with little fuss
and
> your competitors are known to underbid, over-promise, deliver very late or never and discuss endlessly over contractual fine language when they start failing.
This may very well be the case in other examples of this going on, but I can assure you neither point was true in my case (but is obviously specific to me, the plural of anecdote is not data, etc).
So you immediately reported this to other government officials?
Whistleblower protection hasn't been around for very long, and I'm not sure how it applies to private companies.
The "official" process is likely a very difficult set of hoops to jump through and you're not going to be promoted in that company more than likely, and fired at the first opportunity.
The "official" process is likely a very difficult set of hoops to jump through and you're not going to be promoted in that company more than likely, and fired at the first opportunity.
This was many years ago and it was state not federal. I asked an attorney friend if I had any obligation to do anything (as well as if I would have any protection if I did) and was told that if I needed the job to keep my mouth shut.
Unfortunately, I did and this was my first professional job out of college. Morally I made the wrong call by not speaking up even though truthfully I don't know if what happened was actually illegal at the time or not.
Unfortunately, I did and this was my first professional job out of college. Morally I made the wrong call by not speaking up even though truthfully I don't know if what happened was actually illegal at the time or not.
The way the UK government is starting to do things is to insource project management and then run tenders for smaller components, explicitly shaped so smaller firms have a chance of competing. This seems like a sane approach.
> is nearly 99% about "playing the game" and about 1% actually delivering something. Especially since it seems that even if you fail spectacularly, you still get paid (possibly paid even more to fix your stupidity).
This is true for almost all business, especially consulting work.
This is true for almost all business, especially consulting work.
I've frequently encountered it as "let's spend a lot of money on making sure we don't waste any money."
Exactly. My interpretation of it is "Appearance of risk is more important than actual risk". Letting a small team run loose with unproven technologies appears risky. Using big names with expensive "enterprise solutions" appears low-risk. But really, those expensive tools and complex processes create whole new categories of risk.
That's brilliant.
How much staff do they need to make sure everything is okay? Oh you have a database? Well you need a DBA on this. Security? Who is making sure the application is secure? Where is the security officer? Etc. etc.
Many of the boxes can only be checked by having more people.
How much staff do they need to make sure everything is okay? Oh you have a database? Well you need a DBA on this. Security? Who is making sure the application is secure? Where is the security officer? Etc. etc.
Many of the boxes can only be checked by having more people.
My impression working for a government contractor years ago was that the government had great systems for getting the best price on typewriters, and that these systems more or less guaranteed the purchase of obsolescent computer technology.
This happens in the private sector too. Sometimes consultants are hired to write the RFPs which, suprise surprise, they tend to win.
> government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow.
To be fair, what that really means is that the contractors that they want to win already know the requirements, and that's the closest way they can get to impartially telling you to fuck off.
Generally, that response connotes that the bid is tailored to contractors already doing work with the agency in question, who are not only intimately aware of the request for proposal, but likely had a hand in its creation.
To be fair, what that really means is that the contractors that they want to win already know the requirements, and that's the closest way they can get to impartially telling you to fuck off.
Generally, that response connotes that the bid is tailored to contractors already doing work with the agency in question, who are not only intimately aware of the request for proposal, but likely had a hand in its creation.
Interesting quote from the article:
> 'You have companies like Lockheed Martin -- they're not web development companies, they don't do work in the private sector, they're not building start-ups,' Gershman said. 'There's very little in the market that forces them to adopt new technologies.'
All of these are true, but I think this glosses over some fundamental structural problems with government contracting. The SSA has almost 60 million beneficiaries, and deals with distributing an entitlement that is extremely sensitive and protected by layers of legal process. 8,000 Social Security checks get lost,[1] and its not a blog post, its national news.
Moreover, a lot of lean and agile processes just aren't consistent with political reality. There's no minimal viable product, there's no invite-only beta testing, there's no iteration, there's no A/B testing, etc. Technical decisions ("let's just roll out to northern Alabama first") become divisive political ones. Purchasing decisions ("we've had good luck with this vendor before") become the basis for accusations of favoritism and corruption.
Perhaps most importantly, there's no failing fast. The private sector accepts the fact that many engineering projects will fail.[2] Good companies take care not to punish failure unreasonably. But in the public sector, failure is just license for those opposed to project to crow: "we shouldn't have been doing this anyway!" If you fail fast, it's easy to pin the blame on you. This creates a perverse incentive to drag a project along so failure happens over years if not decades.
[1] https://medium.com/@jan.curn/how-bug-in-dropbox-permanently-...
[2] Most big IT projects fail: http://www.computerworld.com/s/article/9243396/Healthcare.go.... Most startups fail. Most internal prototypes never amount to anything. This is just how engineering works.
> 'You have companies like Lockheed Martin -- they're not web development companies, they don't do work in the private sector, they're not building start-ups,' Gershman said. 'There's very little in the market that forces them to adopt new technologies.'
All of these are true, but I think this glosses over some fundamental structural problems with government contracting. The SSA has almost 60 million beneficiaries, and deals with distributing an entitlement that is extremely sensitive and protected by layers of legal process. 8,000 Social Security checks get lost,[1] and its not a blog post, its national news.
Moreover, a lot of lean and agile processes just aren't consistent with political reality. There's no minimal viable product, there's no invite-only beta testing, there's no iteration, there's no A/B testing, etc. Technical decisions ("let's just roll out to northern Alabama first") become divisive political ones. Purchasing decisions ("we've had good luck with this vendor before") become the basis for accusations of favoritism and corruption.
Perhaps most importantly, there's no failing fast. The private sector accepts the fact that many engineering projects will fail.[2] Good companies take care not to punish failure unreasonably. But in the public sector, failure is just license for those opposed to project to crow: "we shouldn't have been doing this anyway!" If you fail fast, it's easy to pin the blame on you. This creates a perverse incentive to drag a project along so failure happens over years if not decades.
[1] https://medium.com/@jan.curn/how-bug-in-dropbox-permanently-...
[2] Most big IT projects fail: http://www.computerworld.com/s/article/9243396/Healthcare.go.... Most startups fail. Most internal prototypes never amount to anything. This is just how engineering works.
Later on the article describes the project as being "adrift" with ill-defined requirements and scope. That will kill absolutely any project and is a completely separate problem from technology. Lockheed failed at the engagement level to define scope. Their development team didn't have a prayer.
Pretty much all government IT projects have ill-defined requirements because GS-15s and Field Grade Military Officers are like little kids. They change their minds frequently and they are so used to having their way that they simply won't take "no" for an answer. You can't explain to them that it will ruin a project because the overwhelming majority of them go through life thinking that they know everyone else's job better than they do. Because of this a typical government IT project will keep having requirements added until the last day of the contract.
You are right, the development team never had a real chance at achieving success.
You are right, the development team never had a real chance at achieving success.
It's worse than that. It's not just GS-15s. It starts with Congress.
Congress writes the bills that somebody (higher than GS15s?) turn into implementing regulations that GS-15s turn into program requirements. If Congress writes bills that are simple, clear, and short, then you have a chance at simple implementing regulations and therefore simple program requirements (not certainty, but a chance). But if Congress writes convoluted, lengthy crap, it's almost impossible to write a program that will correctly implement it.
Congress writes the bills that somebody (higher than GS15s?) turn into implementing regulations that GS-15s turn into program requirements. If Congress writes bills that are simple, clear, and short, then you have a chance at simple implementing regulations and therefore simple program requirements (not certainty, but a chance). But if Congress writes convoluted, lengthy crap, it's almost impossible to write a program that will correctly implement it.
>higher than GS15s?
Yeah, I forgot to mention Congress and the SES(the higher than GS-15) guys. Yet, even when there is a simple set of program requirements, the guys I already mentioned are going to keep adding shit on until the very end. If you give a high-ranking government employee authority over an IT project, they start imagining that they are Steve Wozniak, Bill Gates, Johnny Ive, Steve Jobs, and Leonardo DaVinci all rolled into one. Then bad things happen.
Yeah, I forgot to mention Congress and the SES(the higher than GS-15) guys. Yet, even when there is a simple set of program requirements, the guys I already mentioned are going to keep adding shit on until the very end. If you give a high-ranking government employee authority over an IT project, they start imagining that they are Steve Wozniak, Bill Gates, Johnny Ive, Steve Jobs, and Leonardo DaVinci all rolled into one. Then bad things happen.
And... Congress can change the law halfway through the project and then what? That's the ultimate requirements change, and no way to push back.
>>Perhaps most importantly, there's no failing fast. The private sector accepts the fact that many engineering projects will fail.[2] Good companies take care not to punish failure unreasonably. But in the public sector, failure is just license for those opposed to project to crow: "we shouldn't have been doing this anyway!" If you fail fast, it's easy to pin the blame on you.
This is not unreasonable. After all, projects in the private sector are funded by private funds. If you fail, you only waste your own money. Whereas public projects are funded by the taxpayer. When they fail, what's really happening is that taxpayer money is going into the pockets of some private company, but there's only a failed project to show for it. So yeah, of course the backlash is going to be much stronger.
This is not unreasonable. After all, projects in the private sector are funded by private funds. If you fail, you only waste your own money. Whereas public projects are funded by the taxpayer. When they fail, what's really happening is that taxpayer money is going into the pockets of some private company, but there's only a failed project to show for it. So yeah, of course the backlash is going to be much stronger.
In the private sector, investors provide capital that companies (and employees thereof) use to pursue projects. When the company fails, the employees are generally still paid (while some companies have employees who participate in profits / share risk with investors, it's rare for it to be a dominant form of compensation). Thus, the situation is more similar than you describe: if the projects fail, the investor/owner has paid out a lot of money to employees and vendors and is left with nothing.
The difference is that smart investors/owners are aware that the possibility of failure is inextricably tied to any attempt to excel, and allows people to try things that might fail in the pursuit of success.
In our current political climate (and probably in other climates as well), governments are dumber, and tend to drive their staff to manage towards over-safety and covering of asses rather than towards best outcomes.
The difference is that smart investors/owners are aware that the possibility of failure is inextricably tied to any attempt to excel, and allows people to try things that might fail in the pursuit of success.
In our current political climate (and probably in other climates as well), governments are dumber, and tend to drive their staff to manage towards over-safety and covering of asses rather than towards best outcomes.
[deleted]
[deleted]
All of what you say is correct; but I think the answer is to look for ways to change the status quo. There are ways to do that, it just takes engagement and creative thinking.
Can't you build an MVP with dummy data?
I have actually worked on the DCPS product as a former LM employee, so here's what I saw: there are two sides to the problem. The biggest problem is the number of stakeholders. Currently, each state has their own system to process disability cases, and each system has its own way of doing things. SSA & Lockheed are trying to consolidate these 50+ systems into one national system. Every state this product is rolled out in creates some new challenge to make it work the way the state's SSA office expects. Hopefully this new exec will have the power to make some decisions and say "no" once in awhile.
Lockheed has their own set of problems, mostly hinging on the contract they've signed with SSA. They are forced to hire a certain number of subs per their contract - hiring a percentage of subs is not unusual, but the level in this contract is very high. As such, the subs know they can force feed trash into LM. The resumes coming from the subs were complete bullshit. Things like "10 years of experience administering X", but then in the interview, you find out they know next to nothing. Especially bad with databases when you even try to ask them how to write a join. One meeting we had while I was still there was a "skills assessment" - from 1 to 10, how well do you know X? There was no negative to this meeting, we just wanted to know who knew what. The number of "10s" in that meeting was beyond belief.
So sure, Lockheed may not have a lot of experience here, but that problem is exacerbated by the stakeholders and the contractors they are forced to hire.
Lockheed has their own set of problems, mostly hinging on the contract they've signed with SSA. They are forced to hire a certain number of subs per their contract - hiring a percentage of subs is not unusual, but the level in this contract is very high. As such, the subs know they can force feed trash into LM. The resumes coming from the subs were complete bullshit. Things like "10 years of experience administering X", but then in the interview, you find out they know next to nothing. Especially bad with databases when you even try to ask them how to write a join. One meeting we had while I was still there was a "skills assessment" - from 1 to 10, how well do you know X? There was no negative to this meeting, we just wanted to know who knew what. The number of "10s" in that meeting was beyond belief.
So sure, Lockheed may not have a lot of experience here, but that problem is exacerbated by the stakeholders and the contractors they are forced to hire.
Why are the forced to hire subcontractors, and a certain # of them? Do you know what the thinking is behind that?
The overall idea behind being "forced" to hire subs is that it helps promote smaller, local businesses. In the past this worked out very well, because the subs brought in some very knowledgeable people. I believe the old number was 25% of the people, and you could pull from anywhere. If you had a 20 person team, you could have 15 LM, 2 from Company X, and 3 from Company Y.
On DCPS, the problem is that LM is forced to hire 45% (FORTY-FIVE) of the subs through one specific subcontractor. That company had a previous contract with SSA, and SSA liked them. Since the sub essentially lost their work with SSA to LM, and they know LM was contractually obligated to hire through them, they stuffed us with a ton of below average folks.
On the face of it, I agree with the idea of a large contractor having to sub out some of the work to the smaller players; it's just that LM overplayed their hand trying to get this contract, and got screwed for it. But that's another story on how LM management is trying to make revenue numbers for bonuses...
On DCPS, the problem is that LM is forced to hire 45% (FORTY-FIVE) of the subs through one specific subcontractor. That company had a previous contract with SSA, and SSA liked them. Since the sub essentially lost their work with SSA to LM, and they know LM was contractually obligated to hire through them, they stuffed us with a ton of below average folks.
On the face of it, I agree with the idea of a large contractor having to sub out some of the work to the smaller players; it's just that LM overplayed their hand trying to get this contract, and got screwed for it. But that's another story on how LM management is trying to make revenue numbers for bonuses...
It's still got nothing on the NHS £10bn IT flop...
I always wonder when I hear about these projects, where exactly has the money gone? It can't possibly be in hardware, that would make some of the biggest data centres in the world just for the NHS. I can't even see how it can be manpower, how many IT people can you get for £10bn? And surely if it were corruption, someone would notice so much money being squirreled away...
There's just no logical explanation behind these things.
I always wonder when I hear about these projects, where exactly has the money gone? It can't possibly be in hardware, that would make some of the biggest data centres in the world just for the NHS. I can't even see how it can be manpower, how many IT people can you get for £10bn? And surely if it were corruption, someone would notice so much money being squirreled away...
There's just no logical explanation behind these things.
Management. Business Analysts. "Metawork" in the words of a coworker who used to architect BPM implementations at the parent corporation of my job. Getting consensus from possibly hundreds of stakeholders.
Generally, the disconnect is that management is under the assumption that you can manage your way to success, so they add more overhead to watch and prod at the proceedings. Development is almost always an afterthought in most business projects. The first thing they do? Get consensus, which is generally everyone putting in ideas so that they seem involved—see Atwood's Duck[1]. After you have a ton of often contradictory ideas, you need to have them prioritized. And since nobody managing the project is technical, they need to hold countless "prioritization meetings with stakeholders and implementers." Often times the meetings will grind to a standstill while process analysts and engineers argue over the wording of things. Usually the analyst will tell the engineer that the business has standardized on a set of terms, of which are not the trade names for things. Maybe everyone uses a set of terms which aren't the "correct" wording.
After things are prioritized based on the wrong criteria, they're then pushed off to an engineering group who are tasked with implementing things exactly to spec. Since the definitions are incorrect, they need to constantly circle back and check with the stakeholders to double check. Occasionally checking in will involve a "subject matter expert" who is on vacation, and the only one allowed to answer the question. (Thus stalling that item and anything depending on it.) Others will often say "it's above my paygrade" in an attempt to avoid culpability.
After the project has run long and only has a majority of the features complete, people are reassigned to other projects, and the app is sent to QA to hand-test features. Sometimes it will involve the "stakeholders" sitting in a room for days or weeks, manually clicking their way through the interface. Maybe the project is declared a boondoggle, deleted, and replaced with someone else's pet project.
So yes, not logical.
1: http://blog.codinghorror.com/new-programming-jargon/
Generally, the disconnect is that management is under the assumption that you can manage your way to success, so they add more overhead to watch and prod at the proceedings. Development is almost always an afterthought in most business projects. The first thing they do? Get consensus, which is generally everyone putting in ideas so that they seem involved—see Atwood's Duck[1]. After you have a ton of often contradictory ideas, you need to have them prioritized. And since nobody managing the project is technical, they need to hold countless "prioritization meetings with stakeholders and implementers." Often times the meetings will grind to a standstill while process analysts and engineers argue over the wording of things. Usually the analyst will tell the engineer that the business has standardized on a set of terms, of which are not the trade names for things. Maybe everyone uses a set of terms which aren't the "correct" wording.
After things are prioritized based on the wrong criteria, they're then pushed off to an engineering group who are tasked with implementing things exactly to spec. Since the definitions are incorrect, they need to constantly circle back and check with the stakeholders to double check. Occasionally checking in will involve a "subject matter expert" who is on vacation, and the only one allowed to answer the question. (Thus stalling that item and anything depending on it.) Others will often say "it's above my paygrade" in an attempt to avoid culpability.
After the project has run long and only has a majority of the features complete, people are reassigned to other projects, and the app is sent to QA to hand-test features. Sometimes it will involve the "stakeholders" sitting in a room for days or weeks, manually clicking their way through the interface. Maybe the project is declared a boondoggle, deleted, and replaced with someone else's pet project.
So yes, not logical.
1: http://blog.codinghorror.com/new-programming-jargon/
It is a bit scary that an accurate description of reality could easily have been sourced from Dilbert.
The tough reality is that certain projects have a scope such that a small team could never reasonably be expected to capture the requirements of the project. Also, remember that large enterprise projects are generally built by people who are not users themselves. This is where the technology crowd often draws false conclusions, reflecting on projects of their own, where they have intimate knowledge of the problem space. There are likely very few forward-thinking engineers who also have visibility into the intricacies of the various SS systems. This is where the layers get piled on and the efficiency vanishes.
The tough reality is that certain projects have a scope such that a small team could never reasonably be expected to capture the requirements of the project. Also, remember that large enterprise projects are generally built by people who are not users themselves. This is where the technology crowd often draws false conclusions, reflecting on projects of their own, where they have intimate knowledge of the problem space. There are likely very few forward-thinking engineers who also have visibility into the intricacies of the various SS systems. This is where the layers get piled on and the efficiency vanishes.
This is true. And it doesn't just apply to governments, but any organisation that is large.
Once a organisation gets to a sufficient size, organising and allocating resources and communication starts to eat up a disproportionate amount of time, if the organisation is left to drift into whatever shape it likes.
Once a organisation gets to a sufficient size, organising and allocating resources and communication starts to eat up a disproportionate amount of time, if the organisation is left to drift into whatever shape it likes.
I can recommend the book "Plundering the Public Sector" by David Craig that explains how projects like the NHS IT fiasco happened:
http://www.amazon.co.uk/Plundering-Public-Sector-Letting-Con...
[NB Note recommended for UK taxpayers who have high blood pressure and/or easily upset]
http://www.amazon.co.uk/Plundering-Public-Sector-Letting-Con...
[NB Note recommended for UK taxpayers who have high blood pressure and/or easily upset]
"I can't even see how it can be manpower"
From personal experience in software development, dilution of responsibility can be incredibly expensive.
Scenario 1: I wonder about a minor detail, call up a random user, we hash it out in 5 minutes. Cost is about 10 minutes of total labor, figure maybe $10
Scenario 2: I have to organize a meeting with the exec stakeholder, the dept manager, and the entire dept staff. Which due to scheduling conflicts is in two weeks. Finally the meeting arrives and due to bike shedding we spend 2 hours talking about everything other than the topic until its hashed out. So the cost is two weeks of my salary doing nothing but surfing HN, lets say $3K, plus 2 hrs of mgmt for exec at $100/hr and the manager at $75 thats $350, plus double that time for them to make the "command decision" and later inform me, so thats $700 of management, plus the ten person working team has to burn an extra 2 hrs of overtime each at lets say $25/hr and 10 people that's $750 so we'll call it $4450.
So figuring out the default value, if any, for a blank of a form can cost anywhere from ten bucks at a personally responsible organization up to about five grand at a fully diluted responsibility organization.
Its obviously orders of magnitude more if you involve consultants, people flying in from out of state, etc.
From personal experience in software development, dilution of responsibility can be incredibly expensive.
Scenario 1: I wonder about a minor detail, call up a random user, we hash it out in 5 minutes. Cost is about 10 minutes of total labor, figure maybe $10
Scenario 2: I have to organize a meeting with the exec stakeholder, the dept manager, and the entire dept staff. Which due to scheduling conflicts is in two weeks. Finally the meeting arrives and due to bike shedding we spend 2 hours talking about everything other than the topic until its hashed out. So the cost is two weeks of my salary doing nothing but surfing HN, lets say $3K, plus 2 hrs of mgmt for exec at $100/hr and the manager at $75 thats $350, plus double that time for them to make the "command decision" and later inform me, so thats $700 of management, plus the ten person working team has to burn an extra 2 hrs of overtime each at lets say $25/hr and 10 people that's $750 so we'll call it $4450.
So figuring out the default value, if any, for a blank of a form can cost anywhere from ten bucks at a personally responsible organization up to about five grand at a fully diluted responsibility organization.
Its obviously orders of magnitude more if you involve consultants, people flying in from out of state, etc.
The hourly rates are kind of low across the board. May be doubling it or tripling it?
gov.uk which had a great success, was done by hiring their own people to work on it. I don't see why anyone would think outsourcing is good for public projects.
I worked on a high profile government project that had five different vendors involved. FIVE. The inter-organizational management problems were astonishing. I'm pretty sure that by the time I left, managers actually outnumbered technical staff.
The biggest problem, though, was the lack of pushback. The government officials would ask for the moon, and the vendors would promise to deliver it. There was almost no one saying NO - the word that project most desperately needed to hear. Features outweighed core infrastructure reliability, so configuration management was a nightmare.
The biggest problem, though, was the lack of pushback. The government officials would ask for the moon, and the vendors would promise to deliver it. There was almost no one saying NO - the word that project most desperately needed to hear. Features outweighed core infrastructure reliability, so configuration management was a nightmare.
Because ramping up a huge staff to build a new project generally requires keeping them forever. It's much easier to request a budget for a project then it is to try to expand headcount of well-protected government jobs.
It's very good for the companies that benefit from the outsourcing.
>'The project was conceived to replace 54 separate components used in the SSA's disability determination system.'
This is the bit that jumps out at me as clear indicator of the sort dysfunction that dooms projects like this more so than anything else.
It's indicative of a the classic exploding scope and fragmented responsibilities leading to an overall lack of accountability that unfortunately characterizes IT in and around Government.
Gershman seems to agree.
"If there's not strong leadership, problems arise with coordinating work between contractors, and there's no one person looking at the system as a single entity -- everyone is making sure their piece is done, and fulfilling their responsibility up to a certain point,"
However, I'm not wholly keen on the the start-ups as a solution direction taken later. It's not that it's wrong, just beside the point unless the start-up savior is specifically targetting culture and process change as opposed to technical implementations.
The 54-armed kraken of a defined by committee project will drown and devour a fleet of confused start-ups lost in the mist of *BE requirements and contract mods just as readily as the Lockheed leviathan.
This is the bit that jumps out at me as clear indicator of the sort dysfunction that dooms projects like this more so than anything else.
It's indicative of a the classic exploding scope and fragmented responsibilities leading to an overall lack of accountability that unfortunately characterizes IT in and around Government.
Gershman seems to agree.
"If there's not strong leadership, problems arise with coordinating work between contractors, and there's no one person looking at the system as a single entity -- everyone is making sure their piece is done, and fulfilling their responsibility up to a certain point,"
However, I'm not wholly keen on the the start-ups as a solution direction taken later. It's not that it's wrong, just beside the point unless the start-up savior is specifically targetting culture and process change as opposed to technical implementations.
The 54-armed kraken of a defined by committee project will drown and devour a fleet of confused start-ups lost in the mist of *BE requirements and contract mods just as readily as the Lockheed leviathan.
What I meant by startups is that in that world, there's market forces driving you to do things better, faster and cheaper. Not necc. that we need startups to solve the IT problems of government (although that could help), but that we need to create an environment where there's market forces pushing government contractors to do things for less people/less money and better results (which are all very, very possible).
This is business as usual in government contracting. The goal is not really to deliver working software - instead, it's multiple companies working together to figure out how they can maximally extract income from an entity with unfathomably deep pockets. It's layer upon layer of red tape sprinkled with the occasional sociopath just to spice things up.
The filthiest part is that nobody but the taxpayer is interested in things being more efficient. The government agencies don't really care because there is no real financial accountability. Besides, if they use up all their budgets, they just get to ask for more next year. And contractors generally have their profits limited to a percentage of the project cost, so their goal is to balloon the cost as large as possible.
For several years I worked on a very large government project. When you're sitting in a two-hour, 100 person, sprint review every two weeks and everyone is fucking off on their phones the whole time, it literally makes you question your sanity.
The filthiest part is that nobody but the taxpayer is interested in things being more efficient. The government agencies don't really care because there is no real financial accountability. Besides, if they use up all their budgets, they just get to ask for more next year. And contractors generally have their profits limited to a percentage of the project cost, so their goal is to balloon the cost as large as possible.
For several years I worked on a very large government project. When you're sitting in a two-hour, 100 person, sprint review every two weeks and everyone is fucking off on their phones the whole time, it literally makes you question your sanity.
The sad thing is not that the government pissed away $300M on this project; but that the prime, LMFS, will continue to get away with it because there are no penalties!
But the system is rigged that way: if you do decide to hold them accountable, you'll find that they have documented every little specification change that the SSA put in after the project was awarded. So now the blame comes back to the SSA. Both sides will point fingers at each other; the consultants on this project will laugh (on their way to the bank), and things will remain the same.
In other words: the system of contracting in place today is broken. From labyrinthine requirements to bid for one, to obtuse specifications to lack of oversight; it's one huge mess.
And the system is corrupt to boot. Primes like LMFS, etc. employ former employees of these agencies (and vice versa), so it's very hard for an outsider to break in.
But the system is rigged that way: if you do decide to hold them accountable, you'll find that they have documented every little specification change that the SSA put in after the project was awarded. So now the blame comes back to the SSA. Both sides will point fingers at each other; the consultants on this project will laugh (on their way to the bank), and things will remain the same.
In other words: the system of contracting in place today is broken. From labyrinthine requirements to bid for one, to obtuse specifications to lack of oversight; it's one huge mess.
And the system is corrupt to boot. Primes like LMFS, etc. employ former employees of these agencies (and vice versa), so it's very hard for an outsider to break in.
Pennies in an ocean:
http://www.theguardian.com/commentisfree/2011/aug/03/nhs-dat...
> ...final cost to be as high as £20bn, indicating a cost overrun of 440% to 770%
http://www.theguardian.com/commentisfree/2011/aug/03/nhs-dat...
> ...final cost to be as high as £20bn, indicating a cost overrun of 440% to 770%
"For past 5 years, Release 1.0 is consistently projected to be 24-32 months away."
I have worked on project like this. Usually it's the sign of management failure (poor communication, poorly defined requirements, bad governance, inability to retain knowledgable employees) rather than technical feasibility. It's not like the software is pushing technical boundaries.
"Echoing McKinsey, Gershman said that without a strong leadership element, IT projects of this magnitude are almost doomed to fail from the start."
This is a big problem. Because employees know these problems will fail, the smartest avoid them like the plague.
I have worked on project like this. Usually it's the sign of management failure (poor communication, poorly defined requirements, bad governance, inability to retain knowledgable employees) rather than technical feasibility. It's not like the software is pushing technical boundaries.
"Echoing McKinsey, Gershman said that without a strong leadership element, IT projects of this magnitude are almost doomed to fail from the start."
This is a big problem. Because employees know these problems will fail, the smartest avoid them like the plague.
It's amazing to me how consistently government IT projects fail and how widespread the problem is. A simple google teaches us that this has been and still is a consistent problem for quite a lot of countries. Some with an incredibly strong tech sector (the US is obviously a prime example). Surely there must be a way to organize/orchestrate government IT projects in a way that improves delivery and risk management. And...300 million USD?
Scratch out "government" above - the overall failure rate of IT projects in general is crazy. The difference is that private project failures don't get news coverage (barring utter disasters that bankrupt the company).
[deleted]
No wonder taxes are so unbelievably high. The FBI did the same thing a few years ago. http://www.cnn.com/2005/US/02/03/fbi.computers/
Yes, it's not at all programs like the absolutely-crucial F-35 program costing $399B up front.
Feeding from the government trough is where people REALLY get rich. This is why Palantir is pretty successful, because getting in bed with the government means being able to feed off our tax money much easier at elevated prices.
> This is why Palantir is pretty successful
Ahhhh
Ahhhh
McKinsey and Co probably charged the government a couple of million dollars to get some of its junior partners to write that report which eventually a senior partner signed off before golf and took the credit.
Says the cynic in me...
Says the cynic in me...
> Throughout the past six years, the project has been stuck in the beta phase. According to McKinsey, "for past 5 years, Release 1.0 is consistently projected to be 24-32 months away."
[deleted](3)
Rather offtopic, but does anyone else find the dollar glyph in the header font jarring?
Now that you mention it, yeah it is... It looks like a lot of the symbols and punctuation characters in Garamond Pro are slanted with no logical pattern or reason. Would drive me nuts if I were using it.
http://i.imgur.com/Cbd74Gq.png
http://i.imgur.com/Cbd74Gq.png
When I was consulting, I responded to a few RFP's but they were always either (a) obviously tailored to the capabilities of a single firm so there was no way to win unless you were them or (b) so ridiculously vague or out of this world impossible ("We need you to basically rewrite GitHub for us and the budget is $30k") that it wasn't even worth the time. Even now where I work, we have to deal with government agencies literally telling us they can't tell us any requirements at all because that would give us an advantage over other firms somehow. We just have to pitch our product to them and hope that it fits, even though at this point in our development cycle we could (and really, want to) make adjustments, feature additions, etc. to fit them if we were allowed to.
Government contracting in IT, from my point of view, is nearly 99% about "playing the game" and about 1% actually delivering something. Especially since it seems that even if you fail spectacularly, you still get paid (possibly paid even more to fix your stupidity).