Inside Team Romney’s whale of an IT meltdown(arstechnica.com)
arstechnica.com
Inside Team Romney’s whale of an IT meltdown
http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/
28 comments
A cynic would say that this is just the software consulting version of what management consultants like Romney do.
Yes and no.
On one hand, management consultants are typically working with the executives of a company, and usually one of those executives will have been a former consultant (the executive boardrooms of companies are filled with them). Not to mention that both consultants and executives usually have the same set of "skills."
On the other hand, one former management consultant I spoke to told me that executives would often hire consultants to tell them what they already knew. The only purpose of bringing the consultants on-board was so that their recommendations would serve as political validation in case they had to explain their decision (i.e. if they failed).
On one hand, management consultants are typically working with the executives of a company, and usually one of those executives will have been a former consultant (the executive boardrooms of companies are filled with them). Not to mention that both consultants and executives usually have the same set of "skills."
On the other hand, one former management consultant I spoke to told me that executives would often hire consultants to tell them what they already knew. The only purpose of bringing the consultants on-board was so that their recommendations would serve as political validation in case they had to explain their decision (i.e. if they failed).
Sadly it's not only management consultants doing that one: there's a certain proportion of the engineering-consulting business that consists of management calling in someone with an engineering degree to give a PowerPoint presentation to validate a point or provide political cover for a decision. I don't have any experience in that myself, but more than one of my friends are in jobs like that, which pay well but don't seem to produce much job satisfaction.
As a physician, I routinely do this. I may know the answer, but the patient just sees me as a primary care guy. This is quadruply true if it is a friend or ten times true if it is a family member. Politically far more favorable for me to consult. The specialists (aka, consultants), especially in private practice, are often happy to get an occasional easy consult, and the patient/friend/family appreciates me "recruiting big guns for the problem". If I stuck to insisting I know and I'm right, then I lose credibility, no matter how much evidence I cite. It's a social/political problem, not a scientific problem. And specialists consult each other constantly. There have been days in training I wrote more consults than orders (eg: meds, tests).
Yap pretty much. There was an article from an insider about how these large consulting firms work.
They are hired in order to protect against liability. They are paid money to say what clients want to hear (but in an official "report" and "power point format") and then if shit hits the fan executives can always point to "But look we hired the best of the best, these experts all agreed we should go this, so it is clearly not our fault".
They are hired in order to protect against liability. They are paid money to say what clients want to hear (but in an official "report" and "power point format") and then if shit hits the fan executives can always point to "But look we hired the best of the best, these experts all agreed we should go this, so it is clearly not our fault".
Oh, I don't disagree. I think you see a lot of both: real value-added in operations, logistics, etc, and a lot of purchased CYA. And let's be fair--CYA is something we all do in corporate America, at least if you're smart.
>This sounds like a lot of failed IT projects in large corporations -- dreamed up by upper management types who don't know anything about tech, then farmed out to consultants and tech vendors for the actual implementation. The consultants and vendors then either proceed to wander in the weeds expensively due to lack of direction, or actively seize on the client's ignorance and take them for a ride.
In my experience, this is 100% correct.
At least this mess has been exposed to some degree. There's something of a saying around these kind of projects to the effect of "This project -WILL- succeed".
The subtext being that that no matter what the facts are, the project will be promoted publicly as a success.
It's incredibly disheartening to see an abortive mess transformed into a sparkling success via fake dashboards, paid press and the like.
In my experience, this is 100% correct.
At least this mess has been exposed to some degree. There's something of a saying around these kind of projects to the effect of "This project -WILL- succeed".
The subtext being that that no matter what the facts are, the project will be promoted publicly as a success.
It's incredibly disheartening to see an abortive mess transformed into a sparkling success via fake dashboards, paid press and the like.
O2 had such a "success" in Germany, to the tune of (IIRC) over 100M€. A colleague who was involved told me it had to be officially successful so they could at least write off the money as expenses.
And yet, despite this disaster,
When I first heard about this, I remarked to a friend "what the hell are they doing giving out bonuses? The bonus should be that your candidate gets elected!"
Mitt Romney’s campaign handed out more than $200,000
in bonuses last month to senior staffers, according
to new disclosure records filed Thursday.
Richard Beeson, Romney’s national political director,
received a $37,500 payment on Aug. 31 in addition to his
salary, according to records filed with the Federal
Election Commission.
When I first heard about this, I remarked to a friend "what the hell are they doing giving out bonuses? The bonus should be that your candidate gets elected!"
This is totally offtopic. I have no idea why he gave people bonuses in October, but I don't think it has anything to do with a system that apparently wasn't deployed until November.
Some people are volunteers, and some people work for money. I see no problem in somebody getting paid a bonus for good work, and not everybody on the campaign should be in it for pure politics. And yes, it is possible to do a good work and still not have your candidate elected. After all, only one candidate wins, that does not mean everybody on other candidate's team sucked.
The Orca thing was obviously an epic fail, but that doesn't mean other people didn't deserve bonuses - maybe their part of the job was just fine.
The Orca thing was obviously an epic fail, but that doesn't mean other people didn't deserve bonuses - maybe their part of the job was just fine.
The bonuses in August were probably due to him winning the nomination. http://www.nytimes.com/2012/08/29/us/politics/delegates-get-...
He probably also had other unknown milestones throughout the campaign in an effort to keep staffers motivated.
He probably also had other unknown milestones throughout the campaign in an effort to keep staffers motivated.
Indeed. Particularly in contract to the Obama machine, which was built by passionate technologies motivated mostly to help Obama win, not make money.
Interestingly, Obama campaign manager Jim Messina said they built something similar in 2008, and it crashed too.
http://www.buzzfeed.com/rubycramer/messina-obama-won-on-the-...
http://www.buzzfeed.com/rubycramer/messina-obama-won-on-the-...
For all the praise Obama’s team won in 2008 for its high-tech wizardry, its success masked a huge weakness: too many databases. Back then, volunteers making phone calls through the Obama website were working off lists that differed from the lists used by callers in the campaign office. Get-out-the-vote lists were never reconciled with fundraising lists. It was like the FBI and the CIA before 9/11: the two camps never shared data. “We analyzed very early that the problem in Democratic politics was you had databases all over the place,” said one of the officials. “None of them talked to each other.” So over the first 18 months, the campaign started over, creating a single massive system that could merge the information collected from pollsters, fundraisers, field workers and consumer databases as well as social-media and mobile contacts with the main Democratic voter files in the swing states.
http://swampland.time.com/2012/11/07/inside-the-secret-world...
Sounds like they spent quite a bit of time and money on that. Nice article, by the way, though it's obviously a bit of a puff piece.
http://swampland.time.com/2012/11/07/inside-the-secret-world...
Sounds like they spent quite a bit of time and money on that. Nice article, by the way, though it's obviously a bit of a puff piece.
I'm kind of scratching my head on this. This is little more than a secure system that tallies information, and generates work items, with a user base in the tens of thousands. Why is it so filled with fail?
Wait, according to some commenters here this means Obama is completely incompetent and unfit to be president? I wonder if these people want to rethink their position...
When I worked for the Winter Olympics in Vancouver, it was pounded into our heads all the time that there was no second chance. Everything we did would get one chance only.
For that reason, we had two Technical Rehearsals. The purpose was to test both our systems and our procedures, and identify how they could be refined for the real thing. It was someone's job to actually design a huge number of expected and unexpected issues so that tech operations teams at each venue could be tested as to their response. Issues ranged from broken printers to network meltdowns to broadcasting failures, etc. We had two Technical Rehearsals for hopefully obvious reasons. And before the Technical Rehearsals, we also had the test events (i.e. world class but lower profile sporting competitions held at the Olympic venues to see whether we fell short in putting up a world class competition event).
We also had Disaster Rehearsals to test our disaster recovery systems and procedures, though I wasn't personally involved in those. I did get a string of text messages when it was happening though. "Primary datacentre flooding, starting emergency processes to switch to secondary datacentre now." etc. Never mind that nobody could ever believe that the primary datacentre could get flooded in a million years. The attitude was that any disaster was possible, and we had to prepare for the impossible.
If you have one shot only, and that shot is important, you need to have a large process committed to making sure that one shot goes well. And that process needs to be managed by someone experienced in that type of thing.
For that reason, we had two Technical Rehearsals. The purpose was to test both our systems and our procedures, and identify how they could be refined for the real thing. It was someone's job to actually design a huge number of expected and unexpected issues so that tech operations teams at each venue could be tested as to their response. Issues ranged from broken printers to network meltdowns to broadcasting failures, etc. We had two Technical Rehearsals for hopefully obvious reasons. And before the Technical Rehearsals, we also had the test events (i.e. world class but lower profile sporting competitions held at the Olympic venues to see whether we fell short in putting up a world class competition event).
We also had Disaster Rehearsals to test our disaster recovery systems and procedures, though I wasn't personally involved in those. I did get a string of text messages when it was happening though. "Primary datacentre flooding, starting emergency processes to switch to secondary datacentre now." etc. Never mind that nobody could ever believe that the primary datacentre could get flooded in a million years. The attitude was that any disaster was possible, and we had to prepare for the impossible.
If you have one shot only, and that shot is important, you need to have a large process committed to making sure that one shot goes well. And that process needs to be managed by someone experienced in that type of thing.
Like the IBM "Black Team":
http://www.t3y.com/tangledwebs/07/tw0706.html
http://www.t3y.com/tangledwebs/07/tw0706.html
Ironic that Mitt Romney ran the Salt Lake City Olympic committee in light of this.
Also, the Ars article seemed to imply they wanted to keep how it worked a secret, so big testing like this would have tipped Obama what they were doing.
Also, the Ars article seemed to imply they wanted to keep how it worked a secret, so big testing like this would have tipped Obama what they were doing.
[deleted]
Zero training? A single web server? No end users even touching the system till it's time for mission critical use? No frigging redirect of http->https? And it looks like, as has happened more than once in my professional life, people outside of the management structure saw possible risks and were blown off by those in charge.
Even if you supported Romney, this has to give you some pause about the man's management skills and who he hired to run things.
Even if you supported Romney, this has to give you some pause about the man's management skills and who he hired to run things.
> Even if you supported Romney, this has to give you some pause about the man's management skills and who he hired to run things.
Wait, what?
There's a difference between people taking responsibility for what their staff do (thus, he should take responsibility for it) and people actually being responsible for what their staff do - especially if those staff are not staff but sub-contracted service providers.
Wait, what?
There's a difference between people taking responsibility for what their staff do (thus, he should take responsibility for it) and people actually being responsible for what their staff do - especially if those staff are not staff but sub-contracted service providers.
If you don't hire people with adequate management skills to assess risk and take proactive steps to mitigate problems before they become dangerous to the mission, that's a failure at the highest level. Someone should've see this coming, or Romney clearly doesn't know how to delegate responsibility well.
Now let's talk about Eric Holder, Obama, and a dead ATF agent. Something tells me you won't see it the same way.
There is a magnitude of difference between the two. When you run a top heavy campaign to get elected and a critical project fails miserably, that falls on you. When you preside over a government that consists of a large number of relatively autonomous agencies and departments, and something goes wrong deep inside one of them, that falls on the project managers and department heads.
To provide some scale, Romney's campaign employed approximately 500 staff. The Federal government employs more than 4 million.
To provide some scale, Romney's campaign employed approximately 500 staff. The Federal government employs more than 4 million.
Eric Holder is not one of 4 million. Eric Holder is a personal selection. It's not like it is a random failure in some lowly backwater office. It is a failure and a coverup going to the very top - to people personally appointed by the President. So if you think people are responsible for a mobile app done for them - then they should be responsible for the mess in their administration too.
And if we start counting how many projects endorsed, sponsored and initiated by the federal government failed recently, we could be here all night.
And if we start counting how many projects endorsed, sponsored and initiated by the federal government failed recently, we could be here all night.
No, there is an accusation that there is a cover up going on - this has not been proven in any sense.
I think Holder claiming the email having words "Fast and Furious" doesn't talk about Operation Fast and Furious, and his and Obama's refusal to comply with document request is ample evidence of coverup going on.
http://www.cbsnews.com/8301-31727_162-20115038-10391695.html Read the docs yourself and try to explain how comes Holder says he did not hear about F&F until 2011 when these documents are from 2010 and are addressed to him and his immediate subordinates. Only one explanation is logical - Eric Holder flatly lied to the Congress. And got away with it. That's the coverup, and if seeing it with your own eyes is not a proof, then I don't know what could prove it.
http://www.cbsnews.com/8301-31727_162-20115038-10391695.html Read the docs yourself and try to explain how comes Holder says he did not hear about F&F until 2011 when these documents are from 2010 and are addressed to him and his immediate subordinates. Only one explanation is logical - Eric Holder flatly lied to the Congress. And got away with it. That's the coverup, and if seeing it with your own eyes is not a proof, then I don't know what could prove it.
This whole "fast and furious" thing is just mind-bogglingly stupid. Seriously-- it takes a lot of effort to be this stupid.
Do you realize the reason why Mexican drug gangs buy guns in Arizona? It's because the gun laws there are extremely, extremely weak, to the point where random 18-year-olds can walk out of the door with $20,000 of guns, with no oversight. Who keeps the gun laws this weak? Republicans. Who then cynically try to frame Obama for a problem they created.
Even if David Voth ever asked anyone to sell a gun to a bad guy-- and if you read the Fortune article, it seems pretty clear that he probably never did-- so what? Are we going to start prosecuting undercover narcotics agents for dealing drugs? This whole thing just shows such a tremendous lack of logic that it's just... gah.
The especially ironic thing is that since the witch hunt, the ATF's seizures of weapons has dropped 90%. There will be a lot more Brian Terrys as a result of this stupidity.
Do you realize the reason why Mexican drug gangs buy guns in Arizona? It's because the gun laws there are extremely, extremely weak, to the point where random 18-year-olds can walk out of the door with $20,000 of guns, with no oversight. Who keeps the gun laws this weak? Republicans. Who then cynically try to frame Obama for a problem they created.
Even if David Voth ever asked anyone to sell a gun to a bad guy-- and if you read the Fortune article, it seems pretty clear that he probably never did-- so what? Are we going to start prosecuting undercover narcotics agents for dealing drugs? This whole thing just shows such a tremendous lack of logic that it's just... gah.
The especially ironic thing is that since the witch hunt, the ATF's seizures of weapons has dropped 90%. There will be a lot more Brian Terrys as a result of this stupidity.
Even if what you said about F&F were true, this does not change the fact Holder lied about it to Congress. If he thought it's a good operation - he should have said "yes, I knew about it since 2010 and it's just fine and nothing wrong with it", not "I just learned about it in 2011".
Now back to F&F. If you took a moment to educate yourself, you'd learn the whole point of F&F was to CIRCUMVENT the gun laws and allow the cartels to buy massive amount of guns from dealers that otherwise they would not be able to buy. The dealers signalled the ATF multiple times, but where told to shut up and sell. The rank agents asked to arrest the buyers and take the guns, but were told to stand down and not interfere since it is an operation to track the guns into Mexico, so everything is going fine and the guns should "walk". In fact, there was no way to track them and no tracking actually happened - the guns just disappeared. Blaming gun laws for the sales that were specifically protected by the government agency whose purpose is to monitor gun sales - well, yes, mind-bogglingly stupid. Using government agency fuckup to prove that we need more government involvement - mind-bogglingly stupid. Trying to sell guns to Mexico without actually being able to find them and without telling anybody on Mexican side a thousand guns is heading their way - mind-bogglingly stupid. Blaming Republicans for the mess - I'd say it takes a lot of effort to be this stupid, but apparently it does not. It comes naturally to some - whatever the problem is, Bush is at fault. Doesn't require any effort or any thinking at all. Saves a lot of mental energy that can be spent on blaming Bush for more stuff.
The whole F&F operation was a poorly executed screwup. It should have never happened and whoever thought it is a good idea should be reassigned to issuing parking tickets. However, the worst part is not this - anybody who follows the current events knows that for government projects screwup is a normal mode of operation. The worst is that AG Holder lied to the Congress about it - brazenly, openly and with full understanding that he will get away with it - and with President's help he did. For me and you, lying under oath would mean jail time. For him, according to you, it's just fine and if anything is wrong, the Republicans are at fault anyway.
>>> The especially ironic thing is that since the witch hunt, the ATF's seizures of weapons has dropped 90%.
Given that the purpose of F&F was specifically NOT to seize weapons that could be seized - relating the inevitable blowup of F&F to the lower seizures, is, well, you know... gah.
Now back to F&F. If you took a moment to educate yourself, you'd learn the whole point of F&F was to CIRCUMVENT the gun laws and allow the cartels to buy massive amount of guns from dealers that otherwise they would not be able to buy. The dealers signalled the ATF multiple times, but where told to shut up and sell. The rank agents asked to arrest the buyers and take the guns, but were told to stand down and not interfere since it is an operation to track the guns into Mexico, so everything is going fine and the guns should "walk". In fact, there was no way to track them and no tracking actually happened - the guns just disappeared. Blaming gun laws for the sales that were specifically protected by the government agency whose purpose is to monitor gun sales - well, yes, mind-bogglingly stupid. Using government agency fuckup to prove that we need more government involvement - mind-bogglingly stupid. Trying to sell guns to Mexico without actually being able to find them and without telling anybody on Mexican side a thousand guns is heading their way - mind-bogglingly stupid. Blaming Republicans for the mess - I'd say it takes a lot of effort to be this stupid, but apparently it does not. It comes naturally to some - whatever the problem is, Bush is at fault. Doesn't require any effort or any thinking at all. Saves a lot of mental energy that can be spent on blaming Bush for more stuff.
The whole F&F operation was a poorly executed screwup. It should have never happened and whoever thought it is a good idea should be reassigned to issuing parking tickets. However, the worst part is not this - anybody who follows the current events knows that for government projects screwup is a normal mode of operation. The worst is that AG Holder lied to the Congress about it - brazenly, openly and with full understanding that he will get away with it - and with President's help he did. For me and you, lying under oath would mean jail time. For him, according to you, it's just fine and if anything is wrong, the Republicans are at fault anyway.
>>> The especially ironic thing is that since the witch hunt, the ATF's seizures of weapons has dropped 90%.
Given that the purpose of F&F was specifically NOT to seize weapons that could be seized - relating the inevitable blowup of F&F to the lower seizures, is, well, you know... gah.
You should really read the Forbes article:
"Quite simply, there's a fundamental misconception at the heart of the Fast and Furious scandal. Nobody disputes that suspected straw purchasers under surveillance by the ATF repeatedly bought guns that eventually fell into criminal hands. Issa and others charge that the ATF intentionally allowed guns to walk as an operational tactic. But five law-enforcement agents directly involved in Fast and Furious tell Fortune that the ATF had no such tactic. They insist they never purposefully allowed guns to be illegally trafficked. Just the opposite: They say they seized weapons whenever they could but were hamstrung by prosecutors and weak laws, which stymied them at every turn. Indeed, a six-month Fortune investigation reveals that the public case alleging that Voth and his colleagues walked guns is replete with distortions, errors, partial truths, and even some outright lies. Fortune reviewed more than 2,000 pages of confidential ATF documents and interviewed 39 people, including seven law-enforcement agents with direct knowledge of the case. Several, including Voth, are speaking out for the first time."
"Quite simply, there's a fundamental misconception at the heart of the Fast and Furious scandal. Nobody disputes that suspected straw purchasers under surveillance by the ATF repeatedly bought guns that eventually fell into criminal hands. Issa and others charge that the ATF intentionally allowed guns to walk as an operational tactic. But five law-enforcement agents directly involved in Fast and Furious tell Fortune that the ATF had no such tactic. They insist they never purposefully allowed guns to be illegally trafficked. Just the opposite: They say they seized weapons whenever they could but were hamstrung by prosecutors and weak laws, which stymied them at every turn. Indeed, a six-month Fortune investigation reveals that the public case alleging that Voth and his colleagues walked guns is replete with distortions, errors, partial truths, and even some outright lies. Fortune reviewed more than 2,000 pages of confidential ATF documents and interviewed 39 people, including seven law-enforcement agents with direct knowledge of the case. Several, including Voth, are speaking out for the first time."
What is written there (esp. Fortune) is contrary to known facts and written from the words of very interested and involved people covering their asses after their operation blew up in their faces. I do not know who specifically gave the orders - was it prosecutors, ATF, or whoever else - but the fact is orders were given to not intercept illegal purchases and to let the guns "walk". From DOJ IG report:
We found that the lack of seizures and arrests was primarily attributable to the pursuit of a strategic goal shared by both the [Phoenix] ATF and the U.S. Attorney’s Office -- to eliminate a trafficking organization -- and the belief that confronting subjects and seizing firearms could compromise that goal.
This tactic was significantly different from previous tactic, e.g. in Project Gunrunner - where straw purchasers were arrested at the moment they handed off the guns. Here, they were allowed to proceed - to great dissatisfaction of the field agents. Whoever though up this was thinking that this would lead up the chain to eliminate the bigger organisation - but due to multiple failures in the execution - such as ATF agent tracking the buyer going on vacation and the buyer just disappearing with the guns - it did not happen and guns just disappeared in Mexico to the hands of drug cartels.
Nothing to do with gun laws. Anti-guns bunch would have to look in other place to feed their political agenda.
We found that the lack of seizures and arrests was primarily attributable to the pursuit of a strategic goal shared by both the [Phoenix] ATF and the U.S. Attorney’s Office -- to eliminate a trafficking organization -- and the belief that confronting subjects and seizing firearms could compromise that goal.
This tactic was significantly different from previous tactic, e.g. in Project Gunrunner - where straw purchasers were arrested at the moment they handed off the guns. Here, they were allowed to proceed - to great dissatisfaction of the field agents. Whoever though up this was thinking that this would lead up the chain to eliminate the bigger organisation - but due to multiple failures in the execution - such as ATF agent tracking the buyer going on vacation and the buyer just disappearing with the guns - it did not happen and guns just disappeared in Mexico to the hands of drug cartels.
Nothing to do with gun laws. Anti-guns bunch would have to look in other place to feed their political agenda.
Nobody is blaming Bush (protip: he is not the president any more, and even if he were, the president is not responsible for determining gun laws in Arizona.) However, the fact that a random 18-year old can buy $20,000 of guns in Arizona and just walk out the door is the real cause of the problem. The ATF, by law, can't even set up a database of who bought what gun. I am in favor of gun rights, but this is just ridiculous.
Given that the purpose of F&F was specifically NOT to seize weapons that could be seized - relating the inevitable blowup of F&F to the lower seizures, is, well, you know... gah.
Here's a deep thought for you. If "Fast and Furious" was really the cause of those guns getting to Mexico, then why are guns still getting to Mexico, now that "Fast and Furious" is over? And why did so many of them get there before the operation started?
Given that the purpose of F&F was specifically NOT to seize weapons that could be seized - relating the inevitable blowup of F&F to the lower seizures, is, well, you know... gah.
Here's a deep thought for you. If "Fast and Furious" was really the cause of those guns getting to Mexico, then why are guns still getting to Mexico, now that "Fast and Furious" is over? And why did so many of them get there before the operation started?
>>> However, the fact that a random 18-year old can buy $20,000 of guns in Arizona and just walk out the door is the real cause of the problem.
Nope, it is not. Random 18-year olds do not resell thousands of guns to Mexican drug cartels. The guys that did were known to ATF and sellers repeatedly alerted ATF - the agency that by law is designed to stop it - about them. The system worked - until it did not, because ATF thought they would be geniuses to allow the guns to "walk".
>>> The ATF, by law, can't even set up a database of who bought what gun.
By which law? This sounds like complete baloney - ATF knew who bought the guns. They just didn't intervene because this was the plan - to let the guns "walk", not because some law prevented them from knowing who bought what.
>>> If "Fast and Furious" was really the cause of those guns getting to Mexico
What you mean "if"? It is the fact that F&F was the reason of those guns getting to Mexico. If F&F was not underway, these buyers would have been stopped and the guns were confiscated before they reach Mexico.
>>> why are guns still getting to Mexico,
Because there are other ways of getting guns to Mexico, duh. You'd make a fine defense lawyer: "If my client sold these drugs on the street, as this footage and this undercover officer suggests, how comes drugs are still sold on the street when my client already had been arrested?". Too bad this defense makes no sense at all - of course ATF is not the only channel Mexican gangs can smuggle guns. That does not diminish the fact those specific guns were smuggled with full ATF cooperation and help, which should not have happened. And this does not diminish the fact AG lied about it after the fact that this ill-conceived mess of an operation was exposed. They knew it's a mess, they lied to cover their asses - and they got away with it. And Obama shares the blame for it.
Really, with this kind of argument I would not be so quick to throw around the "stupidity" word if I were you. I'd first learn to figure out difference between "X caused this specific instance of Y" and "X caused all instances of Y ever".
Nope, it is not. Random 18-year olds do not resell thousands of guns to Mexican drug cartels. The guys that did were known to ATF and sellers repeatedly alerted ATF - the agency that by law is designed to stop it - about them. The system worked - until it did not, because ATF thought they would be geniuses to allow the guns to "walk".
>>> The ATF, by law, can't even set up a database of who bought what gun.
By which law? This sounds like complete baloney - ATF knew who bought the guns. They just didn't intervene because this was the plan - to let the guns "walk", not because some law prevented them from knowing who bought what.
>>> If "Fast and Furious" was really the cause of those guns getting to Mexico
What you mean "if"? It is the fact that F&F was the reason of those guns getting to Mexico. If F&F was not underway, these buyers would have been stopped and the guns were confiscated before they reach Mexico.
>>> why are guns still getting to Mexico,
Because there are other ways of getting guns to Mexico, duh. You'd make a fine defense lawyer: "If my client sold these drugs on the street, as this footage and this undercover officer suggests, how comes drugs are still sold on the street when my client already had been arrested?". Too bad this defense makes no sense at all - of course ATF is not the only channel Mexican gangs can smuggle guns. That does not diminish the fact those specific guns were smuggled with full ATF cooperation and help, which should not have happened. And this does not diminish the fact AG lied about it after the fact that this ill-conceived mess of an operation was exposed. They knew it's a mess, they lied to cover their asses - and they got away with it. And Obama shares the blame for it.
Really, with this kind of argument I would not be so quick to throw around the "stupidity" word if I were you. I'd first learn to figure out difference between "X caused this specific instance of Y" and "X caused all instances of Y ever".
So - a Democrat administration makes a mistake - and it's the Republicans' fault. got it.
Counter to the typical view of F&F:
http://features.blogs.fortune.cnn.com/2012/06/27/fast-and-fu...
http://features.blogs.fortune.cnn.com/2012/06/27/fast-and-fu...
I would point at the campaign manager as the person who should be responsible. The Campaign Manager is the CEO, the candidate is actually the product being sold. Granted, the candidate is being sold with promise that he or she will be a good CEO, but, during the campaign the CM is the CEO.
But then he is a product that can talk back and makes his own decisions for the entire campaign.
While I get your analogy, it simply isn't the way campaigns actually work. The candidate is the CEO - he's making the big decisions based on data fed to him or her by his campaign staff.
While I get your analogy, it simply isn't the way campaigns actually work. The candidate is the CEO - he's making the big decisions based on data fed to him or her by his campaign staff.
Leaders have to allocate attention to detail based on the strategic importance of that detail. The innovative application completely supplanting traditional Election Day strike list processes? The CEO had better be assured that's going to work by several different checks, and by multiple sources. Even then he probably had better have some backups in place.
I voted for Romney and yeah, this does have me reconsidering just how competent he is.
I voted for Romney and yeah, this does have me reconsidering just how competent he is.
Are we sure that Romney thought this system was that critical? It could be reasoned, for instance, that it was more of a fallback mechanism than anything else; it's not very likely fervent Romney supporters were just going to forget about Election Day. The article says the system was conceptualized by two deputies, and complex organizations like this one have many moving parts; it is entirely plausible, imo, that Romney saw this project as a relatively minor component of a much larger organization, something nice to have but not crucial. After all, elections have been won without similar systems in place before.
I dislike Romney and did not vote for him (or Obama), but I don't think this should necessarily be seen as a major strategic failure. Was this project going to get Romney the 1 million + additional voters he needed? It's hard to think that Romney believes that.
I dislike Romney and did not vote for him (or Obama), but I don't think this should necessarily be seen as a major strategic failure. Was this project going to get Romney the 1 million + additional voters he needed? It's hard to think that Romney believes that.
Serious politicians study GOTV the way serious generals study logistics. The technical capacity to maintain a supply line is a crucial strategic detail and is carefully monitored by the most senior military officers. They don't manage it themselves but they make quite sure that it is being managed and managed properly.
Several accounts state that Orca positively thwarted traditional Election Day GOTV efforts. Campaigns think those matter enough to put considerable effort into them. So yes, this could well have made the difference, and its failure does have me rethinking Romney's actual management ability.
Several accounts state that Orca positively thwarted traditional Election Day GOTV efforts. Campaigns think those matter enough to put considerable effort into them. So yes, this could well have made the difference, and its failure does have me rethinking Romney's actual management ability.
He hired Microsoft and what sounds like a Microsoft Partner. No one gets fired for buying Microsoft.
Oh yes, people do get fired for buying microsoft. Switch industries and you'll see. Or try spending 2 years building your company knowledge base into sharepoint and you'll see.
Sharepoint. Ugh. The mandated use of Sharepoint completely impeded communication on a multi-institution task I work on -- it was clunky when up, and often not up.
The solution? Instead of the project administering Sharepoint, the admin was handed off to the division of the corporation that specializes in administering Sharepoint. (It's a very large company.) Now we have two-factor authentication and double logins to upload monthly status reports. And still nobody can actually communicate.
The solution? Instead of the project administering Sharepoint, the admin was handed off to the division of the corporation that specializes in administering Sharepoint. (It's a very large company.) Now we have two-factor authentication and double logins to upload monthly status reports. And still nobody can actually communicate.
SharePoint. Oh my head. When I saw us using that in the future ops center for Fukushima, it was the same feeling I had when I came back from sea duty and there was this thing called PowerPoint: I felt like they were asking me to put my brain in egg slicer. Giving a bunch of Marines with history degrees network and expecting them to understand version control is a step to far. Maybe in ten years. But the interface is sooooo bad, they'll never figure out how to use the one feature the system is intended to provide.
And then a 18 yo REMF downloads all your secret classified documents and uploads them to the Internet.
Due to security policies ordered by a 48 yo REMF.
Is there really an age gap in basic understanding of the Internet? I submit most 18 yo kids know as little as most 48 yo management types. Or 48 yo teachers, or 48 yo anything-other-than-hacker types. If there is any function that filters Internet literacy, it's self selection of certain people (mostly men) into CS/IT.
I'm referring to SIPRNet security policies that gave millions of its users access to the same "Top Secret" data, and where the DoD decision tree strongly correlates to seniority.
I was referring to Private Bradley Manning and Sharepoint. To be fair, it took a few years to crack the MS file encryption. But now the world has wikileaks.
Is sharepoint generally objectionable, or simply unsuited to certain environments/workloads/userbases?
SharePoint dev here. The problem is that Microsoft attempted to build the product to cover every environment/workload/userbase, and as a result fails pretty miserably at all of them.
-You want a CMS? Use SharePoint!
-How about site for delivering BI dashboards? Use SharePoint!
-Large, mostly static public facing site? Use SharePoint!
/Ugh
-You want a CMS? Use SharePoint!
-How about site for delivering BI dashboards? Use SharePoint!
-Large, mostly static public facing site? Use SharePoint!
/Ugh
That's true! I met Microsoft last year at the start of a large project for a client, and we were asking them to recommend a CMS to run in a .NET environment. They DID suggest Sharepoint. I laughed at them - "you are joking" I said. But they insisted, and we had to add Sharepoint to the shortlist of recommended CMS's just so that the client thought we considered everything their Microsoft salespeople were saying. But it was always a joke at that point.
Except Mitt Romney, apparently!
I meant the guys who he hired to take care of this stuff.
Of course, after Tuesday, they're all looking for work now.
Of course, after Tuesday, they're all looking for work now.
[deleted]
[deleted]
It would give me more pause if Mitt Romney were involved personally in this sort of a project.
What, he didn't have any say in who he hired to make this work?
The POTA doesn't actually personally work on many projects. He relies on people to get them done. If Romney couldn't get something as crucial as this right, then how the heck would he have gone running the country?
The POTA doesn't actually personally work on many projects. He relies on people to get them done. If Romney couldn't get something as crucial as this right, then how the heck would he have gone running the country?
Planet of the Apes?
The CEO of any large company is usually not personally involved in the running of day to day operations. The president is not personally involved in the administration of Social Security, Medicare, the military, or any of the other multitude of services the government provides. But at the end of the day both are held responsible for the successful functioning of their organization. How can you be expected to run a country if you can't even run a campaign?
If he can't pick a reliable deputy to manage a single IT project how could he possibly manage a government? The President has to delegate well: Mitt either couldn't or didn't place a key service under someone reliable.
If Romney has to pick n reliable deputies for various things, then for sufficiently large n, he's inevitably going to make a mistake you can pounce on, even if he's really good at picking out people. Good judgement is not an all-or-nothing dichotomy.
Oh, certainly - I would just have thought that getting out the vote is so widely recognized as a key factor in modern campaigns that he'd have put his best people on it. Would we be talking about leadership at all if, say, many of his ads were a little ham-handed?
If there is anything a president's job isn't, it is to micromanage.
That excuse covers just about anything.
There's a big difference between micromanaging and verifying that your critical infrastructure is in place when you launch.
There's a big difference between micromanaging and verifying that your critical infrastructure is in place when you launch.
I was unclear. I am saying that since it is not the job of a president to micromanage, it is legitimate to criticise him for fuckups he was not directly involved with.
If his job were micromanaging, then "Hey, he wasn't in the room" would excuse him of responsibility, but that is not the case. Since his job would be pretty much the greatest opposite of micromanaging I can conceive of, there can be no excuse. The failure of his campaign is his failure.
That still seems a bit unclear to me actually. Not sure how to effectively reword this.
If his job were micromanaging, then "Hey, he wasn't in the room" would excuse him of responsibility, but that is not the case. Since his job would be pretty much the greatest opposite of micromanaging I can conceive of, there can be no excuse. The failure of his campaign is his failure.
That still seems a bit unclear to me actually. Not sure how to effectively reword this.
Let me try:
The United States has the following federal executive departments:
* Agriculture
* Commerce
* Defense
* Education
* Energy
* Health and Human Services
* Homeland Security
* Housing and Urban Development
* Interior
* Justice
* Labor
* State Transportation
* Treasury
* Veterans Affairs
Now there is literally no way you can micromanage you way through this. Nobody, and I mean nobody can possibly single handedly do it.
However, you can delegate. Part of that is to choose honest brokers and managers who know their area well enough to ensure that their departments work well.
In this situation it is your job to delegate to the right people. If you are the President of the United States of America and you aren't able to do this with your own campaign to get elected, you must take responsibilty for your failure and accept a great deal of responsibilty for the fact you did not get elected to high office.
Harsh, but being the POTA is not a job everyone can do. I know I couldn't.
The United States has the following federal executive departments:
* Agriculture
* Commerce
* Defense
* Education
* Energy
* Health and Human Services
* Homeland Security
* Housing and Urban Development
* Interior
* Justice
* Labor
* State Transportation
* Treasury
* Veterans Affairs
Now there is literally no way you can micromanage you way through this. Nobody, and I mean nobody can possibly single handedly do it.
However, you can delegate. Part of that is to choose honest brokers and managers who know their area well enough to ensure that their departments work well.
In this situation it is your job to delegate to the right people. If you are the President of the United States of America and you aren't able to do this with your own campaign to get elected, you must take responsibilty for your failure and accept a great deal of responsibilty for the fact you did not get elected to high office.
Harsh, but being the POTA is not a job everyone can do. I know I couldn't.
But he wasn't president. Romney was managing a campaign. And a function critical to his campaign - you gonna tell me he wasn't involved? Or at least briefed?
I mean, wasn't he expecting to see the results?
I mean, wasn't he expecting to see the results?
Yes, that is the sentiment I was trying to convey.
[deleted]
"The buck stops...somewhere else."
It all trickles down, right?
No one should allow a business or non-profit to face a situation like this. Only bad managers put something so important into such difficult circumstances. Why go live the day of the election? Are they insane?
Unless the contract (to build this application) had some kind of clause like "This thing must work perfectly or all the money will be refunded" then this is a great contract to get. Let us consider this cynically for a moment. I wish I could land contracts like this one. Can you imagine building an app that the client will only need for 1 day, and you still get paid even if the thing does not work? The chance for fraud is huge. Who would even allow this? The people pitching this must have been some very talented salespeople (perhaps they were convincingly able to argue that they were such huge Republican supporters that if they lost then that would be punishment enough, as it would be so very sad for them to see Obama win?)
I've worked with a lot of startups and clients and the staff always needs training and you never, ever know what all the bugs are until you've gotten some feedback from the staff who are using it for real. To suggest that a software project is done the day you first give it to the staff is idiotic -- that's simply the moment you move to real world testing.
Some developers might suggest that a really good suite of unit tests might provide enough insurance, but that is not true -- I've worked with non-technical staff who tend to confuse bad UI decisions with software bugs, and if a UI decision is bad enough, I think it is fair to classify it as a bug. If some_important_action is crucial to the staff, and none of the staff can find the link to some_important_action, then the UI decision that hid it is effectively as bad as a Fatal Exception, but no amount of unit tests can tell you that the staff finds the UI incomprehensible.
Unless the contract (to build this application) had some kind of clause like "This thing must work perfectly or all the money will be refunded" then this is a great contract to get. Let us consider this cynically for a moment. I wish I could land contracts like this one. Can you imagine building an app that the client will only need for 1 day, and you still get paid even if the thing does not work? The chance for fraud is huge. Who would even allow this? The people pitching this must have been some very talented salespeople (perhaps they were convincingly able to argue that they were such huge Republican supporters that if they lost then that would be punishment enough, as it would be so very sad for them to see Obama win?)
I've worked with a lot of startups and clients and the staff always needs training and you never, ever know what all the bugs are until you've gotten some feedback from the staff who are using it for real. To suggest that a software project is done the day you first give it to the staff is idiotic -- that's simply the moment you move to real world testing.
Some developers might suggest that a really good suite of unit tests might provide enough insurance, but that is not true -- I've worked with non-technical staff who tend to confuse bad UI decisions with software bugs, and if a UI decision is bad enough, I think it is fair to classify it as a bug. If some_important_action is crucial to the staff, and none of the staff can find the link to some_important_action, then the UI decision that hid it is effectively as bad as a Fatal Exception, but no amount of unit tests can tell you that the staff finds the UI incomprehensible.
>I wish I could land contracts like this one. Can you imagine building an app that the client will only need for 1 day, and you still get paid even if the thing does not work? The chance for fraud is huge. Who would even allow this? The people pitching this must have been some very talented salespeople...
Anecdotally, this is exactly how large consultancies, particularly those who specialize in serving the public sector, operate.
1) Deploy an army of "talented" salespeople to establish favorable executive-level relationships by exploiting every available loophole to provide trips, gifts, dinners and sporting events. Once the top has been won over, everyone below will fall in line.
2) Write contracts meeting the barest minimum interpretation of the requirements. Design them to specifically exploit uninformed procurement departments which are better equipped to source millions in copier paper or bolts than complex software.
3) Farm out any actual deliverable to the cheapest sub available nationwide. Ideally, you will have hidden travel and per diem for the talent in #2 leaving you free to fly in folks from the cheapest parts of the country.
4) Upon encountering any serious issue once work is underway simply scapegoat the current PM and shuffle in a new one.
5) Go to #1 employing the latest project as a reference that likely won't even be properly checked. In the event that diligence is performed, feed them to one of your friendly executives.
Anecdotally, this is exactly how large consultancies, particularly those who specialize in serving the public sector, operate.
1) Deploy an army of "talented" salespeople to establish favorable executive-level relationships by exploiting every available loophole to provide trips, gifts, dinners and sporting events. Once the top has been won over, everyone below will fall in line.
2) Write contracts meeting the barest minimum interpretation of the requirements. Design them to specifically exploit uninformed procurement departments which are better equipped to source millions in copier paper or bolts than complex software.
3) Farm out any actual deliverable to the cheapest sub available nationwide. Ideally, you will have hidden travel and per diem for the talent in #2 leaving you free to fly in folks from the cheapest parts of the country.
4) Upon encountering any serious issue once work is underway simply scapegoat the current PM and shuffle in a new one.
5) Go to #1 employing the latest project as a reference that likely won't even be properly checked. In the event that diligence is performed, feed them to one of your friendly executives.
That explains what I saw on one of my past projects, including replacing PM, travel&per diem, and presenting failed projects as a good reference.
Was this a talented sales pitch or just your standard politics and an old boys club connection? No mention of the consulting company.. If it was my campaign, I'd hang them out to dry.
Seems the IT staff was out of their depth but there were some very serious and very fundamental issues if the users were confused as it sounds.
I say it as a joke but a barbed one: this guy thinks he was fit to manage a nation when he couldn't knock out a web app for 30000 users!?! Really?
Seems the IT staff was out of their depth but there were some very serious and very fundamental issues if the users were confused as it sounds.
I say it as a joke but a barbed one: this guy thinks he was fit to manage a nation when he couldn't knock out a web app for 30000 users!?! Really?
Fortunatey, it's not the president's job to "manage a nation."
As a side note, an almost do-nothing president would be the best kind. (And, as a side note to the side note, that was what was intended when the office was created.)
As a side note, an almost do-nothing president would be the best kind. (And, as a side note to the side note, that was what was intended when the office was created.)
"If it was my campaign, I'd hang them out to dry."
I doubt Romney even knew the details of this mess. I'd be surprised if he knows it now.
As is typical of many "high level" exec's; Romney relied on the wrong people.
I doubt Romney even knew the details of this mess. I'd be surprised if he knows it now.
As is typical of many "high level" exec's; Romney relied on the wrong people.
And it is worse in the US because the client is the Romney organisation not the Republican Party - and it definately wont be around next time to learn the lesson.
>"We asked if our laptops needed to be WiFi capable," Dittuobo told Ars. "Dan Centinello went into how the Garden had just finished expansion of its wireless network and that yes, WiFi was required. I was concerned about hacking, jamming the signal,* etc...*
This brings up a whole nother can of worms - electoral cyber warfare. What if the Dems, or some sympathizer group, say some local Anonymous, decided to disrup the GOP's centralized GOTV operation?
They could point LOIC at the IP receiving all the real-time field reports, set up hidden wireless jammers around Boston Garden, and any number of other hacks and exploits to disrupt this centralized system.
I use the GOP and Anonymous as an example here, but it could work in any direction. It's one thing for a campaign to lose due to its own incompetence and negligence, but quite another to lose due to outside interference and being the target of cyberwarefare.
This brings up a whole nother can of worms - electoral cyber warfare. What if the Dems, or some sympathizer group, say some local Anonymous, decided to disrup the GOP's centralized GOTV operation?
They could point LOIC at the IP receiving all the real-time field reports, set up hidden wireless jammers around Boston Garden, and any number of other hacks and exploits to disrupt this centralized system.
I use the GOP and Anonymous as an example here, but it could work in any direction. It's one thing for a campaign to lose due to its own incompetence and negligence, but quite another to lose due to outside interference and being the target of cyberwarefare.
Or what if the ones implementing the system were the 'sympathizer group'. They wouldn't have to do any sort of jamming or anything, just give out insufficient instructions in how to use the system, incorrect passwords and make sure the system crashes a bunch.
Intercepting field reports? By the sounds of it, Team Obama had all the info they needed and this just opens the door to serious legal problems.
In general I haven't encountered too many Republicans in software development. Libertarians, yes. Republicans, no.
Intercepting field reports? By the sounds of it, Team Obama had all the info they needed and this just opens the door to serious legal problems.
In general I haven't encountered too many Republicans in software development. Libertarians, yes. Republicans, no.
This is a great point, and it's one of the more frustrating things to encounter as a developer who knows more than just "the language I'm writing a website in". Security is not optional and security must be managed at every level of the infrastructural stack.
But that's hard, and nobody's going to know anyway, so let's just roll with it.
But that's hard, and nobody's going to know anyway, so let's just roll with it.
Politico's article on ORCA today claims that one of the major problems was that they deployed the system in their election night HQ just prior to the election, and the surge of legitimate traffic on election day triggered their DDOS defences. They'd never tested it in that location with realistic traffic levels.
Deployed just prior: poor project management; late delivery.
DDOS defense: crashing
realistic traffic levels: more than the 3 ppl on the coding team
DDOS defense: crashing
realistic traffic levels: more than the 3 ppl on the coding team
with one webserver they were their own worst enemy; no need to look elsewhere
If a hacker looks into their web and db infrastructure...I think he would just shake his head and move on.
How effective these things actually are in US?
Why am I asking... I am in from Czech Republic, where I can't imagine anyone knocking door over door and asking people to vote. Actually, this behavior is so connected here to Jehova's Witnesses (yeah) that it would have only the negative effect.
Similarly, I can't imagine any candidate seriously considering calling random people and persuading them to vote. Again, this behaviour is so connected to annoying salesperson (most often mobile operators) that it would hurt the candidate.
And I don't think our democracy suffers because we don't have that in here.
Why am I asking... I am in from Czech Republic, where I can't imagine anyone knocking door over door and asking people to vote. Actually, this behavior is so connected here to Jehova's Witnesses (yeah) that it would have only the negative effect.
Similarly, I can't imagine any candidate seriously considering calling random people and persuading them to vote. Again, this behaviour is so connected to annoying salesperson (most often mobile operators) that it would hurt the candidate.
And I don't think our democracy suffers because we don't have that in here.
Our land-line telephone received at least six calls each day for two months, from political organizations representing candidates at the federal, state, and local levels. We never answer that phone, only let the answering machine take the call and return it in the very rare event it is someone "real". But we did not get calls on election day because we live in a state (CA) that both parties assumed (correctly) would give a majority to the President.
"Get out the vote" operations (GOTV) are real and sometimes effective in states where the outcome is expected to be close. In this election there were a small number of these "swing" states where both parties thought they had a chance to win.
In such states, if the early indications are that the voting is close, the party organization can redouble its efforts to contact known supporters (contributors, or merely people who are registered to that party in the public records, or people who have signed a petition or gotten on the party mailing list somehow) and persuade them to vote.
(This is because in the USA, unlike some countries, it is not mandatory to vote, and rarely do even half the eligible voters do so. So if you can persuade a person who was not planning to vote, to do so, that is a net gain of 1 vote.)
The purpose of this "Orca" system was to collate the reports from volunteers watching the polling places and identify precincts where a few extra votes might make a difference.
You can see how serious the effort was, in that the Republicans had mustered 37,000 volunteers willing to do this sort of thankless work: to stand around polling places, politely asking people how they had voted and phone in the counts; or, sit in an office and make telephone calls to people who do not want to be called and politely urge them to go and vote. That is a large effort, and apparently wasted by poor planning.
"Get out the vote" operations (GOTV) are real and sometimes effective in states where the outcome is expected to be close. In this election there were a small number of these "swing" states where both parties thought they had a chance to win.
In such states, if the early indications are that the voting is close, the party organization can redouble its efforts to contact known supporters (contributors, or merely people who are registered to that party in the public records, or people who have signed a petition or gotten on the party mailing list somehow) and persuade them to vote.
(This is because in the USA, unlike some countries, it is not mandatory to vote, and rarely do even half the eligible voters do so. So if you can persuade a person who was not planning to vote, to do so, that is a net gain of 1 vote.)
The purpose of this "Orca" system was to collate the reports from volunteers watching the polling places and identify precincts where a few extra votes might make a difference.
You can see how serious the effort was, in that the Republicans had mustered 37,000 volunteers willing to do this sort of thankless work: to stand around polling places, politely asking people how they had voted and phone in the counts; or, sit in an office and make telephone calls to people who do not want to be called and politely urge them to go and vote. That is a large effort, and apparently wasted by poor planning.
There are very few countries where voting is mandatory, perhaps none. And still The USA has lower voter turnout than most, probably more because of the separate registration step than anything else.
But the biggest difference is the per-state first pas the post mechanism, which is what creates the phenomenon of "swing states" where relatively few voter decide everything, and almost any expense of effort by campaigns is worthwhile.
But the biggest difference is the per-state first pas the post mechanism, which is what creates the phenomenon of "swing states" where relatively few voter decide everything, and almost any expense of effort by campaigns is worthwhile.
Voting is mandatory in Australia: https://en.wikipedia.org/wiki/Electoral_system_of_Australia#...
Historically intensive voter contact operations are only cultural acceptable in countries that weren't largely occupied by either the Fascists or the Communists.
So basically it is UK (less Northern Ireland), US, Canada, Australia and a few others. An Italian friend once said to me that if anyone knocked on his door asking how he voted he would stall them at the door, whilst trying to jump out the back window and get away over the hills...
So basically it is UK (less Northern Ireland), US, Canada, Australia and a few others. An Italian friend once said to me that if anyone knocked on his door asking how he voted he would stall them at the door, whilst trying to jump out the back window and get away over the hills...
From Slate back in July: an interesting background piece about the campaign's approach to staffing for data analysis.
http://www.slate.com/articles/news_and_politics/victory_lab/...
What the debacle brings to mind is the consequences of putting "idea guys" in charge of software development when there is a hard deadline.
"Orca had been conceived by two men—Romney's Director of Voter Contact Dan Centinello and the campaign's Political Director Rich Beeson."
I imagine them saying, "We shall have an app," while thinking, "How hard can it be?" While the developers were rationalizing, "Well, a web app is technically an app...and a lot easier to complete than an iPhone app, an Android app, and the web app we will still need to build for the old people with a Windows desktop."
http://www.slate.com/articles/news_and_politics/victory_lab/...
What the debacle brings to mind is the consequences of putting "idea guys" in charge of software development when there is a hard deadline.
"Orca had been conceived by two men—Romney's Director of Voter Contact Dan Centinello and the campaign's Political Director Rich Beeson."
I imagine them saying, "We shall have an app," while thinking, "How hard can it be?" While the developers were rationalizing, "Well, a web app is technically an app...and a lot easier to complete than an iPhone app, an Android app, and the web app we will still need to build for the old people with a Windows desktop."
A Web App is actually the right call. It was for volunteers doing GOTV in the field, not as much for people sitting at a desktop. And you would want to be able to issue updates quickly and carefully control access -- things that are much more difficult with an App Store app.
Using Windows was a mistake, that's for sure.
Good thing for Democrats they didn't know about the open source ushahidi platform.
Good thing for Democrats they didn't know about the open source ushahidi platform.
NGP-VAN, which is probably the most important company in the Democratic sphere, uses a Microsoft stack. And it didn't lead to a spate of post-election recriminations about how its tech fell apart.
Using the MS stack for a situation where you will have huge spikes of activity on an app with an expected lifespan of less than 24 hours is not best practice.
I'm an open source developer – working on Linux servers since 1997 or so, etc. etc. Please stop what you're doing: it's not very funny and it's not helpful, but it does give OSS people a bad reputation.
You can host plenty of high volume things on Microsoft's code – Azure, Bing, etc. have proven scale. We can argue about cost, ease of use, etc. but a decent team should have no trouble making a serious service. The problem here is that they didn't hire a decent team – from the sounds of it, it was the usual executive-level schmoozing and managerial incomplete you see throughout large organizations of every type. They could have used HN'a favorite stack and failed just as hard.
You can host plenty of high volume things on Microsoft's code – Azure, Bing, etc. have proven scale. We can argue about cost, ease of use, etc. but a decent team should have no trouble making a serious service. The problem here is that they didn't hire a decent team – from the sounds of it, it was the usual executive-level schmoozing and managerial incomplete you see throughout large organizations of every type. They could have used HN'a favorite stack and failed just as hard.
This comment reminds me of AL Gore lecturing Frank Zapa.
Hey, dogs can swim but that doesn't mean I use them to pull a barge up the river.
I will not shut up about the ground truth I and many other people know about Microsoft and their software.
I will not shut up about the ground truth I and many other people know about Microsoft and their software.
The way you're doing it won't will anyone over. Open source has been winning where I work because we've demonstrated better results with less cost. I suggest you try to outperform rather than out-market.
A poor craftsman blames their tools.
How did you determine using Windows was a mistake from the information presented in the article ?
Cause it's a bias of his, likely.
What I get from the article is a classic tale of crappy IT management, the like of which I've seen and heard of several times.
What I get from the article is a classic tale of crappy IT management, the like of which I've seen and heard of several times.
"Using Windows was a mistake" is a truism.
Also, from the article, project failed. Failure is regularly equated with having made mistakes.
Also, from the article, project failed. Failure is regularly equated with having made mistakes.
In the same vein, one could claim use of humans to write code was a mistake. After all, the project failed, this means they made a mistake, so obviously they should have used trained hamsters instead. Do you see where your logic failed you?
> "Using Windows was a mistake" is a truism.
No. No, it is not.
> Failure is regularly equated with having made mistakes.
And what mistake did Windows make here? Keep the baseless insults inside your head, please.
No. No, it is not.
> Failure is regularly equated with having made mistakes.
And what mistake did Windows make here? Keep the baseless insults inside your head, please.
> "Using Windows was a mistake" is a truism
If you use or have used StackOverflow or Newegg, realize that you too are using Windows. Projects' success and failure is much more correlated with the folks running them than the technologies they use.
If you use or have used StackOverflow or Newegg, realize that you too are using Windows. Projects' success and failure is much more correlated with the folks running them than the technologies they use.
A: ungraceful crash under load
Sorry, a database-backed web app works just fine on a Microsoft stack, there are thousands of them out there and I say that as someone who vastly prefers unix but this is clearly a project management fiasco and architecture design fiasco, not a technology stack fundamentally unable to handle the demands.
I agree the MS stack works fine in the dev lab.
Stack Overflow (you may have heard of them) runs a Windows stack with Linux used only for caching & logging (because it's "cheap").
http://blog.stackoverflow.com/2008/09/what-was-stack-overflo... http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
http://blog.stackoverflow.com/2008/09/what-was-stack-overflo... http://highscalability.com/blog/2011/3/3/stack-overflow-arch...
Which also can happen with a terribly coded application running on Linux. Or with a terribly configured apache server.
This was a meatspace failure, more than anything else.
This was a meatspace failure, more than anything else.
You can fail on any platform with bad implementation and/or bad management and/or bad planning. That's the lesson here.
(and at Twitter when they fail whaled for months on end with Rails - it wasn't Rails fault - it was the application of Rails in too high a volume deployment)
(and at Twitter when they fail whaled for months on end with Rails - it wasn't Rails fault - it was the application of Rails in too high a volume deployment)
Why do you choose to throw out an insult at Windows? Is there anything about this story that points to Windows being at fault? How about you think before you flame next time, okay?
I have been a professional webmaster since 1995. I have worked for California's largest privately held webhosting company. I am an MCP in Windows server and have been the only guy on staff willing to support customers with IIS.
I base my statement partly on that experience.
I base my statement partly on that experience.
Stackoverflow crashes daily
Don't take it personal. Inferior products need support, too. You will always be able to find work making IIS sites as long as MS is selling server software.
Looks like someone has a bit of an emotional investment on a tool, eh?
Nope, I just strongly dislike tangential and unfounded insults.
"A Web App is actually the right call."
Yes.
Calling it "an app" however was a bad call. From a practical standpoint, that is. From the standpoint of "an idea guy" and a "communications expert" it was of course a requirement.
Yes.
Calling it "an app" however was a bad call. From a practical standpoint, that is. From the standpoint of "an idea guy" and a "communications expert" it was of course a requirement.
What else do you call it? A "mobile website"? I don't think that's the real problem... the real problem is none of the users actually had an opportunity to use prior to it being absolutely mission critical.
"It's a website: http://orca.mittromney.com. It works on smartphones."
But so much can go wrong with an app, especially in the field. Give every team leader a fax machine, a phone, and some pens.
There's nothing wrong with web app over mobile app. Actually, recently here on HN was a number of articles about coming trend of switching from native apps to HTML5/web apps. It is a valid choice, especially when you need to communicate a lot of data, so you need network connection anyway.
Of course, that web app needs to be working - but the choice itself is completely fine. Given the circumstances, I would seriously consider going the same direction if I had to do an app like that.
Of course, that web app needs to be working - but the choice itself is completely fine. Given the circumstances, I would seriously consider going the same direction if I had to do an app like that.
Some key points -
1.) 11 database servers. 1 app server 2.) Not redirecting http to https 3.) not stress tested or apparently tested in any way 4.) users got their first taste of the system the morning of the election.
1.) 11 database servers. 1 app server 2.) Not redirecting http to https 3.) not stress tested or apparently tested in any way 4.) users got their first taste of the system the morning of the election.
Sounds like Microsoft was trying to sell a few SQL server licenses.
The assumption that software can work from the first try is proven wrong, time and again. Software is something that needs to grow, mature and stabilize. The Romney campaign had the thing that's usually hardest to get - the human beta test resources. It should have been dry run for months.
The campaign could not manage launching a web app and they wanted to run the entire USA !?!?!?!
They already do. Federal government regularly fails spectacularly and it regularly costs us taxpayers billions. And so do local, state, municipal, etc. governments. Presenting it as if this is some unheard of failure that disqualifies everybody involved including the candidate is silly. A lot of successful websites failed, many multiple times and for prolonged period, despite employing top talent and spending millions on them. Failure is always an option.
Yes, people who make mistakes can still run the entire USA. It has been done before.
This kind of false equivocation is dishonest and intellectually lazy.
I disagree. His point is very valid.
His use of "mistakes" is a verbal fallacy called equivocation. Not all mistakes are the same.
wikipedia: Equivocation ("to call by the same name") is classified as both a formal and informal logical fallacy. It is the misleading use of a term with more than one meaning or sense
wikipedia: Equivocation ("to call by the same name") is classified as both a formal and informal logical fallacy. It is the misleading use of a term with more than one meaning or sense
This "mistake" is what most people would call 'utter incompetence'.
I don't know about you but there were days when I was trying to get a web app to support IE6 that I was envious of how easy the president has it.
Heck, I remember a Republican candidate who was amazed by the scanner at the supermarket checkout.
You remember a report. Or a report of a report. The reporter wasn't even there.
http://www.snopes.com/history/american/bushscan.asp
http://www.snopes.com/history/american/bushscan.asp
If you're the prez and express a significant reaction to common scanner technology that's 12 years old, that's news, whether it's "amazed" or "curious".
Aren't you amazed by the scanner at a supermarket checkout? Omnidirectional laser scanners are a thing of beauty. (Yes I'm serious).
We're all lucky that an organization too inept to manage a campaign is ultimately not allowed to manage the country.
Coming from a launch that had its own share of issues, I can say that things like these happen for a variety of reasons:
1. Developers overestimate their capabilities 2. Developers underestimate timelines
3. Client overestimates developer throughput 4. Client underestimates man hours required for any given feature.
5. Someone expects someone else to do QA. 6. Project manager assigns everything an ASAP priority
And this is just to start.
1. Developers overestimate their capabilities 2. Developers underestimate timelines
3. Client overestimates developer throughput 4. Client underestimates man hours required for any given feature.
5. Someone expects someone else to do QA. 6. Project manager assigns everything an ASAP priority
And this is just to start.
Perhaps Republicans should have spent more time actually displaying the business and organizational acumen and discipline that they claimed as an advantage over the Obama team, and less time suppressing the black vote.
Snark postscriptum: Also, next time, GOP, you might wanna hire some IT guys who make more than $12 an hour.
Snark postscriptum: Also, next time, GOP, you might wanna hire some IT guys who make more than $12 an hour.
Another misstep of the Romney camp seems relevant, since a lot of the discussion so far is about project management.
In September they admitted to wholesale copying of a key Obama fundraising page, with Zac Moffat blaming the issue on "junior staff confusion":
http://www.buzzfeed.com/zekejmiller/romney-campaign-appears-...
The copy was lifted in its entirely. The similarity in the design is more subjective. (They were so similar that a lot of the discussion online speculated about a shared vendor -- which isn't the case. Quick Donate was a custom product internal to the Obama campaign.)
Romney campaign seems to have had serious problems in a lot of their internal processes when it came to tech/web. (AMERCIA)
In September they admitted to wholesale copying of a key Obama fundraising page, with Zac Moffat blaming the issue on "junior staff confusion":
http://www.buzzfeed.com/zekejmiller/romney-campaign-appears-...
The copy was lifted in its entirely. The similarity in the design is more subjective. (They were so similar that a lot of the discussion online speculated about a shared vendor -- which isn't the case. Quick Donate was a custom product internal to the Obama campaign.)
Romney campaign seems to have had serious problems in a lot of their internal processes when it came to tech/web. (AMERCIA)
In the past week, there's been discussion of how sophisticated the campaigns are at managing and leveraging the vast volumes of data they collect. I've argued that while they may have lots of data, and that they think they're being sophisticated with it, the results don't seem to indicate that they're particularly clever (or, more importantly, clever enough to be devious with it).
Having access to data is one thing...having the logistics and middle managers who have good insights is the other. I think it's safe to say that no matter how much money and resources these operations have, it doesn't mean anything without campaign operatives who have a real IT/data background.
Having access to data is one thing...having the logistics and middle managers who have good insights is the other. I think it's safe to say that no matter how much money and resources these operations have, it doesn't mean anything without campaign operatives who have a real IT/data background.
It's too bad the Romney campaign was unable to tap the services of an experienced businessman who is expert at management and delegation.
>To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm.
Lawsuit incoming in 3...2...1...
I can't imagine the billionaires who funded the campaign and the various Super PACs are going to be pleased to find out Romney likely lost because of a failed GOTV effort.
Lawsuit incoming in 3...2...1...
I can't imagine the billionaires who funded the campaign and the various Super PACs are going to be pleased to find out Romney likely lost because of a failed GOTV effort.
It's a stretch to think that a functioning mobile app on election day would have made the difference between Romney winning and losing. The election wasn't really that close. And I say that as someone who voted for him.
It's not a stretch to think that a functioning GOP GOTV campaign could have made the difference in 4 swing states that tipped the election to Obama (CO, FL, OH, VA), where the vote differential was just under 400,000.
http://fivethirtyeight.blogs.nytimes.com/2012/11/08/as-natio...
https://docs.google.com/spreadsheet/ccc?key=0Au5WjgbQnYrSdFd...
Romney had 37,000 field volunteers, plus phone bank. I don't know how many were in each of the four tipping point states, but had each of them been able to bring in between 10 and 20 voters who didn't vote (of the 2 million extra who voted for McCain in 2008 but were absent this year), Romney could have won.
Instead it sounds like due to the massive Orca failure, they all gave up and went home.
(and fwiw, I say this as someone who voted for Obama. Not sour grapes here, just assessing the IT project failure.)
http://fivethirtyeight.blogs.nytimes.com/2012/11/08/as-natio...
https://docs.google.com/spreadsheet/ccc?key=0Au5WjgbQnYrSdFd...
Romney had 37,000 field volunteers, plus phone bank. I don't know how many were in each of the four tipping point states, but had each of them been able to bring in between 10 and 20 voters who didn't vote (of the 2 million extra who voted for McCain in 2008 but were absent this year), Romney could have won.
Instead it sounds like due to the massive Orca failure, they all gave up and went home.
(and fwiw, I say this as someone who voted for Obama. Not sour grapes here, just assessing the IT project failure.)
The result of Davies fieldwork in UK elections showed that active local party activities (including GOTV) can be worth up to a 2%-4% swing - this is from an election where 2nd generation electoral computing systems were used.
In Scotland we have 3rd generation systems (one system for all elections vertically, local government, Scottish Parliament, Westminster Parliament, European Parliament, and horizontally, 2011, 2008, 2007 back to the 80s).
The US parties be roughly considered as having 2 and half generation systems, so you might expect a difference at the top end of that.
I don't follow US elections that closely but you can work the numbers out.
In Scotland we have 3rd generation systems (one system for all elections vertically, local government, Scottish Parliament, Westminster Parliament, European Parliament, and horizontally, 2011, 2008, 2007 back to the 80s).
The US parties be roughly considered as having 2 and half generation systems, so you might expect a difference at the top end of that.
I don't follow US elections that closely but you can work the numbers out.
[deleted]
Except that there are several reasons Romney lost. The system crash may be one contributing factor, but it is far from the only reason his campaign was ultimately unsuccessful.
Someone told me that Accenture build it in India - poetic justice, I'd think, if that was indeed true.
[deleted]
This is an absolutely perfect use case for cloud hosting like Azure - dev local, test on a few small instances, scale up as needed on election day, then shut it down. From the article it sounds like they paid for eleven servers, which was significantly more expensive and lower powered.
This wouldn't have helped their training problem however - volunteers confused about how to find the site, not sure how to use it, etc.
Several dry runs and stress tests would have helped them alot.
Several dry runs and stress tests would have helped them alot.
11 DB servers and 1 web app? Let's look at how the db was structured. Wait...nevermind forget it.
I think there is a connection between the sort of hand-waving, know nothing-ism of Republican political messaging and this tech fiasco.
That kind of thing might work in the soft field of persuasion, but is pretty useless in creating a complex technical system. (By know nothing-ism of the Republicans, I'm referring to the birtherism, beligerent rhetoric about complicated diplomatic issues, anti-science views about global warming, trying to paint Obama as a socialist, in general the whole "Guns, God, & Gays" type of messaging).
That kind of thing might work in the soft field of persuasion, but is pretty useless in creating a complex technical system. (By know nothing-ism of the Republicans, I'm referring to the birtherism, beligerent rhetoric about complicated diplomatic issues, anti-science views about global warming, trying to paint Obama as a socialist, in general the whole "Guns, God, & Gays" type of messaging).
This is one area you don't want to hack a solution quickly, but want to use something that's been tried and tested in the field. You can be on the bleeding edge when you have time to iterate and fix things, not when you have just one shot. When it comes to having just one shot no one does it better than NASA - http://www.theverge.com/2012/8/9/3232160/curiosity-mastcam-2...
[deleted]
Faith based development.
1. They should have hired best UX engineers for the design. 2. Hosted the solution in the cloud and called and made sure the host is aware of the upcoming traffic. 3. Rolled out the application in test mode at least a week before so that staff can do trail runs. 4. Worry less about hacking and made it easy to login (just via an ID) ~ me being Captain Hindsight.
I'm not so sure. Intentionally disruptive behavior by the opposition -- or even some motivated kid who wanted to cause trouble for the campaign -- would be one of my biggest concerns if I were designing this thing.
As a conservative and a software engineer, it's heartening that they at least had a plan and tried something. The fact that it was poorly implemented and unstable doesn't surprise me. Most last minute projects end up that way.
i find it interesting that they named it Orca as Obamas system was called Narwhal.
for those of you that dont know, Narwhal is an animal, but its also a internet meme. Obviously a very nerdy reference by Obamas team.
The fact that the Romney team had to name their software as a killer for a Narwhal goes to show that they wanted to win at any cost, unfortunately when that happens you sometimes lose sight of the goal, which in this case is to have a working system.
for those of you that dont know, Narwhal is an animal, but its also a internet meme. Obviously a very nerdy reference by Obamas team.
The fact that the Romney team had to name their software as a killer for a Narwhal goes to show that they wanted to win at any cost, unfortunately when that happens you sometimes lose sight of the goal, which in this case is to have a working system.
Or it means that whoever came up with the name was aware of Narwhal and thought it fitting that their system should also be named after a cool-looking whale. If this had not been a political issue, would you have been slower to reject this obvious innocent interpretation?
they mentioned in the piece that it was named as an animal that is the only natural predator of the Narwhale.
Please read the article before you downvote or comment.
Please read the article before you downvote or comment.
[deleted](2)
To build Orca, the Romney campaign turned to Microsoft and an unnamed application consulting firm.
This sounds like a lot of failed IT projects in large corporations -- dreamed up by upper management types who don't know anything about tech, then farmed out to consultants and tech vendors for the actual implementation. The consultants and vendors then either proceed to wander in the weeds expensively due to lack of direction, or actively seize on the client's ignorance and take them for a ride.
The problem is that the people nominally in charge of the project, the upper management types, don't have the experience or expertise to know they're getting taken for a ride until it's too late. So there's no way to hold the contractors accountable, or to get the train back on the rails once it's jumped off.