The Programmer's Dilemma(raganwald.posterous.com)
raganwald.posterous.com
The Programmer's Dilemma
http://raganwald.posterous.com/the-programmers-dilemma-95099
16 comments
I've heard "Doing Agile Wrong" too, however I think what the post is talking about is when the participants accuse each other of doing it wrong, which is different than when an outsider says an entire project was done wrong.
So for example, if I say "We tried Scrum, and it didn't work," and some evangelist tries to defend the project and says "You did it wrong," that is not the same as when I say, "We tried Scrum, our Team was fine, but the Product Owner did it wrong and that's why we failed."
I can't speak to what you've heard, but I certainly have heard a lot of the former, to the point where I reply to all such discussions with a pointer to the No True Scotsman fallacy:
"Agile works."
"No it doesn't, we tried it and failed."
"You did it wrong."
Compare and contrast to:
"Scotsmen are cheap bastards, Ne'er would you catch one standing a round."
"Hamish just bought me a dram of Lagavulin."
"Hamish may have Scots parents, but he lives in Toronto. No True Scotsman would stand a round."
So for example, if I say "We tried Scrum, and it didn't work," and some evangelist tries to defend the project and says "You did it wrong," that is not the same as when I say, "We tried Scrum, our Team was fine, but the Product Owner did it wrong and that's why we failed."
I can't speak to what you've heard, but I certainly have heard a lot of the former, to the point where I reply to all such discussions with a pointer to the No True Scotsman fallacy:
"Agile works."
"No it doesn't, we tried it and failed."
"You did it wrong."
Compare and contrast to:
"Scotsmen are cheap bastards, Ne'er would you catch one standing a round."
"Hamish just bought me a dram of Lagavulin."
"Hamish may have Scots parents, but he lives in Toronto. No True Scotsman would stand a round."
So if I'm reading this right:
If a project that uses Agile fails, the participants have no incentive to blame each other.
If a project that uses Waterfall fails, the participants have an incentive to blame each other.
Because Waterfall in the failure case gives people an incentive to blame each other, it's a poorer methodology than Agile.
If I haven't confused your argument, then the weakness is that you have not shown that Agile is a methodology in which people have no incentive (or ability) to blame each other if the project fails.
If a project that uses Agile fails, the participants have no incentive to blame each other.
If a project that uses Waterfall fails, the participants have an incentive to blame each other.
Because Waterfall in the failure case gives people an incentive to blame each other, it's a poorer methodology than Agile.
If I haven't confused your argument, then the weakness is that you have not shown that Agile is a methodology in which people have no incentive (or ability) to blame each other if the project fails.
My claim is that if a project that uses Agile fails, the participants still have an incentive in dysfunctional cultures to blame each other, it's just that the structure of Agile makes it harder to blame individuals. So I am not claiming that A is better than W or W is better than A at shipping software, I am claiming that W is better than A at blaming people for failures.
I might have confused the issue by trying to draw a distinction between blaming the process from the inside and from the outside. My claim is that it is much easier to blame a process if you aren't a participant in a failing process. For this reason, it is easier to evangelize a new process of any type as a consultant or as a new face than it is if you are an old hand who may be accused of having been a cause of previous failures.
I think this second point is true of any process.
I might have confused the issue by trying to draw a distinction between blaming the process from the inside and from the outside. My claim is that it is much easier to blame a process if you aren't a participant in a failing process. For this reason, it is easier to evangelize a new process of any type as a consultant or as a new face than it is if you are an old hand who may be accused of having been a cause of previous failures.
I think this second point is true of any process.
Can you explain how the structure of Agile makes it harder to blame individuals?
Perhaps we could pick a methodology and then imagine a project that "fails." Given this scenario, the challenge would be to find a way to blame the participants for the failure.
What promises do developers and stakeholders make to each other in the methodology? How do you define "failure" and given that definition, how do you establish a causal relationship between an individual and the result?
What promises do developers and stakeholders make to each other in the methodology? How do you define "failure" and given that definition, how do you establish a causal relationship between an individual and the result?
Well, the great thing about scrum planning is that you know exactly how long it will take and who to hold accountable.
http://marcin.floryan.pl/blog/2011/06/nightmare-agile
http://marcin.floryan.pl/blog/2011/06/nightmare-agile
Sounds a little bit like the prisoner's dilemma...
1. It's pretty much isomorphic, yes
2. The title gestures to that (notice that programmer and prisoner both have 3 syllables, even)
3. The article links straight to the SEP article on the prisoner's dilemma
2. The title gestures to that (notice that programmer and prisoner both have 3 syllables, even)
3. The article links straight to the SEP article on the prisoner's dilemma
A little, except in the prisoner's dilemma, if they both agree and don't blame each other, you get the best outcome. That isn't really the case here. And if they both betray each other, they are both screwed.
Check your definition again. In Game Theory, that's a different game, where cooperating outweighs suckering the other guy. That's called a "Stag Hunt:"
http://plato.stanford.edu/entries/prisoner-dilemma/#StaHunPD
It could be that in some organizations, the rewards are such that a Stag Hunt is a better model.
http://plato.stanford.edu/entries/prisoner-dilemma/#StaHunPD
It could be that in some organizations, the rewards are such that a Stag Hunt is a better model.
No, I mean best outcome for all parties -- ie win-win -- and the story sounds to me like there is no actual good outcome for this tale. I am well aware that in prisoner's dilemma if one person betrays/blames and the other does not, that's when one party gets over.
Sorry for being unclear.
Sorry for being unclear.
No, you're still missing it. In Prisoner's Dilemma, if they both cooperate they still get a bad outcome. It is globally the least bad, but individually worse for each of them than defecting if the other cooperates. Look at the payoff matrix: [1]
If mutual cooperation was the best outcome, there would be no dilemma.
[1]: http://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Strategy_f...
If mutual cooperation was the best outcome, there would be no dilemma.
[1]: http://en.wikipedia.org/wiki/Prisoner%27s_dilemma#Strategy_f...
A Prisoner's Dilemma is defined by the ordinal ranking of alternatives, not the absolute cardinal values. The common example used from which the name of the game is derived happens to be prison sentences, which are all negative payoffs (or zero for no time). But as long as the payoffs maintain the same ordinal ranking, they could positive, negative, or a mix.
In that matrix, total time served if they remain loyal is 2 months (one month for each of them). If one of them betrays the other, total time served is 1 year (for one of them while the other goes free). If they both betray, total time served is 6 months (3 for each of them). So it seems to me that if they both remain silent, that is the best aggregate outcome. However, it isn't always the case that someone will automatically be screwed no matter what. Sometimes if they both remain loyal, they can't be charged with anything. The reason there is temptation in a case like that is because if the other guy breaks, then you are screwed, not because loyalty isn't the overall best outcome but because individual outcomes vary with each possible set of choices in the matrix and there is risk in not knowing what the other person will do.
Maybe you don't care about "overall best outcome". Maybe you only care what happens to you, in which case I assume you would betray me. But in the face of rules that are going to screw the people no matter what, closing ranks causes the least overall harm -- assuming you have sufficient trust to pull that off in the absence of communication. (EDIT: For example, if you are both members of an organized crime syndicate or a resistance movement, the organization suffers the least overall loss/cost if it gives very strict orders that no one talks -- assuming the two people are of equal value to the organization. Even if you are not part of a larger organization, betrayal costs not only time served but trust of the person you betrayed. Depending upon the situation, one month in jail may be less of a loss than losing this person's trust.)
Maybe you don't care about "overall best outcome". Maybe you only care what happens to you, in which case I assume you would betray me. But in the face of rules that are going to screw the people no matter what, closing ranks causes the least overall harm -- assuming you have sufficient trust to pull that off in the absence of communication. (EDIT: For example, if you are both members of an organized crime syndicate or a resistance movement, the organization suffers the least overall loss/cost if it gives very strict orders that no one talks -- assuming the two people are of equal value to the organization. Even if you are not part of a larger organization, betrayal costs not only time served but trust of the person you betrayed. Depending upon the situation, one month in jail may be less of a loss than losing this person's trust.)
"Sometimes if they both remain loyal, they can't be charged with anything."
It's important to not bring too much real-world logic to the matter. What's being discussed is a certain payoff matrix; the "story" of the Prisoner's Dilemma is really just a way to try to wrap the non-math part of your brain around it, but the payoff matrix, exactly as written, is what is under discussion. The payoff matrix where there is a non-zero probability that they will both manage to skate is a different problem, one not necessarily less worthy of discussion, but one that is not the Prisoner's Dilemma any longer.
If you are interested in further discussion, you should google "Iterated Prisoner's Dilemma" for more fun, including stories about computer simulations of various strategies. Prisoner's Dilemma on its own isn't actually all that interesting, it's more useful as a framing device. Iterated Prisoner's Dilemma is actually interesting.
It's important to not bring too much real-world logic to the matter. What's being discussed is a certain payoff matrix; the "story" of the Prisoner's Dilemma is really just a way to try to wrap the non-math part of your brain around it, but the payoff matrix, exactly as written, is what is under discussion. The payoff matrix where there is a non-zero probability that they will both manage to skate is a different problem, one not necessarily less worthy of discussion, but one that is not the Prisoner's Dilemma any longer.
If you are interested in further discussion, you should google "Iterated Prisoner's Dilemma" for more fun, including stories about computer simulations of various strategies. Prisoner's Dilemma on its own isn't actually all that interesting, it's more useful as a framing device. Iterated Prisoner's Dilemma is actually interesting.
Actually, I've had a college class on "Negotiation and Conflict Management" where stuff like Prisoner's Dilemma was covered fairly well. I cleaned up in the negotiation rounds at the end of class, so I suspect my understanding of the concepts we were taught is just fine. But you and I appear to be talking at cross purposes.
Peace.
Peace.
Thanks for the insight Captain Obvious. Did you forget to eat your Wheaties this
morning?
"Be civil. Don't say things you wouldn't say in a face to face conversation." [1] There are many places for insulting others, HN is not one of those places.
[1] http://ycombinator.com/newsguidelines.html
[1] http://ycombinator.com/newsguidelines.html
The problem here is that managers think that writing good software is like baking a cake, if you have the right ingredients, the right process, the right environment and the right people, then you are 99% likely to end up with a satisfactory cake. This is absolutely not the case with building software.
It could very well be the red tape in the office, the politics, maneuvering, posturing, noisy programmer area, the tools programmers were given, time-wasting meetings, and thousands of other reasons.
My feeling is the blame falls on the guy claiming the problem space consists of "the programmer" or "the methodology". The manager figure delivering the bad news to the programmer is a kung fu master of politics and the art of war, he's trying to get blame placed on anywhere but himself.
It could very well be the red tape in the office, the politics, maneuvering, posturing, noisy programmer area, the tools programmers were given, time-wasting meetings, and thousands of other reasons.
My feeling is the blame falls on the guy claiming the problem space consists of "the programmer" or "the methodology". The manager figure delivering the bad news to the programmer is a kung fu master of politics and the art of war, he's trying to get blame placed on anywhere but himself.
I've been developing software for over 15 years and without question communication and politics are just as important as technical skills. You have to get key people on your side and you have to keep everybody up to date on the status of the project.
Actually, I think those managers are right. The problem is that the right ingredients to produce a satisfactory cake are not the same as the right ingredients to succeed in a large corporation.
I disagree. The different between building software and baking a cake is that we've had thousands of years to perfect a few dozen types of cake. The recipes are well-known and easy to follow.
Building software is very much in its infancy and many magnitudes harder than baking a cake.
And any manager who thinks just having the right ingredients, people and recipe will guarantee a good cake has never baked a cake, anyhow.
Building software is very much in its infancy and many magnitudes harder than baking a cake.
And any manager who thinks just having the right ingredients, people and recipe will guarantee a good cake has never baked a cake, anyhow.
Now you need to write the followup, which shows how agile and Scrum are so often used to shift power from the business people to the developers without actually solving the underlying problems. (You'll have to edit parts of this piece where you imply otherwise.)
Unless, of course, agile is that magic bullet Fred Brooks stopped looking for 25 years ago.
Unless, of course, agile is that magic bullet Fred Brooks stopped looking for 25 years ago.
Nothing in the way of a methodology stops underlying problems of a culture that thinks careers are a zero-sum game and that actually delivering the software is secondary to advancement.
But FWIW, all I claim is that Agile does a poorer job of blaming people. The other stuff you're talking about is your point of view, which suggests to me that you are the right person to write the blog post.
But FWIW, all I claim is that Agile does a poorer job of blaming people. The other stuff you're talking about is your point of view, which suggests to me that you are the right person to write the blog post.
[deleted]
A colleague of mine - his wife was a 3rd grade teacher.
One day, he mentioned to her the process that we were doing.
And she laughed. She showed him the daily progress report. "What I did today? What I want to do tomorrow?" That her 3rd graders had to fill out everyday.
Are programmers adults that need to be self-managed + micro-managed like 3rd graders?
One day, he mentioned to her the process that we were doing.
And she laughed. She showed him the daily progress report. "What I did today? What I want to do tomorrow?" That her 3rd graders had to fill out everyday.
Are programmers adults that need to be self-managed + micro-managed like 3rd graders?
> She showed him the daily progress report. "What I did today? What I want to do tomorrow?" That her 3rd graders had to fill out everyday.
To be honest, it sounds just as insane—possibly more—that children have to do this kind of thing. There were fewer assessment exercises of this sort when I was in school, albeit appreciably more than my parents had to contend with. The main lesson I took away from them was to avoid, wherever possible, any job or career where I'd have to waste everyone's time pursuing such ridiculous make-work.
I worry that the more we force this stuff on children from an early age, the more they will take it to be the normal (de jure, not de facto) state of affairs rather than an aberration stemming from the awful tendency to value managerial gestures over productive work.
To be honest, it sounds just as insane—possibly more—that children have to do this kind of thing. There were fewer assessment exercises of this sort when I was in school, albeit appreciably more than my parents had to contend with. The main lesson I took away from them was to avoid, wherever possible, any job or career where I'd have to waste everyone's time pursuing such ridiculous make-work.
I worry that the more we force this stuff on children from an early age, the more they will take it to be the normal (de jure, not de facto) state of affairs rather than an aberration stemming from the awful tendency to value managerial gestures over productive work.
Having graduated from grade school many decades before NCLB, I fear that adults are irrevocably changing education for the worse. Test taking isn't learning, yet it the main priority for many schools (especially poor ones).
Its not like the colleges are any help here with the requirement to take the SAT, PSAT, or ACT for college admission. A student who doesn't know how to take a test isn't going to get the scholarships or even entrance to college.
Poor schools have a lot of problems like poor materials (my brother's history text book had serious editing errors like "Abraham Lincoln was black.") and problems attracting good teachers. The problem seems to be bad enough that they, given the scores handed in, are even to the level of teaching for the tests. I don't see any real hope until parents are allowed to move the tuition money the state pays the schools to the school of their choice. Innovation is hard without choice.
Poor schools have a lot of problems like poor materials (my brother's history text book had serious editing errors like "Abraham Lincoln was black.") and problems attracting good teachers. The problem seems to be bad enough that they, given the scores handed in, are even to the level of teaching for the tests. I don't see any real hope until parents are allowed to move the tuition money the state pays the schools to the school of their choice. Innovation is hard without choice.
"To be honest, it sounds just as insane—possibly more—that children have to do this kind of thing."
That particular exercise is a way of practicing the connections between verbalization, logical reasoning about the future, and emotional desires. It sounds like an excellent way to help the top half of the IQ bell curve to work on projects bigger than will fit in their head overnight.
As to wasted time, it should not take more than a few minutes a day, a bargain compared to the time spent on the party indoctrination in history and social studies.
That particular exercise is a way of practicing the connections between verbalization, logical reasoning about the future, and emotional desires. It sounds like an excellent way to help the top half of the IQ bell curve to work on projects bigger than will fit in their head overnight.
As to wasted time, it should not take more than a few minutes a day, a bargain compared to the time spent on the party indoctrination in history and social studies.
Good thing they didn't ask this when I was in school:
What I did today? -> Went to school. What I want to do tomorrow? -> Stay home and play nintendo.
The second question is not quite honest, more accurately "What BS can I feed my teacher?" I view these questions as a way of attacking integrity, lie or face the consequences. One might argue this teaches a valuable skill. But I find it hard to stomach.
What I did today? -> Went to school. What I want to do tomorrow? -> Stay home and play nintendo.
The second question is not quite honest, more accurately "What BS can I feed my teacher?" I view these questions as a way of attacking integrity, lie or face the consequences. One might argue this teaches a valuable skill. But I find it hard to stomach.
Everyone working in teams needs to share information about what they're doing with the rest of the team, so that they can work together.
If the simplest way to do so is a brief "Yesterday I built 90% of Foo, but hit problem Bar, today I plan to finish Foo and start implementing BaseWidget" then that's great.
If the simplest way to do so is a brief "Yesterday I built 90% of Foo, but hit problem Bar, today I plan to finish Foo and start implementing BaseWidget" then that's great.
Not everyone understands that scrum and agile have this heritage of coming from getting horribly managed projects worked on by extremely mediocre devs in the enterprise trenches on track.
"And this could be why Waterfall survives: It's the process that optimizes for blaming other people."
nah. Waterfall survives because it lets business development share world-views with the customer.
nah. Waterfall survives because it lets business development share world-views with the customer.
Amen Brother! My new internal customer just had a wonderful surprise as we are adopting Agile. He asked if he needed to work through me to reorganize the backlog for upcoming sprints. No - you own scope and timeline. Reorganize as much as you like on items we have not yet begun as long as you respect the total effort estimates per sprint.
Value working software and healthy collaboration with a process that routinely measures real progress and you'll never have to explain why you slipped 3 months since last week's status report.
Value working software and healthy collaboration with a process that routinely measures real progress and you'll never have to explain why you slipped 3 months since last week's status report.
Was this a failure of process or people?
Neither.
It was a failure of management...
Anyone who conducts a post-mortem with their people would be much better off simply looking in the mirror.
Where were the Vice President of Development and the Vice President of Marketing when things starting going bad? Things simply don't go wrong at the end of the project; there had to have been plenty of warning, if only someone was noticing.
Vice presidents do best by having their finger on the pulse of the things that are important and providing the right resources and leadership to enable both their people and processes to be successful. These two were in a time warp, reacting to failure instead of preparing for success.
The President needs to find a way to get the job done at the Vice President level. He's the one who needs either better people or better processes.
Neither.
It was a failure of management...
Anyone who conducts a post-mortem with their people would be much better off simply looking in the mirror.
Where were the Vice President of Development and the Vice President of Marketing when things starting going bad? Things simply don't go wrong at the end of the project; there had to have been plenty of warning, if only someone was noticing.
Vice presidents do best by having their finger on the pulse of the things that are important and providing the right resources and leadership to enable both their people and processes to be successful. These two were in a time warp, reacting to failure instead of preparing for success.
The President needs to find a way to get the job done at the Vice President level. He's the one who needs either better people or better processes.
Exactly, VP is the level at which you've crossed the rubicon and excuses no longer matter. You're in it to win and if you don't it's your (the VP's) fault.
Don't play the game with people like these. That was the real problem. Once their nature and character became evident, which it must have quite early, there was no way to win. Leave and find a better game.
I know it's just a story, but if I had managers like this, I'd be headed for the door so fast, I'd trip over them.
In the original prisoners dilemna, there was real consequences involved: 10-20 years in jail. Getting fired from waterfallcorp doesnt sound like the end of the world. (Some might even consider it a good thing). Entertaining read though.
This feels like an argument on the wrong level of abstraction. Even if the methodology is at fault, it seems silly to say "Waterfall is the problem." That's simply too large of a point to make. You could say that you fell behind during development and couldn't tell that you were falling behind, so the problem snowballed. That's specific enough for people to discuss without resorting to politics. Otherwise you're taking the organization's dysfunctions head-on, which is a losing battle.
Somewhat off topic, but I am reminded of a paper I had to critique a uni: "A Rational Design Process: How and Why to Fake It" [1]
The paper reasons that following a strict process is unrealistic, but that it is still worth it to "fake" a process, by presenting your system to others as though you had followed it (for example through documentation and reports that mimic the waterfall process, presented to players such as the VP and Director of Marketing in the OP).
[1] http://www.idemployee.id.tue.nl/g.w.m.rauterberg/presentatio...
The most interesting thing is that this paper was published before I was born in 1986. Even so, everyone in my class (this was about 18 months ago) was in agreement that the paper was still valid. (Though there would be the reasonable argument that we weren't able to give informed opinions having yet to experience software engineering in the real world at that point)
The paper reasons that following a strict process is unrealistic, but that it is still worth it to "fake" a process, by presenting your system to others as though you had followed it (for example through documentation and reports that mimic the waterfall process, presented to players such as the VP and Director of Marketing in the OP).
[1] http://www.idemployee.id.tue.nl/g.w.m.rauterberg/presentatio...
The most interesting thing is that this paper was published before I was born in 1986. Even so, everyone in my class (this was about 18 months ago) was in agreement that the paper was still valid. (Though there would be the reasonable argument that we weren't able to give informed opinions having yet to experience software engineering in the real world at that point)
I worked on a project for a large corporation where the manager running the project did this. I thought he was insane until it became clear that the fake process created an umbrella for the devs to self-organize under. It wasn't a perfect project but it went better than it would have if the process had been real.
If you are playing poker and you don't know who the sucker is, you're it!
The other day I suddenly realized why I dislike software processes so much. It's because how you build a system and what the system is depend on each other. Each needs to evolve as you learn more (and as the team changes).
Software processes, by definition, predefine the how. That kills the feedback loop and hinders projects from evolving. The same reason why waterfall is a mistake -- because it treats as fixed things that need to change -- applies to software processes in general. Software process is just waterfall at the project level.
Software projects are no less unique than software systems, but we would laugh out the door anyone who claimed to have a metadesign for all programs.
Software processes, by definition, predefine the how. That kills the feedback loop and hinders projects from evolving. The same reason why waterfall is a mistake -- because it treats as fixed things that need to change -- applies to software processes in general. Software process is just waterfall at the project level.
Software projects are no less unique than software systems, but we would laugh out the door anyone who claimed to have a metadesign for all programs.
You can see this sort of thing play out in every episode of the BBC's Apprentice. The prevailing strategy is to blame the someone for messing up their part of the task, rather than to accept that the strategy which everyone agreed upon or which they were forced to adopt was the source of most of the problems. Since they're FORCED to name someone to fire anyway, not doing so from the start just makes them look indecisive or dishonest.
This is why smart people study rhetoric. Everyone is at fault but the guy with the best sounding answer wins.
I'm sorry, but measured against the number of times I've heard the phrase "Doing Agile Wrong", this sound disingenuous.