Gamification is not entirely bullshit(22ideastreet.com)
22ideastreet.com
Gamification is not entirely bullshit
http://22ideastreet.com/blog/2011/08/16/gamification-is-not-entirely-bullshit/
34 comments
[deleted]
With gamification of the workplace in particular, there seems to be a repeating cycle. Each cycle does come with some shifts in focus, but it seems mostly implicit ones, without a lot of reflection.
I've been doing some preliminary research trying to track down each of the cycles and how they relate, but the fact that so much of it is ephemeral and buzzword-heavy, and the studies are done by self-interested people with poor documentation of results, makes it a bit difficult.
As far as I can tell, the earliest proposals for using game-like processes to motivate and engage workers came in the Soviet Union, for the obvious reason that they needed a replacement for monetary motivation ASAP. Lenin called it "socialist competition", and had this vision of friendly, game/play-like competition as a replacement for cutthroat, loser-ends-up-homeless capitalist competition. I wrote a little about that here, though it's more like a preliminary collection of sources than a proper history at the moment: http://www.kmjn.org/notes/soviet_gamification.html
Then sometime in the 1980s and especially the 1990s and 2000s, the idea of the power of games and play started becoming a huge thing in management. There was even a hilarious consulting job title for a while, "funsultant", someone who would come into a company and help make it more "fun" and "gamelike". I suspect there are similarities between today's "gamification consultants" and 1990s "funsultants", with the main shift being less focus on play and fun, and a bigger focus on reward/feedback loops. In effect the '90s game/play management techniques are merging with the much older behaviorist-reinforcement management techniques. I wrote a bit about the funsultants and some of the reactions to them as well (Office Space's flair scene is probably the most memorable example of a backlash to mandatory fun): http://www.kmjn.org/notes/funsultants_and_gamification.html
So I guess what I'm mostly missing from gamification advocates is a clearer picture of what's new here and how it relates to previous approaches. How much of it is about Skinner-style behavior-loop reinforcement? How much of it is about fun? About competition? How do the new approaches, whether we call them "game mechanics" or something else, relate to existing research findings on how effective each of these things are? As far as I can tell nobody actually wants to mention any of those things, instead presenting it as something totally new: game mechanics used to be used only for entertainment, but in the 2010s we realized they could be used to motivate non-entertainment things as well. But I don't think it just fell from the sky with no relationship to previous attempts.
I've been doing some preliminary research trying to track down each of the cycles and how they relate, but the fact that so much of it is ephemeral and buzzword-heavy, and the studies are done by self-interested people with poor documentation of results, makes it a bit difficult.
As far as I can tell, the earliest proposals for using game-like processes to motivate and engage workers came in the Soviet Union, for the obvious reason that they needed a replacement for monetary motivation ASAP. Lenin called it "socialist competition", and had this vision of friendly, game/play-like competition as a replacement for cutthroat, loser-ends-up-homeless capitalist competition. I wrote a little about that here, though it's more like a preliminary collection of sources than a proper history at the moment: http://www.kmjn.org/notes/soviet_gamification.html
Then sometime in the 1980s and especially the 1990s and 2000s, the idea of the power of games and play started becoming a huge thing in management. There was even a hilarious consulting job title for a while, "funsultant", someone who would come into a company and help make it more "fun" and "gamelike". I suspect there are similarities between today's "gamification consultants" and 1990s "funsultants", with the main shift being less focus on play and fun, and a bigger focus on reward/feedback loops. In effect the '90s game/play management techniques are merging with the much older behaviorist-reinforcement management techniques. I wrote a bit about the funsultants and some of the reactions to them as well (Office Space's flair scene is probably the most memorable example of a backlash to mandatory fun): http://www.kmjn.org/notes/funsultants_and_gamification.html
So I guess what I'm mostly missing from gamification advocates is a clearer picture of what's new here and how it relates to previous approaches. How much of it is about Skinner-style behavior-loop reinforcement? How much of it is about fun? About competition? How do the new approaches, whether we call them "game mechanics" or something else, relate to existing research findings on how effective each of these things are? As far as I can tell nobody actually wants to mention any of those things, instead presenting it as something totally new: game mechanics used to be used only for entertainment, but in the 2010s we realized they could be used to motivate non-entertainment things as well. But I don't think it just fell from the sky with no relationship to previous attempts.
I believe this phenomenon it's characteristic of highly interdiciplinary areas, mostly practical and technical but with a strong component of humanism.
The people that want to do work in industry tend to gravitate to some specific fields. And half a decade in managment or engineering school can't teach you the history of philosophy, anthropology, sociology or psychology. I believe this to be the reason why "design" is a big hit nowdays, "managers" and "engineers" are looking at humanism, and the practical humanists: the "designers", for a competetitive advantage.
In the case of gamification, there were people interested not in humanism per se, but in building games. They found the power of some buried ideas for social control either by getting to identify the problem and searching for the answer in the field's body of knowledge, or by rediscovering it themselves, bypassing the field's historical intelectual debate. Some wanted to "sell it", so they needed an "angle" related to their profession.
The people that want to do work in industry tend to gravitate to some specific fields. And half a decade in managment or engineering school can't teach you the history of philosophy, anthropology, sociology or psychology. I believe this to be the reason why "design" is a big hit nowdays, "managers" and "engineers" are looking at humanism, and the practical humanists: the "designers", for a competetitive advantage.
In the case of gamification, there were people interested not in humanism per se, but in building games. They found the power of some buried ideas for social control either by getting to identify the problem and searching for the answer in the field's body of knowledge, or by rediscovering it themselves, bypassing the field's historical intelectual debate. Some wanted to "sell it", so they needed an "angle" related to their profession.
I think that's definitely part of it, but I think there may be a bit of deliberate amnesia as well, out of a desire to portray something as totally new, and perhaps to avoid any unwanted historical baggage.
You also see this in "serious games" and its variants. Even Al Gore was recently talking about how the future of videogames is that maybe they'll be used for things like education, too, not just entertainment. Sure, I even buy that. But that's what everyone was saying in the Apple ][ era also! It's not just about credit, but I think we lose something from the amnesia: some of the Apple ][ games were actually better uses of game mechanics for education than much of the fairly shallow stuff being promoted more recently, and we can even learn something from the examples that were more mixed successes (though admittedly there was a lot of bad "edutainment", too).
You also see this in "serious games" and its variants. Even Al Gore was recently talking about how the future of videogames is that maybe they'll be used for things like education, too, not just entertainment. Sure, I even buy that. But that's what everyone was saying in the Apple ][ era also! It's not just about credit, but I think we lose something from the amnesia: some of the Apple ][ games were actually better uses of game mechanics for education than much of the fairly shallow stuff being promoted more recently, and we can even learn something from the examples that were more mixed successes (though admittedly there was a lot of bad "edutainment", too).
That's true, the process doesn't need to be 100% "conscious".
PS: Great insight on the article about soviet "gamification", I never thought abut that in that way, thanks.
PS: Great insight on the article about soviet "gamification", I never thought abut that in that way, thanks.
Well put.
Gamification does indeed work. It is a marketing tactic that translates well to a certain type of product. Once the dust settles we will see gamification in a lot of unusual places, such as our work. I can imagine a world where everything is either upvoted or downvoted (although I'm not saying it migh be pleasant).
Gamification does indeed work. It is a marketing tactic that translates well to a certain type of product. Once the dust settles we will see gamification in a lot of unusual places, such as our work. I can imagine a world where everything is either upvoted or downvoted (although I'm not saying it migh be pleasant).
In a more meta way, just about any innovation goes through a 2-steps forward, 1-step back cycle. The 1-step back is always due to wide-scale adoption where the original message is somewhat lost in translation, but as a whole actually moves things forward (ie. agile 'methodology').
I think when it comes to gamification it's perhaps better to reframe it as a more targeted engagement structure. If you think about how big the actual video game industry is, how much people are _paying_ to solve problems, that there could be a better way to flip that script, because they are in fact doing work in the guise of entertainment.
Right now the whole concept is in its infancy, but I imagine five years from now it will be prevalent in most everything we do, perhaps in a very indirect manner. It might even be the perfect cure for procrastination (ducks :)
I think when it comes to gamification it's perhaps better to reframe it as a more targeted engagement structure. If you think about how big the actual video game industry is, how much people are _paying_ to solve problems, that there could be a better way to flip that script, because they are in fact doing work in the guise of entertainment.
Right now the whole concept is in its infancy, but I imagine five years from now it will be prevalent in most everything we do, perhaps in a very indirect manner. It might even be the perfect cure for procrastination (ducks :)
> I think when it comes to gamification it's perhaps better to reframe it as a more targeted engagement structure. If you think about how big the actual video game industry is, how much people are _paying_ to solve problems, that there could be a better way to flip that script, because they are in fact doing work in the guise of entertainment.
----
Disagree. Play is distinct from work because it is responsibility free. Work is a chore that you do because you must. For this reason Gamification is and will forever remain bullshit.
----
Disagree. Play is distinct from work because it is responsibility free. Work is a chore that you do because you must. For this reason Gamification is and will forever remain bullshit.
I up-voted you because I like you directly teasing out this point.
This is what I meant by flipping the script. It's just a matter of perception, and that itself can be changed. It shifts with culture. If you think work is a chore then it will be a chore. If you think it is fun then it will be that, at least some of the time. If innovations come around that make it more fun regardless, culture will shift where less will think of it as a chore. These things build upon themselves.
Keep in mind play is not responsibility-free - it has an associated time/opportunity cost. Taking a trip around Europe for a couple weeks, I'd say that's great. The experience can be life-changing... it can bring alot of value for the time spent. Being locked in a room playing some online game for a few weeks... not so great. It might be fun to some degree, but it can be deleterious for one's health, personal relationships, work life, etc. It can actually be harmful.
As a working engineer for the past dozen years I can agree that many times work is a chore. But, and this is a big but, every once in awhile I get a project that is so challenging, so meaty, so impactful that it is far more fun than anything else I can imagine doing. Those projects are the kinds of things I stay at work late for, when I get home even VPN to the office to continue working until I pass out, then get in early to the office because I can't wait to get back to it. YMMV, but I think quite a few HN folk can relate. Some people this might describe their job entirely. The gamification is implicit in the work itself.
I think there's a huge industry waiting to be tapped here, and once it finds its footing human productivity will go up a few notches. The biggest problem is what we currently have is a bit cheesy and too obvious, perhaps to the point where it makes the user feel foolish/duped. That needs to be overcome.
This is what I meant by flipping the script. It's just a matter of perception, and that itself can be changed. It shifts with culture. If you think work is a chore then it will be a chore. If you think it is fun then it will be that, at least some of the time. If innovations come around that make it more fun regardless, culture will shift where less will think of it as a chore. These things build upon themselves.
Keep in mind play is not responsibility-free - it has an associated time/opportunity cost. Taking a trip around Europe for a couple weeks, I'd say that's great. The experience can be life-changing... it can bring alot of value for the time spent. Being locked in a room playing some online game for a few weeks... not so great. It might be fun to some degree, but it can be deleterious for one's health, personal relationships, work life, etc. It can actually be harmful.
As a working engineer for the past dozen years I can agree that many times work is a chore. But, and this is a big but, every once in awhile I get a project that is so challenging, so meaty, so impactful that it is far more fun than anything else I can imagine doing. Those projects are the kinds of things I stay at work late for, when I get home even VPN to the office to continue working until I pass out, then get in early to the office because I can't wait to get back to it. YMMV, but I think quite a few HN folk can relate. Some people this might describe their job entirely. The gamification is implicit in the work itself.
I think there's a huge industry waiting to be tapped here, and once it finds its footing human productivity will go up a few notches. The biggest problem is what we currently have is a bit cheesy and too obvious, perhaps to the point where it makes the user feel foolish/duped. That needs to be overcome.
For the vast majority of people work is always a chore. It's a commodity you use to barter with your employer; in exchange you receive money that allows you to live.
You argue that sometimes work is "fun". I think a more accurate description is that work can sometimes be engaging. Like when you're presented with a problem that's right at the edge of your abilities but not out of reach. Yet it's still work. Unlike play, work carries with it stress and responsibility. There are consequences if you screw up. Worse if you stop entirely. None of these are true for play.
Gamification is bullshit because it encourages the perception that these differences don't exist or that they don't matter. It's a flat out lie. Work is not play. Accept it. Move on.
You argue that sometimes work is "fun". I think a more accurate description is that work can sometimes be engaging. Like when you're presented with a problem that's right at the edge of your abilities but not out of reach. Yet it's still work. Unlike play, work carries with it stress and responsibility. There are consequences if you screw up. Worse if you stop entirely. None of these are true for play.
Gamification is bullshit because it encourages the perception that these differences don't exist or that they don't matter. It's a flat out lie. Work is not play. Accept it. Move on.
Play is a frame of mind, not a contractual relationship. If it weren't, nobody would take mmorpgs seriously, and they do; the consequences may not involve your paycheck but it's harsh when you let your team down.
Playfulness shows up at work all the time, although not necessarily in ways that promote the bottom line.
Play mindsets can be encouraged in the workplace when they don't emerge spontaneously. Whether those efforts are successful for a given person on any given day...? It's still pushing a rock uphill.
Game design and reward systems are trying to find a path that doesn't suck. It may take another decade of experimentation before ten thousand failed attempts show the obvious and elegant ways to make more workplace leaders look like your favorite camp counsellor, recess organizer, or dungeon master.
Playfulness shows up at work all the time, although not necessarily in ways that promote the bottom line.
Play mindsets can be encouraged in the workplace when they don't emerge spontaneously. Whether those efforts are successful for a given person on any given day...? It's still pushing a rock uphill.
Game design and reward systems are trying to find a path that doesn't suck. It may take another decade of experimentation before ten thousand failed attempts show the obvious and elegant ways to make more workplace leaders look like your favorite camp counsellor, recess organizer, or dungeon master.
> Play is a frame of mind
It's more than that. The environment matters. Real play is risk free. You can do whatever you want and there are no consequences. This is not true for work.
At the end of the day business doesn't want their employees to treat work as play; they just want them to be more engaged while at work. But instead of looking at why their workers are disengaged they hire people who set up cute little games and contests and tell people to pretend they're not actually working. Which is all well and good until somebody drops the ball and they get fired. Whoops. Sorry Bob. I guess you didn't understand we weren't actually playing after all.
Gimme a break. Gamification is just bullshit newage snakeoil.
It's more than that. The environment matters. Real play is risk free. You can do whatever you want and there are no consequences. This is not true for work.
At the end of the day business doesn't want their employees to treat work as play; they just want them to be more engaged while at work. But instead of looking at why their workers are disengaged they hire people who set up cute little games and contests and tell people to pretend they're not actually working. Which is all well and good until somebody drops the ball and they get fired. Whoops. Sorry Bob. I guess you didn't understand we weren't actually playing after all.
Gimme a break. Gamification is just bullshit newage snakeoil.
What others perceive as work can certainly be play. For example, if I've saved enough money in my rainy day fund then there are no consequences for ceasing 'work', so perhaps by your reasoning what seems to be work could really be play to me.
I wish my rainy day fund were that big.
I wish my rainy day fund were that big.
Off topic, but that inline 'tweet' button to tweet a choice quote from the article was an interesting touch. I hadn't seen this technique before.
I think it would be even more awesome if the tweet button was even more inline with the text, almost like a citation link in wikipedia, except you could make it an actual twitter bird character!
I think any time you can help people, even if it's just 140 characters, the more likely they'll do what you ask.
The most bizarre bit of gamification I've run across lately was in Wells Fargo's ATMs. I deposited a check the other day and got an alert on the ATM screen:
Photo here: http://artlung.com/blog/2012/05/03/gamification-in-the-wild-...
You've earned a new badge
Express Depositor
[Learn More]
Photo here: http://artlung.com/blog/2012/05/03/gamification-in-the-wild-...
Oh goodness.
Have you seen this at any other Wells Fargo ATMs? Do you know if there's a web interface? The only mention I can find of this is the linked image.
Have you seen this at any other Wells Fargo ATMs? Do you know if there's a web interface? The only mention I can find of this is the linked image.
Gamification is bullshit by design. Gameful mechanics may not be, but we're not there yet.
Point 1 is that Zichermann (who wrote Games-Based Marketing) misrepresents the research into motivation. Whether this is maliciousness or incompetence is up for debate (personally I think the latter with a dose of arrogance). You can see Deturding shred his O'Reilly book, where he goes into this in a lot of detail [1]. One point that is made in the blog post is that most people are socializers, according to Bartle player types. Nick Yee has already shown those player types to be largely useless outside of MUDs, and Bartle never claimed them to be otherwise. In addition, there is no evidence that most people are socializers, and Zichermann doesn't cite this claim. I've not been able to find anyone who's tracked down any research which shows that this is true.
It's odd that the author mentions Pink's Drive at the end, as it's pretty much the anti-thesis of Zichermann. Drive advocates intrinsic motivation, Zichermann is all about extrinsic. If you don't get a trinket, it isn't worth doing. Extrinsic motivation erodes intrinsic motivation. And that's why Bogost slams gamification in the first place: the trinkets you get are valueless, and present no expense to the company. At least frequent flier mile rewards do something.
Point 2 is that this post makes no mention of the quality of interaction. When you play with small percentages of large absolute numbers, you're going to find a small minority of people who respond to this. Enough to fill a leaderboard. But what are they doing? Are they really rating stuff? Are they just hitting 5 stars on everything? We know that HITs in the Mechanical Turk suffer from poor interactions, and those people are getting paid (an extrinsic reward worth more than a leaderboard place, no?). What about the other 99% that aren't on the leaderboard?
Point 3 is that, as gradstudent pointed out, context is highly important when discerning work from play. The research shows that artists, for example, make better work when they aren't being paid. Even if you change nothing in the environment but making the piece contracted, the quality of the work suffers. People know when they're working and when they aren't (this was in Drive). A leaderboard in Space Invaders might be fun, but a leaderboard at work is a really good way for middle management to find the worst 10% of performers, and becomes a tool for control, and people recognize this instantly.
So yes, gamification is bullshit. It's so much bullshit, that Jane McGonigal, who popularized the use of such things, has had to back away from the term as it's so poisonous, and she's using "gameful" instead.
What this all came about from was "People play games more so than doing other things, why is that?" Gamification took all the wrong elements. Games are all about feeding intrinsic motivation with tight feedback loops. Self-determination theory and Reiss' 16 Intrinsic Motivations provide much better frameworks for understanding games. Rigby and Ryan's Glued to Games looks at SDT, and Radoff's Game On has a section of the 16 motivations.
For anyone interested in achieving the stated goals of gamification, but is concerned on how to do it right, should probably read Radoff. It's the most honest, and legitimate, treatment of the subject I have yet found, by someone who actually wears the game designer t-shirt and the businessman suit with equal comfort.
[1]http://gamification-research.org/2011/09/a-quick-buck-by-cop...
Point 1 is that Zichermann (who wrote Games-Based Marketing) misrepresents the research into motivation. Whether this is maliciousness or incompetence is up for debate (personally I think the latter with a dose of arrogance). You can see Deturding shred his O'Reilly book, where he goes into this in a lot of detail [1]. One point that is made in the blog post is that most people are socializers, according to Bartle player types. Nick Yee has already shown those player types to be largely useless outside of MUDs, and Bartle never claimed them to be otherwise. In addition, there is no evidence that most people are socializers, and Zichermann doesn't cite this claim. I've not been able to find anyone who's tracked down any research which shows that this is true.
It's odd that the author mentions Pink's Drive at the end, as it's pretty much the anti-thesis of Zichermann. Drive advocates intrinsic motivation, Zichermann is all about extrinsic. If you don't get a trinket, it isn't worth doing. Extrinsic motivation erodes intrinsic motivation. And that's why Bogost slams gamification in the first place: the trinkets you get are valueless, and present no expense to the company. At least frequent flier mile rewards do something.
Point 2 is that this post makes no mention of the quality of interaction. When you play with small percentages of large absolute numbers, you're going to find a small minority of people who respond to this. Enough to fill a leaderboard. But what are they doing? Are they really rating stuff? Are they just hitting 5 stars on everything? We know that HITs in the Mechanical Turk suffer from poor interactions, and those people are getting paid (an extrinsic reward worth more than a leaderboard place, no?). What about the other 99% that aren't on the leaderboard?
Point 3 is that, as gradstudent pointed out, context is highly important when discerning work from play. The research shows that artists, for example, make better work when they aren't being paid. Even if you change nothing in the environment but making the piece contracted, the quality of the work suffers. People know when they're working and when they aren't (this was in Drive). A leaderboard in Space Invaders might be fun, but a leaderboard at work is a really good way for middle management to find the worst 10% of performers, and becomes a tool for control, and people recognize this instantly.
So yes, gamification is bullshit. It's so much bullshit, that Jane McGonigal, who popularized the use of such things, has had to back away from the term as it's so poisonous, and she's using "gameful" instead.
What this all came about from was "People play games more so than doing other things, why is that?" Gamification took all the wrong elements. Games are all about feeding intrinsic motivation with tight feedback loops. Self-determination theory and Reiss' 16 Intrinsic Motivations provide much better frameworks for understanding games. Rigby and Ryan's Glued to Games looks at SDT, and Radoff's Game On has a section of the 16 motivations.
For anyone interested in achieving the stated goals of gamification, but is concerned on how to do it right, should probably read Radoff. It's the most honest, and legitimate, treatment of the subject I have yet found, by someone who actually wears the game designer t-shirt and the businessman suit with equal comfort.
[1]http://gamification-research.org/2011/09/a-quick-buck-by-cop...
Too late for an edit, but I wanted to note that this post does make mention of quality of interaction, I just forgot by the time I wrote the passage.
The first rater of a deal would get a bunch of points regardless of their rating of a deal, and so their rating didn’t matter. They would just spam (in the gamer-speak sense) the “10″ rating since it was closest to the next deal button, and do this hundreds of times without looking at the deal at all. Future voters would get points based on the crowd average, so the first voter would skew the points rating all of the deals in one sitting. It ended up that the ratings on the deals were not really accurate, thus undermining the crowdsourcing of the deals and thwarting our business goals. However, this example was more of an implementation failure than a failure of gamification in general.
I think the OP is too hard on himself here. It's very hard to imagine a game mechanic which would elicit the feedback you want: namely that you want people to give an honest opinion of something. How do you place value on an opinion? How can you tie in something so subjective to a mastery goal?
I was placed on a very similar task as part of some consulting I've been doing. I haven't found a solution, and I don't think a good one exists. The best I could come up with was that users had two interactions, not one. The first one was their honest opinion, the second one was a guess on what the prevailing opinion was. The OP only did the second, which is only going to result in a very quick convergence on groupthink. This was rejected by the company, as they didn't want to have two interactions instead of one, which is understandable. I sometimes think they truly believe that what they want is just waiting to be discovered, and I'm just not doing a good job of it.
The OP's passage highlights one of the most insidious aspects of gameification that I think often goes unsaid: it encourages you to find a holy land that may not exist, and in trying to get there, you accidentally do active harm to your goals. You lead your business down the road to hell with your gameification intentions, and your thinking gets so totally focused on this you don't see the other, probably better, options available. FWIW, I think the very same thing can be said about any company that uses the word "social" without actually backing it up with a real, compelling use case.
The first rater of a deal would get a bunch of points regardless of their rating of a deal, and so their rating didn’t matter. They would just spam (in the gamer-speak sense) the “10″ rating since it was closest to the next deal button, and do this hundreds of times without looking at the deal at all. Future voters would get points based on the crowd average, so the first voter would skew the points rating all of the deals in one sitting. It ended up that the ratings on the deals were not really accurate, thus undermining the crowdsourcing of the deals and thwarting our business goals. However, this example was more of an implementation failure than a failure of gamification in general.
I think the OP is too hard on himself here. It's very hard to imagine a game mechanic which would elicit the feedback you want: namely that you want people to give an honest opinion of something. How do you place value on an opinion? How can you tie in something so subjective to a mastery goal?
I was placed on a very similar task as part of some consulting I've been doing. I haven't found a solution, and I don't think a good one exists. The best I could come up with was that users had two interactions, not one. The first one was their honest opinion, the second one was a guess on what the prevailing opinion was. The OP only did the second, which is only going to result in a very quick convergence on groupthink. This was rejected by the company, as they didn't want to have two interactions instead of one, which is understandable. I sometimes think they truly believe that what they want is just waiting to be discovered, and I'm just not doing a good job of it.
The OP's passage highlights one of the most insidious aspects of gameification that I think often goes unsaid: it encourages you to find a holy land that may not exist, and in trying to get there, you accidentally do active harm to your goals. You lead your business down the road to hell with your gameification intentions, and your thinking gets so totally focused on this you don't see the other, probably better, options available. FWIW, I think the very same thing can be said about any company that uses the word "social" without actually backing it up with a real, compelling use case.
Seeing as you seem to be well-read in the area - what's your opinion of the "gamification" on khanacademy?
The reason I've been reading it all is because I'm writing my thesis on it :)
I haven't spent too much time with Khan Academy, so can't say too much about it. There are two things to keep in mind:
1. Extrinsic motivation erodes intrinsic motivation. The most obvious example is asking a kid to take out the trash. Once you pay him/her, they'll only do it for money again. While this example is silly because no-one has any intrinsic motivation to take out the trash, it gets really dicey when you add rewards to things like reading. Are they reading to develop a love of reading, or for the carrot you put at the end? What happens when the carrot goes away? The research suggests that once the carrot disappears, so does the reading. The psychology isn't completely cut-and-dried on this, as no psychology seems to be, but it's compelling to me, at least.
The problem is that some scholars, particularly Alfie Kohn, would say that things like gold stars are rewards, and thus you're eroding things that way. However, you have to balance that with providing legitimate feedback for knowing how well you're doing, as people want to feel competent. Both these come from self-determination theory (read up on it, it's neat). The trick is to make sure the reward doesn't try to control behavior, but rather provide feedback on behavior. eg. "you did x 5 times" (good) vs "you did x 5 times as required" (bad). It's a really fine line.
As far as I can see, the main goals in Khan Academy seem to be informational and are good.
2. Badges are game systems themselves (I heard this from Nicole Lazzaro [1]), and they will result in increasing behavior of certain things for people that pay attention. They do not necessarily result in increased behavior of value. eg. a badge for posting 5 times is earned whether your posts are insightful, or whether they simply say "first post". And, again, these are extrinsic rewards, and once they dry up, people stop caring. There's also a strong possibility that rewards suffer from inflation, and you have to hit people with bigger and bigger rewards once they start relying on them.
The badges at Khan Academy make me wary [2]. Check out this one: "Quickly & correctly answer five questions in a row." What's the goal here? Khan Academy's goal, one would think, is to encourage students to think deeply about problems and arrive to correct solutions. It spends a lot of time saying students are able to work at their own pace, and this is a benefit, but then we have a badge which encourages quick, possibly slap-dash, behavior. It's more controlling than informational, requesting the student do something not necessarily natural to him/her in order to achieve it.
The ones given out for achieving proficiency are better; they align with what they want students to do and provide informational feedback that students have done it, increasing their motivation.
====
Now, this is not to say that all achievement systems are bad or doesn't work. Stack Overflow's seems to do really well. If it was doing poorly, we'd expect a large drop off of people when the extrinsic rewards dry up, largely at reputation 10,000, when you get moderator tools. The current data from Stack Overflow indicates only about 5% or so of users drop off once they get the tools (I can dig out references if you're interested). Upvotes and achievements provide useful feedback about your mastery of SO, so with that low attrition rate, it indicates they are doing things right.
The elephant in the room that Stack Overflow presents is that it may indicate that gamification does work in its entirety, and that all this motivation theory is invalid in this domain for some reason. This is possible, but such a strong statement would require much harder evidence than we have. Right now, it's more plausible that Stack Overflow is simply doing things well and seems to be aligning with the recommendations of the research, rather than indicating a wider problem with the applicability of this stuff.
[1] http://www.xeodesign.com/founder.html [2] http://www.khanacademy.org/badges
I haven't spent too much time with Khan Academy, so can't say too much about it. There are two things to keep in mind:
1. Extrinsic motivation erodes intrinsic motivation. The most obvious example is asking a kid to take out the trash. Once you pay him/her, they'll only do it for money again. While this example is silly because no-one has any intrinsic motivation to take out the trash, it gets really dicey when you add rewards to things like reading. Are they reading to develop a love of reading, or for the carrot you put at the end? What happens when the carrot goes away? The research suggests that once the carrot disappears, so does the reading. The psychology isn't completely cut-and-dried on this, as no psychology seems to be, but it's compelling to me, at least.
The problem is that some scholars, particularly Alfie Kohn, would say that things like gold stars are rewards, and thus you're eroding things that way. However, you have to balance that with providing legitimate feedback for knowing how well you're doing, as people want to feel competent. Both these come from self-determination theory (read up on it, it's neat). The trick is to make sure the reward doesn't try to control behavior, but rather provide feedback on behavior. eg. "you did x 5 times" (good) vs "you did x 5 times as required" (bad). It's a really fine line.
As far as I can see, the main goals in Khan Academy seem to be informational and are good.
2. Badges are game systems themselves (I heard this from Nicole Lazzaro [1]), and they will result in increasing behavior of certain things for people that pay attention. They do not necessarily result in increased behavior of value. eg. a badge for posting 5 times is earned whether your posts are insightful, or whether they simply say "first post". And, again, these are extrinsic rewards, and once they dry up, people stop caring. There's also a strong possibility that rewards suffer from inflation, and you have to hit people with bigger and bigger rewards once they start relying on them.
The badges at Khan Academy make me wary [2]. Check out this one: "Quickly & correctly answer five questions in a row." What's the goal here? Khan Academy's goal, one would think, is to encourage students to think deeply about problems and arrive to correct solutions. It spends a lot of time saying students are able to work at their own pace, and this is a benefit, but then we have a badge which encourages quick, possibly slap-dash, behavior. It's more controlling than informational, requesting the student do something not necessarily natural to him/her in order to achieve it.
The ones given out for achieving proficiency are better; they align with what they want students to do and provide informational feedback that students have done it, increasing their motivation.
====
Now, this is not to say that all achievement systems are bad or doesn't work. Stack Overflow's seems to do really well. If it was doing poorly, we'd expect a large drop off of people when the extrinsic rewards dry up, largely at reputation 10,000, when you get moderator tools. The current data from Stack Overflow indicates only about 5% or so of users drop off once they get the tools (I can dig out references if you're interested). Upvotes and achievements provide useful feedback about your mastery of SO, so with that low attrition rate, it indicates they are doing things right.
The elephant in the room that Stack Overflow presents is that it may indicate that gamification does work in its entirety, and that all this motivation theory is invalid in this domain for some reason. This is possible, but such a strong statement would require much harder evidence than we have. Right now, it's more plausible that Stack Overflow is simply doing things well and seems to be aligning with the recommendations of the research, rather than indicating a wider problem with the applicability of this stuff.
[1] http://www.xeodesign.com/founder.html [2] http://www.khanacademy.org/badges
For me, Stack Overflow's most rewarding feedback is social recognition for having been helpful, which is largely independent of the formal rewards it offers. I bet it's the same for a lot of other people too.
Yes, pretty much. The other open question is what Stack Overflow would be if they removed the reputation system entirely.
The literature on motivation from people that subscribe to self-determination theory indicates I think they would mostly prefer the system gone. Any reward system you put in place is bad, it's just there are certain degrees of bad.
My gut tells me that Stack Overflow has benefitted from the reputation system as another explicit feedback on how well you are doing. I think the badges are largely unimportant, save for being a more fun way of saying "here's what you can do on the site" than another tedious FAQ. If the badges were important, then we head down a nasty extrinsic motivation road. I think it's done better with the rep system than without.
That said, I sometimes wonder how many people are put off by the built-in competitive aspect of Stack Overflow; any writing you do is evaluated and measured against others. For competitive people that end up bumped down all the time, that seems like it would be disheartening, for social people always been graded it might be disheartening too. The winner-takes-all set up, rather than, say, a wiki answer that people are constantly tweaking, seems like it could be problematic. I know that I stopped bothering answering questions very quickly as I felt I had no mastery of the art, other people were getting the upvotes, so I felt like my answers were largely ignored.
The literature on motivation from people that subscribe to self-determination theory indicates I think they would mostly prefer the system gone. Any reward system you put in place is bad, it's just there are certain degrees of bad.
My gut tells me that Stack Overflow has benefitted from the reputation system as another explicit feedback on how well you are doing. I think the badges are largely unimportant, save for being a more fun way of saying "here's what you can do on the site" than another tedious FAQ. If the badges were important, then we head down a nasty extrinsic motivation road. I think it's done better with the rep system than without.
That said, I sometimes wonder how many people are put off by the built-in competitive aspect of Stack Overflow; any writing you do is evaluated and measured against others. For competitive people that end up bumped down all the time, that seems like it would be disheartening, for social people always been graded it might be disheartening too. The winner-takes-all set up, rather than, say, a wiki answer that people are constantly tweaking, seems like it could be problematic. I know that I stopped bothering answering questions very quickly as I felt I had no mastery of the art, other people were getting the upvotes, so I felt like my answers were largely ignored.
Loving the thoughtfulness here, and I have a few more resources I need to read now.
Re: Quick and Correct badges -
It is important in many cases to be able to recall math facts quickly. Automatic recall has also been correlated with understanding of more difficult concepts (for example: [1]). I am by no means saying that we've gotten the reward system perfectly right, but we were thoughtful when creating it.
[1] http://www.ldonline.org/spearswerling/Developing_Automatic_R...
Re: Quick and Correct badges -
It is important in many cases to be able to recall math facts quickly. Automatic recall has also been correlated with understanding of more difficult concepts (for example: [1]). I am by no means saying that we've gotten the reward system perfectly right, but we were thoughtful when creating it.
[1] http://www.ldonline.org/spearswerling/Developing_Automatic_R...
I certainly did not mean to indicate I didn't think you guys had not put a lot of thought into things. If anything, my point was that all of these things are really hard, and I certainly don't count myself among the handful of people in the world who I would trust to get it right. I also have zero pedagogical chops, so don't take my word for anything!
My gut says that, if asked, I would probably recommend that the automatic recall goal is better placed as part of individual, fun "quickfire" lessons "Get these five right before the timer runs out". This would help to ensure you limit the behavior to a place where you want it, rather than somewhere where you might prefer the students to slow down and absorb what they're doing.
You could offer an informational badge showing the student completed a quickfire lesson, and I think you'd then be closer to kosher on my read of the psych literature.
But like I said, don't take my word for it :)
My gut says that, if asked, I would probably recommend that the automatic recall goal is better placed as part of individual, fun "quickfire" lessons "Get these five right before the timer runs out". This would help to ensure you limit the behavior to a place where you want it, rather than somewhere where you might prefer the students to slow down and absorb what they're doing.
You could offer an informational badge showing the student completed a quickfire lesson, and I think you'd then be closer to kosher on my read of the psych literature.
But like I said, don't take my word for it :)
Thank you for your in depth and insightful reply - what you say makes a lot of sense. Good luck with your thesis and I would personally love to see a summary of your findings posted here on HN :)
Based on going through nearly the entire Khan math test graph, I can say that the thing that was great was the instant feedback that the problems give you.
For those not familiar, you need to get many questions right in a row to move. These problems are generated along with "hints" specifically tailored to the problem - as-in the problem is worked step-by-step for you as you click "hint". What was also great is that it adapts (it seemed at least) to give you problems like the those you're having trouble on.
All of this gives you the feeling: "I'm really getting this, I'm not so bad at math after all" - all intrinsic. I looked at the badges as just "neat" - but the big chart of activity was somewhat inspiring because you can see yourself persevering. Then again - this is also intrinsic.
For those not familiar, you need to get many questions right in a row to move. These problems are generated along with "hints" specifically tailored to the problem - as-in the problem is worked step-by-step for you as you click "hint". What was also great is that it adapts (it seemed at least) to give you problems like the those you're having trouble on.
All of this gives you the feeling: "I'm really getting this, I'm not so bad at math after all" - all intrinsic. I looked at the badges as just "neat" - but the big chart of activity was somewhat inspiring because you can see yourself persevering. Then again - this is also intrinsic.
I certainly do need to interact with Khan Academy more often, I need to brush up on my probabilities for technical interviews this summer, so I'll use that as a step into the product.
FWIW, having seen Salman Khan speak, it's hard not to be swept up with his cause. He's a man on a mission, and if anyone has the vision and charisma to pull off Khan Academy's goals, it's him.
FWIW, having seen Salman Khan speak, it's hard not to be swept up with his cause. He's a man on a mission, and if anyone has the vision and charisma to pull off Khan Academy's goals, it's him.
I think an important distinction is also that the purpose of the math test graph is to specifically quantify prior learning (ie. it provides a reasonably realistic measurement of learning) -- this is not gamification necessarily. The badges are gamification. Our goal should be realistic and effective quantification of learning/improvement (passive feedback that generally increases intrinsic motivation), rather than implementing external motivators or artificial targets (which badges typically are).
"people were rating thousands of deals for free to get on an anonymous leaderboard in a small game universe"
This was something I observed on a project of my own (albeit slightly differently), and it was a double-edged sword.
On the one hand, these people would increase their activity on the app so as to get more points and rank higher. It wasn't long before we got people trying to game the system (we didn't do a great deal to prevent it, there was nothing at all clever about our implementation), or trying to spam a few of our API endpoints.
The result of this was the impression of increased popularity. Our counters would rise, and we'd look busier. Score.
On the other hand, we got a massive bump in superfluous activity and little to no insight into real user behaviour (other than suggestions we had a bunch of spammers).
This created an unusual situation where we didn't want to block the spammers, because it made us look good. But we wanted to block the spammers, because they were spamming us to get on the leaderboard (which offered financial incentives for placing highly on).
But, other than that, this sort of 'gamification' was, to someone who'd never implemented or truly observed it, a revealing insight into what lengths some people will go to just to see their name at the top of a table, or on a page as the last active user, or whatever.
I think I find my own experience has a fair bit in common with the author of this article's.
This was something I observed on a project of my own (albeit slightly differently), and it was a double-edged sword.
On the one hand, these people would increase their activity on the app so as to get more points and rank higher. It wasn't long before we got people trying to game the system (we didn't do a great deal to prevent it, there was nothing at all clever about our implementation), or trying to spam a few of our API endpoints.
The result of this was the impression of increased popularity. Our counters would rise, and we'd look busier. Score.
On the other hand, we got a massive bump in superfluous activity and little to no insight into real user behaviour (other than suggestions we had a bunch of spammers).
This created an unusual situation where we didn't want to block the spammers, because it made us look good. But we wanted to block the spammers, because they were spamming us to get on the leaderboard (which offered financial incentives for placing highly on).
But, other than that, this sort of 'gamification' was, to someone who'd never implemented or truly observed it, a revealing insight into what lengths some people will go to just to see their name at the top of a table, or on a page as the last active user, or whatever.
I think I find my own experience has a fair bit in common with the author of this article's.
At dinner the other night, our five year old and his five year old friend wouldn't eat their broccoli--until I told them that broccoli was worth 2000 points. The points had no reward attached, other than getting "points", and making sure the other guy didn't have more imaginary points.
Not entirely but Is gamification mostly bullshit?
2) A lot of people start using X to get the same results.
3) A "consulting" industry starts to rise around X.
4) Someone gets tired of the overused X and calls it bullshit.
5) Everyone that didn't succeeded with X, probably after some kind of investment inspired by the new "consulting" industry, gets in wagon and calls it bullshit.
6) Some time after the "bullshit" narrative sets in, someone finds out that X can be useful. If he tries to defend it he either does it by carefully arguing "I'm on your side but...", or just change it's name.
This kind periodic behaviour seems to deamplificate until it reach some stable state, generally it's absorbed by academia and gets to be taught in schools. From there it can get some amplifications, and if it does the pattern repeats.