The Psychology of Tackling Hard Problems(davidvaldman.com)
davidvaldman.com
The Psychology of Tackling Hard Problems
http://davidvaldman.com/post/20027940591/psychofhardproblems
57 comments
A lot of people like to repeat the idea that "failure is a learning experience" but it is only recently that the essence of this idea in practical terms became concrete for me (thanks to "177 Mental Toughness Secrets of the World Class"): you need to be prepared to fail, which - among other things - means you need to be aware of resisting failure.
Resisting failure can manifest itself very subtly, but the most common manifestation for me is resisting something because it involves new learning ("I just want to solve the core problem, this is never going to end!"), or doing something that I "know" I won't be as good as others at e.g. the user interface.
It's obviously better to get something working than not, and knowing this keeps me committed to working in principle. So I don't abandon the project outright. What actually happens is more insidious: I do something else to avoid the pain of dealing with the problem at hand. As time passes, I (a) lose momentum on the project and (b) the likelihood of a bright, new shiny idea occurring to me, with challenges I "know" I can solve, increases.
When (b) happens, I move on, and it feels good because I'm making progress whereas with the other project I "couldn't" make progress. I never specifically or intentionally abandon the original project, but in effect I do, with the rationalisation that I have a higher priority item to focus on i.e. something I believe I can 'make progress' on. At least, as it turns out, according to my dysfunctional definition of progress, which is rooted in perfectionism.
What I've learned is to monitor my objective progress: did I take steps to solve the problem, and can I solve it for the bulk of the use cases that are really important, and if not can I find somebody else to solve it? e.g. do I have a user interface yet or not, and do I need to find somebody to do it for me? Not my subjective progress: e.g. "I really want this to look great and professional like all those other sites out there, and I'll never get there because I don't know how and I don't want to blow money on this because I'm not even sure how I want it to look, maybe I should research more and find something similar, this is boring it's not what I want to do, I'm making no progress, etc."
Resisting failure can manifest itself very subtly, but the most common manifestation for me is resisting something because it involves new learning ("I just want to solve the core problem, this is never going to end!"), or doing something that I "know" I won't be as good as others at e.g. the user interface.
It's obviously better to get something working than not, and knowing this keeps me committed to working in principle. So I don't abandon the project outright. What actually happens is more insidious: I do something else to avoid the pain of dealing with the problem at hand. As time passes, I (a) lose momentum on the project and (b) the likelihood of a bright, new shiny idea occurring to me, with challenges I "know" I can solve, increases.
When (b) happens, I move on, and it feels good because I'm making progress whereas with the other project I "couldn't" make progress. I never specifically or intentionally abandon the original project, but in effect I do, with the rationalisation that I have a higher priority item to focus on i.e. something I believe I can 'make progress' on. At least, as it turns out, according to my dysfunctional definition of progress, which is rooted in perfectionism.
What I've learned is to monitor my objective progress: did I take steps to solve the problem, and can I solve it for the bulk of the use cases that are really important, and if not can I find somebody else to solve it? e.g. do I have a user interface yet or not, and do I need to find somebody to do it for me? Not my subjective progress: e.g. "I really want this to look great and professional like all those other sites out there, and I'll never get there because I don't know how and I don't want to blow money on this because I'm not even sure how I want it to look, maybe I should research more and find something similar, this is boring it's not what I want to do, I'm making no progress, etc."
Attempting to learn music late (~20)taught me this.
Start with a strong theory that you pursue until you know it's 99% wrong, sob ,get a new hunch (if you're really masoc.. passionated), repeat.
After many cycles likes these two things happens :
- the tunnel ain't that dark anymore, you can see shades, sometimes even shiny spots*. You get a carnal feeling that those little details you dismissed earlier have a lot more value.
- you went from 100% blind to 99.9% blind.
Tackling something vast and unknown by yourself should be discussed more often, I feel it's a large human nature myth in `modern` cultures. We rely too much on society to accelerate problem solving and get stuck into a little spot of dependencies.
ps: oh and by the way, fear/anger triggered by failure was adressed by stoicists somehow. ( at least according to wikipedia ~_~ )
Start with a strong theory that you pursue until you know it's 99% wrong, sob ,get a new hunch (if you're really masoc.. passionated), repeat.
After many cycles likes these two things happens :
- the tunnel ain't that dark anymore, you can see shades, sometimes even shiny spots*. You get a carnal feeling that those little details you dismissed earlier have a lot more value.
- you went from 100% blind to 99.9% blind.
Tackling something vast and unknown by yourself should be discussed more often, I feel it's a large human nature myth in `modern` cultures. We rely too much on society to accelerate problem solving and get stuck into a little spot of dependencies.
ps: oh and by the way, fear/anger triggered by failure was adressed by stoicists somehow. ( at least according to wikipedia ~_~ )
In psychology this problem is called auto-handicap (or http://en.wikipedia.org/wiki/Self-handicapping) and generally is a strong force behind any form of procrastination.
No, that's a reason to turn back. We're talking about the opposite - why someone would not only try, but keep going even after countless data points indicate failure.
3 reasons to not turn back in 2 categories
Category A : walking in the dark tunnel is enjoyable
1) having fun so keep walking
Category B: its not fun
2) Peer pressure, either family, friends, investors, cofounders/employees
3) Expected upside at the end of the tunnel is greater than the walk back
In reality its a combination of 2) & 3) and why pg&co strongly recommend co-founders. e.g. 3) starts you out, 2) keeps you going.
Category A : walking in the dark tunnel is enjoyable
1) having fun so keep walking
Category B: its not fun
2) Peer pressure, either family, friends, investors, cofounders/employees
3) Expected upside at the end of the tunnel is greater than the walk back
In reality its a combination of 2) & 3) and why pg&co strongly recommend co-founders. e.g. 3) starts you out, 2) keeps you going.
I would phrase category A differently: when the work is valuable for its own sake. That's not necessarily fun. It's more likely grueling. Even when it's fun, the fun parts don't add up to nearly as much fun as the painful parts do pain. Yet one can continue because one senses (I almost wrote "knows" - can one know?) that the work is valuable, necessary, important. That's deeper than fun.
What I find interesting is that this comes from within. The other two motivations you list are external. I wonder if the external ones burn out sooner.
When your motivation is intrinsic (that is, you have a felt sense of the work and you know what it is and you think it's important) you have a basis for deciding when to listen or not listen to other people's opinions. But when it's extrinsic, you're more likely to do what other people say and perhaps give up if it doesn't work.
Edit: another thing about intrinsic motivation is that because other people don't have the same sense that you do, what you're doing may seem absurd and they will tell you you're doing it wrong. So the challenge of this path is not just the work is hard and that you fail a lot; it's also that it will probably force you beyond the pale of social proof, something humans are not hard-wired to enjoy.
What I find interesting is that this comes from within. The other two motivations you list are external. I wonder if the external ones burn out sooner.
When your motivation is intrinsic (that is, you have a felt sense of the work and you know what it is and you think it's important) you have a basis for deciding when to listen or not listen to other people's opinions. But when it's extrinsic, you're more likely to do what other people say and perhaps give up if it doesn't work.
Edit: another thing about intrinsic motivation is that because other people don't have the same sense that you do, what you're doing may seem absurd and they will tell you you're doing it wrong. So the challenge of this path is not just the work is hard and that you fail a lot; it's also that it will probably force you beyond the pale of social proof, something humans are not hard-wired to enjoy.
"That is, at some point, having walked down the tunnel for quite some time, the rational thing would appear to be to turn back. Why would someone not turn back?"
Sunk cost fallacy: http://en.wikipedia.org/wiki/Sunk_costs#Loss_aversion_and_th...
Sunk cost fallacy: http://en.wikipedia.org/wiki/Sunk_costs#Loss_aversion_and_th...
"Fallacy" implies it's a bad idea, which implies hindsight, which is itself a fallacy.
I have to agree with this. I'm also in math grad school after numerous startups. The hardest part about transitioning from math undergrad to graduate school is that you know assign problems in undergrad have a solution, you just have to go out and find it. Research problems just might not have a solution in reach of existing technology.
However, I find the difficulty and uncertainty of research mathematics much worse than anything I encountered even in research-heavy startups. Depends on what you're working on, I guess.
However, I find the difficulty and uncertainty of research mathematics much worse than anything I encountered even in research-heavy startups. Depends on what you're working on, I guess.
The idea of fear in research is not a new one. I'm reminded of the classic quotes by Dirac, "Scientific progress advances in units of courage, not intelligence." and of the classic essay by Ivan Sutherland, "Courage and Technology."
I think fear can be a valuable signal. If your success rate is a discouraging 1%, fear might be trying to tell you that you need to find a smarter way to solve the problem than blind (or even moderately educated) guessing. Often in mathematics, at least, you can discover things about the structure of a solution before you find it, things which in turn can help you find the solution. Often the discouragement of a few failed attempts pushes me to step back and trying to think about the problem in a deeper and smarter way.
I think fear can be a valuable signal. If your success rate is a discouraging 1%, fear might be trying to tell you that you need to find a smarter way to solve the problem than blind (or even moderately educated) guessing. Often in mathematics, at least, you can discover things about the structure of a solution before you find it, things which in turn can help you find the solution. Often the discouragement of a few failed attempts pushes me to step back and trying to think about the problem in a deeper and smarter way.
[deleted]
"There’s a simple reason why tackling a hard problem can lead to depressive symptoms: you’re necessarily wrong 99% of the time."
This immediately rang out to me...any time I'm having problems outside of working (relationship/depression), it kills my ability to be resilient and work through a problem until it's solved..I think working through emotional pain is an excellent skill on its own.
This immediately rang out to me...any time I'm having problems outside of working (relationship/depression), it kills my ability to be resilient and work through a problem until it's solved..I think working through emotional pain is an excellent skill on its own.
One can not do right in one area of life whilst he is occupied in doing wrong in another. Life is one indivisible whole. - Gandhi
The more I live, the more I realize that this statement isn't about moralizing and is about a fundamental truth about our psychology.
The more I live, the more I realize that this statement isn't about moralizing and is about a fundamental truth about our psychology.
I discovered this truth myself in my senior year of high school. Prior to that year I got through high school doing roughly no work outside of class and managed to get low A's in my classes through intellect. I also competed in track and field throughout high school somewhat seriously.
However, in my senior year of high school, I ended up putting in much more time and dedication to track (as well as joining the cross country team, making it an all year commitment), so I spent an average of ~22ish hour/wk doing workouts related to that over the course of the year. I expected the little time I spent on school to drop to zero, however I was surprised that I was actually much more motivated to work on my relatively easy/unstimulating schoolwork. After working hard at running before and after school, it just felt unnatural to not work hard at schoolwork.
However, in my senior year of high school, I ended up putting in much more time and dedication to track (as well as joining the cross country team, making it an all year commitment), so I spent an average of ~22ish hour/wk doing workouts related to that over the course of the year. I expected the little time I spent on school to drop to zero, however I was surprised that I was actually much more motivated to work on my relatively easy/unstimulating schoolwork. After working hard at running before and after school, it just felt unnatural to not work hard at schoolwork.
Studying Zen made me realize that most of the teachings which people interpret moralistically (or theologically) actually point to a psychological reality.
At least with Buddhist teachings, I'd definitely agree. Similarly with some of the Ancient Greek schools' teachings (Epicureans, Stoics, etc), even in parts of their mysticism. I don't know enough about, say, Hinduism, Taoism, or Shinto to comment on those, but I get the impression they're fairly similar in that regard, too.
However, I can't help but feel that a lot of Abrahamic teachings seem to be mostly about increasing the mimetic fitness of the faith. For instance: "go forth and multiply" is always good for keeping the size of your congregation up; evangelism likewise; "you shall not boil a kid in its mother's milk" reads to me as "don't eat your future dairy animals", a handy survival tip in the bronze age; and so forth.
However, I can't help but feel that a lot of Abrahamic teachings seem to be mostly about increasing the mimetic fitness of the faith. For instance: "go forth and multiply" is always good for keeping the size of your congregation up; evangelism likewise; "you shall not boil a kid in its mother's milk" reads to me as "don't eat your future dairy animals", a handy survival tip in the bronze age; and so forth.
[deleted]
Hackers frequently have an idea, build 90% of an MVP and then never launch. Why would they get so close and never cross the finish line?
It seems obvious: if you don't launch, you can't fail.
A subtle fear of rejection is understandably common in a population that experienced it frequently in their formative years.
It seems obvious: if you don't launch, you can't fail.
A subtle fear of rejection is understandably common in a population that experienced it frequently in their formative years.
I think the key is in not seeing "being wrong" as a "failure"
"I've failed" != "I'm a failure"
No one gets everything right all of the time. Accepting that and seeing hard problems as a challenge to overcome rather than a potential failure waiting to happen is, in my opinion, the way forward.
"I've failed" != "I'm a failure"
No one gets everything right all of the time. Accepting that and seeing hard problems as a challenge to overcome rather than a potential failure waiting to happen is, in my opinion, the way forward.
It's not so much that you see yourself as a failure, in my experience--it's more that, without frequent successes (however tiny) to push you forward, you run out of steam--that is, dopamine--and find it hard to want to try the next thing. It's the opposite of getting addicted to something: with no bells and dings and coins falling out, you become apathetic.
>>..you run out of steam--that is, dopamine
From: http://en.wikipedia.org/wiki/Paul_Erd%C5%91s
"His colleague Alfréd Rényi said, "a mathematician is a machine for turning coffee into theorems", and Erdős drank copious quantities. (This quotation is often attributed incorrectly to Erdős himself. The German original, "Ein Mathematiker ist eine Maschine, die Kaffee in Sätze verwandelt"[11] of the sentence is a wordplay on the double meaning of "Satz": "theorem" or "residue of coffee", lost in the English translation)[12]
After 1971 he also took amphetamines, despite the concern of his friends, one of whom (Ron Graham) bet him $500 that he could not stop taking the drug for a month.[13] Erdős won the bet, but complained that during his abstinence mathematics had been set back by a month: "Before, when I looked at a piece of blank paper my mind was filled with ideas. Now all I see is a blank piece of paper." After he won the bet, he promptly resumed his amphetamine use."
As they say in The Men Who Stare at Goats, "Amphetamines- Not to be abused, but very fucking handy."
EDIT / Disclaimer: If you (the reader) decide to try this, do your own, thorough research. It IS risky and it can end very badly for some people. You've been warned.
Lighter:
Nootropcis, which may be interesting to the hackers aboard. http://en.wikipedia.org/wiki/Nootropic
Focus XT has been nice for me (like a steady pot of coffee (which is inefficient with tolerance building) at a steady rate without any jittery crashes.
https://www.google.com/search?sourceid=chrome&ie=UTF-8&#...
Nootropcis, which may be interesting to the hackers aboard. http://en.wikipedia.org/wiki/Nootropic
Focus XT has been nice for me (like a steady pot of coffee (which is inefficient with tolerance building) at a steady rate without any jittery crashes.
https://www.google.com/search?sourceid=chrome&ie=UTF-8&#...
Dgrove, quick heads up: you're hellbanned, and most people can't see your posts. Your recent comments have been positive, so I'm letting you know. Try e-mailing PG about it.
One idea: positively reinforce the behavior of taking a solid swing at your problem regardless of whether you successfully solve it.
In other words, cheer yourself on for ruling out apparently promising solutions as well as successfully solving problems.
In other words, cheer yourself on for ruling out apparently promising solutions as well as successfully solving problems.
[deleted]
Sounds like what I've read of the quest for the Higgs Boson -- "We know it's not here, I'd say that's progress! Now let's look over there instead!"
Good post--I'm in the midst of doing a PhD on the side while working, and I've noticed that "I'm afraid it won't work" behavior in myself during both my research and working on hard problems in my day job. Glad to know I'm not alone!
Too many comments here are a re-phrase of "learn from your failure" quote. I hate to sound too negative here, but people learn from their success a whole lot than they learn from their failure. It sounds like something we just tell ourselves after we've failed to make ourselves feel better.
Though I agree that keep trying (and obviously failing many times along the way) is often necessary to get to where you want to go.
Though I agree that keep trying (and obviously failing many times along the way) is often necessary to get to where you want to go.
I'm afraid you've missed the point, which is about the psychology of failing 99% of the time. That's not, like, a little hard. It is excruciating.
We use words like "persistence" or "learn from your failures" so much that we think we know what they mean; so much, in fact, that they become boring. That confuses repetition with understanding. What's valuable about the OP (and the comments are just reflecting this) is that he put into words a tiny bit of that experience in a way that isn't just repetitive.
We use words like "persistence" or "learn from your failures" so much that we think we know what they mean; so much, in fact, that they become boring. That confuses repetition with understanding. What's valuable about the OP (and the comments are just reflecting this) is that he put into words a tiny bit of that experience in a way that isn't just repetitive.
Really? I disagree, though exaggerated, here's another quote, from the end of Little Miss Sunshine, about Proust:
"...French writer. Total loser. Never had a real job. Unrequited love affairs. Gay. Spent 20 years writing a book almost no one reads. But he's also probably the greatest writer since Shakespeare. Anyway, gets down to the end of his life, and he looks back and decides that all those years he suffered, Those were the best years of his life, 'cause they made him who he was. All those years he was happy? You know, total waste. Didn't learn a thing."
"...French writer. Total loser. Never had a real job. Unrequited love affairs. Gay. Spent 20 years writing a book almost no one reads. But he's also probably the greatest writer since Shakespeare. Anyway, gets down to the end of his life, and he looks back and decides that all those years he suffered, Those were the best years of his life, 'cause they made him who he was. All those years he was happy? You know, total waste. Didn't learn a thing."
Makes me think about how, in college, I used to put down a math problem as soon as I thought I knew how to do it. I would put off doing it, and I never understood why. This was totally the reason! It seems obvious since, as you said, when I went about actually fleshing out the answer, I would usually discover something that was wrong. Great observation.
I think the effect is quite general. I definitely notice what you mention in the students I teach/tutor. Many times there are gaps in our knowledge as well, and instead of filling them, we get frustrated and "put the book down."
Strongly agreed, and great insight on the everyday struggle of a research heavy life or career!
It can be a heavy burden at times, but being prepared mentally helps a lot.
As my dad always taught me growing up, "attitude is everything."
Plus, tackling a large stack of problems, and iterating until successful feels so rewarding. Both during and after the fact.
Failure is key to success.
It can be a heavy burden at times, but being prepared mentally helps a lot.
As my dad always taught me growing up, "attitude is everything."
Plus, tackling a large stack of problems, and iterating until successful feels so rewarding. Both during and after the fact.
Failure is key to success.
I remember reading the first page of a juggling book was: Pick up a ball. Drop it on the floor. Be prepared to do this thousands of times.
Kind of takes the fun out of it eh?
Kind of takes the fun out of it eh?
Haha! Excellent quote. Are we talking about life or juggling, or wait, business? ;)
Never gave much thought to my early interest in juggling or the juggling books I read when I was young and how it could possibly be later related to my approach to life and career.
I'm a modest juggler at best, and still young in my professional career in the tech world, but that quote and general sentiment gives me new perspective on how my opinion of failure was shaped growing up.
Never gave much thought to my early interest in juggling or the juggling books I read when I was young and how it could possibly be later related to my approach to life and career.
I'm a modest juggler at best, and still young in my professional career in the tech world, but that quote and general sentiment gives me new perspective on how my opinion of failure was shaped growing up.
To expose yourself to positive black swans you have to take on this psychic toll. This is why it's human nature to prefer to bias for small regular wins and the occasional catastrophe.
http://www.gladwell.com/2002/2002_04_29_a_blowingup.htm
http://www.gladwell.com/2002/2002_04_29_a_blowingup.htm
Great read! This really ties up together with the idea of "keep shipping" - while working on a startup recently - I found myself spending a lot of time thinking about what if this or what if that. The kind of logic that paralyzes you. This is specifically a problem for perfectionists and very intelligent people - you're either afraid it wont be perfect or spend so much time thinking of the most efficient or beautiful solution that in the mean time someone else - may have already solved it. That experience really taught me to keep shipping - stuff wont work - even if it isn't one of those massively hard problems - sometimes things just dont work. The trick is to keep shipping and that 1% will go up to much more!
In the scientific community, things are never a failure. Instead, it's something we like to call 'negative data'.
You learn to embrace the negative data. At the very least, you've just ticked off one thing that assuredly does not resolve the issue.
You learn to embrace the negative data. At the very least, you've just ticked off one thing that assuredly does not resolve the issue.
I like this paraphrased quote: "An expert is a person that has made every possible mistake within his or her field." -Niels Bohr
That way, each mistake I make (and realize/correct) is me leveling up
That way, each mistake I make (and realize/correct) is me leveling up
"I'm afraid that it won't work." I know this may sound silly, but I honestly just realized how wrong that statement is. I have actually used it to dismiss a potential solution...
There are actually two difficulties here:
1. The problem in hand may not have a good solution. This is especially true of research type problems. You have to pick the right one, otherwise it can be a frustrating experience. This is where a broad knowledge of the subject and previous research help.
2. Some potential solutions may be very difficult to implement or explore, so one tends to explore the simpler options first. When these are not enough, you tend to loose faith.
1. The problem in hand may not have a good solution. This is especially true of research type problems. You have to pick the right one, otherwise it can be a frustrating experience. This is where a broad knowledge of the subject and previous research help.
2. Some potential solutions may be very difficult to implement or explore, so one tends to explore the simpler options first. When these are not enough, you tend to loose faith.
iteration = trial-and-error = try-and-fail
For me, the hard part is that an experiment can be a lot of work... all that effort, for NOTHING. :(
Though Tesla has more geek cred these days, Edison also did some stuff. He said: (http://en.wikiquote.org/wiki/Thomas_Edison)
For me, the hard part is that an experiment can be a lot of work... all that effort, for NOTHING. :(
Though Tesla has more geek cred these days, Edison also did some stuff. He said: (http://en.wikiquote.org/wiki/Thomas_Edison)
I have not failed. I've just found 10,000 ways that won't work."...all that effort, for NOTHING" can be an easy belief to fall into, but it really wasn't for nothing. If you think about the work you put in before this point in time as a sunk cost, and look at it as "well, where can I start this problem from now?" you'll see that your starting point is a WHOLE lot different. You find yourself with a much more informed opinion from which to climb up from.
This is true. Knowing what doesn't work is a great way to constrain the future search space.
Anybody else think of that scene from Home Alone when Mcauley Culkin says "Hey, I'm not afraid any more! I said I'm not afraid any more! Do you hear me? I'm not afraid any more!
[Old Man Marley approaches Kevin and stares at him - Kevin runs back inside, screaming like a maniac]
Nope, only you.
I can't agree with this more...this especially rings true here in the valley. Its a tightrope walk to synthesize feedback from failures efficiently and effectively while not breaking pace and letting your emotions have you pause on the undesired outcome. Persistence is magical.
I would add that a second difficulty insolving tough problems is being able to create solutions.
There's definitely that too. But to get at the creative solution, you usually have to wade through tons of wrong options. The key is persistence.
[deleted]
I think part of the fear for me is that if the solution in my head doesn't work I won't be able to come up with another one, but in reality even if the initial solution is not correct it usually leads to ideas of how to change it or how to re-think the problem from a different angle.
I suppose that's why it's better to like knowing the ground truth, than having ideas. You delight in finding the actual answer, than what you think might be the answer.
So is no one going to bring up the tried-and-true weed and coffee method to coding motivational success? Fine, let me be that guy.
I hear ya partner. Nothing gets me going like the old hippie speedball.
One point I'd like to add is that history gives a very misleading picture. History is divided into people who worked through hard problems and triumphed (these are called "geniuses") and people who either gave up or died trying (these are called "failures"). Those seem like opposites, but it's only hindsight that makes them so. Psychologically, when you're in the middle of it, which is when it really matters, there's no way to know. All you know for sure is that you've been walking down a dark tunnel for a really long time.
An interesting question, more interesting than it appears, is: why would someone do that? That is, at some point, having walked down the tunnel for quite some time, the rational thing would appear to be to turn back. Why would someone not turn back?