I know for a fact that at least 7 of the top AI people that I listed above think intelligence explosion / superintelligence is plausible and that it's not at all clear that will go well for humans. I'm pretty sure all of them think AGI will be feasible one day and will be fairly hard to make reliably safe, at the very least to the degree that current safety-critical systems require a bunch of extra work to make safe (beyond what you'd do if your AI system has no safety implications, e.g. detecting cats on YouTube).
I can't say which people believe which things, because most of them communicated with me in confidence, except for Stuart Russell and Shane Legg because they've made public comments about it about their worries about intelligence explosion:
> On the other hand you have (essentially) the entire mainstream academic AI/ML community, who are mainly concerned that the hype is so intense it might lead to another AI winter
This doesn't seem true. E.g. the signatories on the Future of Life Institute's recent open letter about AI progress and safety include:
* Stuart Russell & Peter Norvig, co-authors of the #1 AI textbook
* Tom Dietterich, AAAI President
* Eric Horvitz, past AAAI President
* Bart Selman, co-chair of AAAI presidential panel on long-term AI futures
* Francesca Rossi, IJCAI President
* All founders of Deep Mind and Vicarious, two leading AI companies
Goldman Sachs did not donate to the charity that plausibly saves lives at a price of $2500 (Against Malaria Foundation). If I had to guess, they gave it to groups who don't even try to save lives, or to groups who save lives at a cost 100 times or more greater than $2.5k/life.
If you want to become a better futurist, there's nothing like lots of practice for improving your calibration. The best way I can think of to get lots of practice at technology forecasting is to sign up for George Mason University's DAGGRE program, which will let you participate in a multi-year prediction tournament focused on science and technology forecasting. Details here: http://is.gd/qHnaOY
Paul Graham is one of my favorite writers. He's engaging, surprising, simple, and brief. And he writes on topics of interest to everyone. Does anyone else know good essayists that share these same qualities?
Epic. In my dreams, this article will become so popular that political commentators will use your Disagreement Hierarchy to evaluate political speeches. (Most political debate rarely rises above DH4.)
I'm going to reference your article every time I respond to criticism. Thank you.
I can't say which people believe which things, because most of them communicated with me in confidence, except for Stuart Russell and Shane Legg because they've made public comments about it about their worries about intelligence explosion:
http://futureoflife.org/PDF/stuart_russell.pdf
http://lesswrong.com/lw/691/qa_with_shane_legg_on_risks_from...