Fictional peer reviews rejecting famous papers (2005)(th.informatik.uni-mannheim.de)
th.informatik.uni-mannheim.de
Fictional peer reviews rejecting famous papers (2005)
http://th.informatik.uni-mannheim.de/People/Lucks/reject.pdf
15 comments
Hrm, I once took part in preparation of a grant application related to some algorithms for dynamic frequency assignment in radio networks. The reviewer(s) pointed out that there already exist official institutions that take care of assigning radio frequencies. Sadly, this is not a joke.
Whilst amusing, I find it hard to believe that the repeated excuse of "not practical" would ever have been given. Do academics really have such strong objections to ideas which are purely academic?
"Structured programming is a nice academic exercise, which works well for small examples, but I doubt that any real- world program will ever be written in such a style."
"The paper contains no real-world example to convince us that any model of practical interest can be cast in it"
"Structured programming is a nice academic exercise, which works well for small examples, but I doubt that any real- world program will ever be written in such a style."
"The paper contains no real-world example to convince us that any model of practical interest can be cast in it"
I used to work in a quasi-academic development and production environment (DOD S&T[1]), and I heard that excuse quite a bit. Usually it was because someone had a pet idea that would be threatened or was too afraid of the risk of moving from the established development line.
[1] http://www.acq.osd.mil/chieftechnologist/index.html
[1] http://www.acq.osd.mil/chieftechnologist/index.html
It's much worse than the article describes. I've solved five outstanding problems in physics, or so I claim. Just little ones like dark energy. I dedicated a decade to the effort (not that that proves anything, I'm just saying I didn't slap anything together). Yet I can't get a single paper read. Every editor rejects based on subject matter. A journal on gravity, for example, is in reality a journal on general relativity, so they won't consider a paper that purports a problem with that theory. In informally researching the issue further, the vast majority of physicists I've encountered support the blanket rejection of any paper that purports an issue with existing physics. They'd rather keep their outstanding problems, even physical paradoxes. (Arxiv isn't an option. Papers are regularly deleted there by mods for the same reasons as above.)
So yes, I can easily believe the repeated excuse of "not practical". I've even got angry responses for my subjects.
So yes, I can easily believe the repeated excuse of "not practical". I've even got angry responses for my subjects.
Either that - or you're a crank. We'd have to read the actual paper to decide.
As I said, no one who could expertly read my papers will read them. So while I'm a crank, a crackpot, nowadays that's based on subject matter alone.
Why don't you post the papers on your own website?
I've done that, FWIW, which isn't much. The solutions will likely be lost to science. Perhaps a future internet archaeologist will find and disseminate them to a more scientific audience.
I understand that frustration can be hard to deal with but this attitude really doesn't advance your cause. No one likes to talk to whiny emo people. (Except at goth clubs where it can be kind of fun.)
But I'm not frustrated, nor whining. I have no cause to advance; I no longer seek publication of my ideas.
Your comment demonstrates the following thinking I've seen in today's scientific community though: If you have anything to say against blanket rejection of ideas based on subject matter, you're an even bigger crackpot (or whiner) than we assumed you were.
Your comment demonstrates the following thinking I've seen in today's scientific community though: If you have anything to say against blanket rejection of ideas based on subject matter, you're an even bigger crackpot (or whiner) than we assumed you were.
Because in most cases, that conclusion is correct as well.
Your refusal to link to the actual PDFs doesn't inspire confidence either. You claim breakthroughs and leave us hanging, which isn't very nice.
Your refusal to link to the actual PDFs doesn't inspire confidence either. You claim breakthroughs and leave us hanging, which isn't very nice.
Linking would be going too far off topic, which I won't do. It would indicate an ulterior motive I don't have.
You'd need at least a good laymen's background in relativity theory to understand my work. If you have that, you can give me a way to contact you and I'll do that after this discussion has scrolled off others' lists.
You'd need at least a good laymen's background in relativity theory to understand my work. If you have that, you can give me a way to contact you and I'll do that after this discussion has scrolled off others' lists.
You could also link to your website in your user summary, and have it easily accessible for tangential conversations.
Link?
You sound like a nice guy who has been working hard (though I think it's unlikely that you've legitimately "solved five outstanding problems in physics") and I would like to help you. I will try:
You have a problem, which is that most of the people who make claims like yours are cranks, which I'll define as "people who are emotionally committed to valueless ideas and with whom it is essentially impossible to have a productive conversation, and who are unaware that they are a crank." And most of the people who "could expertly read your papers" have had their time wasted by crank after crank, and gotten tired of it, so they have developed defenses. It's like the spam filter you probably use on your inbox. Maybe you really were given a lot of money by a stranger in Nigeria. It's barely conceivable, and if it's true, it could be very valuable to you. But you filter out such claims because (1) they are so unlikely to be true that the expected value of reading them is still negative, (2) you really have no choice in any event, because you get so many that if you read them all you wouldn't get anything else done.
Compounding your problem is that ideas like yours take a long time and a lot of effort to evaluate. For example, it's hard to understand general relativity - one has to commit some serious energy to it. If your ideas are right, they probably also take a lot of effort to evaluate. Here's the problem: you are not entitled to demand that effort of other people. Food, shelter, freedom, and medical care might be part of your birthright as a human being (though some disagree even as to that) but the right to have experts study your theories is definitely not included in that birthright. You're asking them to take time off from their own pursuits (like their own research, or spending time with their kids) to work on your pursuit. They might do it for a friend, or they might do it because they're curious, or they might do it because they think it could be valuable to them. But you're not entitled to it -- you have to figure out how to get them interested. The ball's in your court on that one.
So, you need to stop whining and develop the social skills that are necessary for a successful career in science. I am not a professional scientist, but many of my college friends are, and I have watched them struggle over the last decade to develop their social skills and their connections so that they can get their ideas understood and taken seriously. For some of them that's harder than their actual research -- but there's not much choice. It's part and parcel of the work. A researcher has two jobs, coming up with valuable ideas, and then getting other people interested in those ideas.
I wish I had some specific advice for you but I don't really know how it works in physics. But I'm sure that if you're really smart enough to solve "five outstanding problems in physics" then you'll be able to figure it if you apply yourself. A good first step might be to go to some physics conferences, meet some people who practice in your field, and befriend them. Show a lot of interest in their research, let them talk about it, ask intelligent questions (it's flattering when people do that.) Don't mention your own research - be vague ("oh, I've been interested in this aspect of gravitation too, which is why I'm so curious about your ideas.") Try to find small ways to help out the people you meet. If you do this for a while, you will eventually make some friends, or at least professional contacts. Then you will at least be able to ask them to take a serious look at your ideas. ("Hey, I wonder if you could help me out with something. I've had some ideas about [unsolved problem] and I know that you're an expert in it - it's always your name that comes up when I ask around. I'd love to get your opinion. Is there a time that we could chat for 15 or 20 minutes?") At least, that's what I'd do if I didn't have any contacts in a field, and needed to make some.
You have a problem, which is that most of the people who make claims like yours are cranks, which I'll define as "people who are emotionally committed to valueless ideas and with whom it is essentially impossible to have a productive conversation, and who are unaware that they are a crank." And most of the people who "could expertly read your papers" have had their time wasted by crank after crank, and gotten tired of it, so they have developed defenses. It's like the spam filter you probably use on your inbox. Maybe you really were given a lot of money by a stranger in Nigeria. It's barely conceivable, and if it's true, it could be very valuable to you. But you filter out such claims because (1) they are so unlikely to be true that the expected value of reading them is still negative, (2) you really have no choice in any event, because you get so many that if you read them all you wouldn't get anything else done.
Compounding your problem is that ideas like yours take a long time and a lot of effort to evaluate. For example, it's hard to understand general relativity - one has to commit some serious energy to it. If your ideas are right, they probably also take a lot of effort to evaluate. Here's the problem: you are not entitled to demand that effort of other people. Food, shelter, freedom, and medical care might be part of your birthright as a human being (though some disagree even as to that) but the right to have experts study your theories is definitely not included in that birthright. You're asking them to take time off from their own pursuits (like their own research, or spending time with their kids) to work on your pursuit. They might do it for a friend, or they might do it because they're curious, or they might do it because they think it could be valuable to them. But you're not entitled to it -- you have to figure out how to get them interested. The ball's in your court on that one.
So, you need to stop whining and develop the social skills that are necessary for a successful career in science. I am not a professional scientist, but many of my college friends are, and I have watched them struggle over the last decade to develop their social skills and their connections so that they can get their ideas understood and taken seriously. For some of them that's harder than their actual research -- but there's not much choice. It's part and parcel of the work. A researcher has two jobs, coming up with valuable ideas, and then getting other people interested in those ideas.
I wish I had some specific advice for you but I don't really know how it works in physics. But I'm sure that if you're really smart enough to solve "five outstanding problems in physics" then you'll be able to figure it if you apply yourself. A good first step might be to go to some physics conferences, meet some people who practice in your field, and befriend them. Show a lot of interest in their research, let them talk about it, ask intelligent questions (it's flattering when people do that.) Don't mention your own research - be vague ("oh, I've been interested in this aspect of gravitation too, which is why I'm so curious about your ideas.") Try to find small ways to help out the people you meet. If you do this for a while, you will eventually make some friends, or at least professional contacts. Then you will at least be able to ask them to take a serious look at your ideas. ("Hey, I wonder if you could help me out with something. I've had some ideas about [unsolved problem] and I know that you're an expert in it - it's always your name that comes up when I ask around. I'd love to get your opinion. Is there a time that we could chat for 15 or 20 minutes?") At least, that's what I'd do if I didn't have any contacts in a field, and needed to make some.
I'm well aware of the issues noted in your first two big paragraphs. These are the typical reasons given in the informal research I mentioned. It boils down to this reasoning: A purported solution to a major outstanding problem in physics is not worth reading, simply because the odds of it being correct are too small.
It's rude to suggest that I'm whining. All I've done here is stated my opinion and reported my relevant experiences. The same as you have.
The quest for publication (or even one other person knowing the solutions to these major problems) was over for me some years ago. I'm only responding to the topic at hand here. I've long been satisfied with my ideas being lost to science. I only ever wanted publication to share the ideas, not to gain anything for myself. The physics community has made it clear that it's not interested in such solutions; I accept that. And that's not whining.
It's rude to suggest that I'm whining. All I've done here is stated my opinion and reported my relevant experiences. The same as you have.
The quest for publication (or even one other person knowing the solutions to these major problems) was over for me some years ago. I'm only responding to the topic at hand here. I've long been satisfied with my ideas being lost to science. I only ever wanted publication to share the ideas, not to gain anything for myself. The physics community has made it clear that it's not interested in such solutions; I accept that. And that's not whining.
You're right, saying that you were whining was out of line and I apologize for it. It is what I would say to my real-life friends to challenge them, but it is not appropriate with someone I don't know well and in any event it is not appropriate on Hacker News.
However, I hope you understand why I had that reaction. It is because you complained about how hard it is to get ideas like yours heard (and certainly "complaint" is a fair word for what you have written) rather than taking it as your responsibility to find a solution. I do take some offense at that, because I have some close friends that have worked very hard to become academically established so that they can find collaborators and get their ideas disseminated. You don't seem to value this work, or you consider it not worth your time. Rather than do it you are content to leave your work to "internet archaeologists" of a "more scientific" era.
Actually I agree with you that it's difficult to get revolutionary ideas heard (let alone funded), and that members of the scientific establishment sometimes go out of their way to stifle ideas that could threaten the theories on which they have built their careers. And today I think many professional academics would agree with you also. It's always been this way. What I admire is people who take this difficult terrain as a given and find a way to get to their destination anyway. These are the people that have driven human understanding forward. Of course it's hard, but it's an acknowledged part of the job description.
However, I hope you understand why I had that reaction. It is because you complained about how hard it is to get ideas like yours heard (and certainly "complaint" is a fair word for what you have written) rather than taking it as your responsibility to find a solution. I do take some offense at that, because I have some close friends that have worked very hard to become academically established so that they can find collaborators and get their ideas disseminated. You don't seem to value this work, or you consider it not worth your time. Rather than do it you are content to leave your work to "internet archaeologists" of a "more scientific" era.
Actually I agree with you that it's difficult to get revolutionary ideas heard (let alone funded), and that members of the scientific establishment sometimes go out of their way to stifle ideas that could threaten the theories on which they have built their careers. And today I think many professional academics would agree with you also. It's always been this way. What I admire is people who take this difficult terrain as a given and find a way to get to their destination anyway. These are the people that have driven human understanding forward. Of course it's hard, but it's an acknowledged part of the job description.
> What I admire is people who take this difficult terrain as a given and find a way to get to their destination anyway.
Same here. Alas, I'm not one of them. I'm not both a scientist and politician. Like a lot of folk on HN I have a software development background. I'm used to working in a field where ideas flow freely. I was shocked (in my naivete) to discover it's largely the opposite in science. If I had it to do over again, I wouldn't have bothered to write down my thoughts. I don't see much point in pushing something on an unreceptive community.
> Of course it's hard, but it's an acknowledged part of the job description.
"Complaint" was accurate, but "lament" is more so. It didn't used to be this way, not as hard as it is today. Scientists of yesteryear were more like hobbyists, like HNers. Now it's a cutthroat industry, with grant money and careers fiercely protected. I see that the advance of science, or physics at least, is slowed to a crawl as a result, even moving backwards in some cases.
As far as sharing ideas goes, I've redirected my energies to web development. It's way more rewarding!
Same here. Alas, I'm not one of them. I'm not both a scientist and politician. Like a lot of folk on HN I have a software development background. I'm used to working in a field where ideas flow freely. I was shocked (in my naivete) to discover it's largely the opposite in science. If I had it to do over again, I wouldn't have bothered to write down my thoughts. I don't see much point in pushing something on an unreceptive community.
> Of course it's hard, but it's an acknowledged part of the job description.
"Complaint" was accurate, but "lament" is more so. It didn't used to be this way, not as hard as it is today. Scientists of yesteryear were more like hobbyists, like HNers. Now it's a cutthroat industry, with grant money and careers fiercely protected. I see that the advance of science, or physics at least, is slowed to a crawl as a result, even moving backwards in some cases.
As far as sharing ideas goes, I've redirected my energies to web development. It's way more rewarding!
Do you accept that you might be wrong?
Of course I do.
Then instead of "I have solved", you should say "I have proposed solutions to."
I said the equivalent: "I have solved...or so I claim".
In fact, that (and your other parenthetical about how having worked on something for 10 years doesn't mean it's valuable) is why I took the time to reply to you above.
Its not uncommon in computer systems conferences to see rejects based on comments about the system not having practical application or being too non-practical. Oddly enough, it is not too uncommon to see papers getting one review rejecting the paper because it is not practical enough and another review (for the same paper) rejecting the paper because it is too practical :P
I've had two very frustrating reviews. On one paper, the first reviewer said it had too much theory and the second reviewer said it had too little theory. Both rejected it. On another grant proposal, a reviewer said in paragraph one, "This is impossible, it can't be done". In paragraph two, he said "...and I know of three companies already doing this."
> On one paper, the first reviewer said it had too much theory and the second reviewer said it had too little theory. Both rejected it.
I haven't read the paper, so it's possible that one or both of the reviewers were way off base. But it's also possible that both reviewers were "correct". Often, when you get conflicting advice like that it's a sign that you're stuck in a compromised middle position that isn't quite working for anybody. For the paper in question, the comments could probably could have been better stated as "You have some theory in the paper which is a little incomplete as it stands. To improve the paper, you could either expand upon it, or cut it, but probably not keep it as it is."
I haven't read the paper, so it's possible that one or both of the reviewers were way off base. But it's also possible that both reviewers were "correct". Often, when you get conflicting advice like that it's a sign that you're stuck in a compromised middle position that isn't quite working for anybody. For the paper in question, the comments could probably could have been better stated as "You have some theory in the paper which is a little incomplete as it stands. To improve the paper, you could either expand upon it, or cut it, but probably not keep it as it is."
On the topic of self-contradicting reviews, I once got one where the reviewer began by noting that "the paper is very difficult to read and almost impossible to understand" and proceeded to remark that "it contains very interesting ideas"...
I fail to see the contradiction. Being difficult to read and impossible to understand does mean it can't contain interesting ideas.
At the very least, (s)he would have had to understand the main ideas to deem them interesting (though perhaps the details were difficult to follow). So yes, you're right that it isn't technically a contradiction. The way it was written did come across to me as intelectually dishonest at the time, though.
I fail to see the contradiction. Begin difficult to read and impossible to understand does mean it can't contain interesting ideas.
The peer review system is riven with ego politics. I've submitted a paper before that got rejected by a committee member because I didn't work closely enough with member's research team. He was the one hold out among four other positive peer reviews. The paper ended up getting published in a subsequent conference where the committee members didn't have a stake in my area.
The usefulness of any given idea isn't usually reflected in peer reviewers' opinions, but whether it actually is a good idea, and sometimes that happens outside the peer review system. For example, I don't think even Larry and Sergei's PageRank algorithm was published in a peer-reviewed conference, and we all know what happened there.
Novelty, on the other hand, is probably a lot harder but even those ideas fall through the ego-drilled cracks of peer review.
The usefulness of any given idea isn't usually reflected in peer reviewers' opinions, but whether it actually is a good idea, and sometimes that happens outside the peer review system. For example, I don't think even Larry and Sergei's PageRank algorithm was published in a peer-reviewed conference, and we all know what happened there.
Novelty, on the other hand, is probably a lot harder but even those ideas fall through the ego-drilled cracks of peer review.
PageRank was an old algorithm, applied to a new corpus (the web, instead of research papers). Their contribution was in executing on the web.
It would have been funnier if the flaws in the reviewers arguments were a little more subtle, but this would have required the author to actually _read_ the papers he was criticizing. Even a bad reviewer that only skimmed half of the papers would have not made the errors in these reviews (e.g. saying things were unaddressed which were section headings in the original)...
I think you overstimate reviewers.
Not sure if reviewer was having a bad day or just lacks the understanding to comprehend ideas beyond what is the norm.
Also quite often professors delegate the review work to their students who may lack the understanding in broader perspective.
while it is a funny joke it does not really make any sense because no serious academic journal reviews papers with less than 3 reviewers. Most use more then 5 or 6 so one reviewer on a bad day really cant have much impact
Can't comment on other areas but in EECS my experience was 2 - 3 was the norm. (Never ever heard of 5 - 6 although I presume you'd get that many if you submitted a paper with "cold fusion" in the title.)
Stuff is becoming so technical these days that from those 2-3 reviewers you are sometimes lucky to get a single person with a real understanding of the subject matter and competent to vote on it. The whole peer review system is almost as broken as the patent system imho.
Stuff is becoming so technical these days that from those 2-3 reviewers you are sometimes lucky to get a single person with a real understanding of the subject matter and competent to vote on it. The whole peer review system is almost as broken as the patent system imho.
Scribd is still in business?
I wonder how many people are still a running Firefox 3 on win2000 or whatever to have a use for the scribd links instead of the original PDFs.
I wonder how many people are still a running Firefox 3 on win2000 or whatever to have a use for the scribd links instead of the original PDFs.
To the ones not eager to open a PDF: go ahead and read it, it's hilarious. Rejections of Dijkstra's, Codd's, Turing's, Shannon's, Hoare's, RSA's groundbreaking papers in computer science due to nit picking and short introductions.
For Shannon's paper:
The author claims that “semantic aspects of communication are irrelevant to the engineering problems,” which seems to indicate that his theory is suitable mostly for transmitting gibberish. Alas, people will not pay to have gibberish transmitted anywhere.
For Shannon's paper:
The author claims that “semantic aspects of communication are irrelevant to the engineering problems,” which seems to indicate that his theory is suitable mostly for transmitting gibberish. Alas, people will not pay to have gibberish transmitted anywhere.
Or even better:
At any point, there are sexy topics and unsexy ones: these days, television is sexy and color television is even sexier. Discrete channels with a finite number of symbols are good for telegraphy, but telegraphy is 100 years old, hardly a good research topic.
At any point, there are sexy topics and unsexy ones: these days, television is sexy and color television is even sexier. Discrete channels with a finite number of symbols are good for telegraphy, but telegraphy is 100 years old, hardly a good research topic.
To the ones eager to open a PDF, http://www.fang.ece.ufl.edu/reject.html (http://news.ycombinator.com/item?id=3493803)
Add to this Berners-Lee's rejection[1] from HyperText 1991 for his paper proposing the Web because it was a) too simple and [2], b) not really a very good Hypertext system [3].
[1] It was actually relegated to a poster. [2] http://www.w3.org/2007/04dc-rdu/w3c-stripes.html#(7) [3] http://eprints.soton.ac.uk/258701/1/youveGotHypertext11b.pdf
[1] It was actually relegated to a poster. [2] http://www.w3.org/2007/04dc-rdu/w3c-stripes.html#(7) [3] http://eprints.soton.ac.uk/258701/1/youveGotHypertext11b.pdf
Just in case it is not obvious: These rejection letters are fictional[1], created as collage of negative reviews Santini received. But it's hilarious to read, I agree.
Since the original site where Santini commented on it is gone, probably it's only a question of time until this document becomes internet "truth". Brave new world.
[1]http://bertrandmeyer.com/2009/08/14/rejection-letter-classic...
Since the original site where Santini commented on it is gone, probably it's only a question of time until this document becomes internet "truth". Brave new world.
[1]http://bertrandmeyer.com/2009/08/14/rejection-letter-classic...
Just in case it is not obvious: These rejection letters are fictional[1], created as collage of negative reviews Santini received.
Thanks for pointing that out. I hope everyone gives your comment the upvote it deserves. That this submission was just a made-up joke was not obvious from
1) the domain currently hosting the document, one of the first things I check on each submission,
2) the submission title (which is NOT the original article title, contrary to the Hacker News guidelines), which sounds much too serious for a title on a joke article,
or
3) the first few comments, which treated the article as though it reported historical facts. Good for you to let us know what the background is.
Thanks for pointing that out. I hope everyone gives your comment the upvote it deserves. That this submission was just a made-up joke was not obvious from
1) the domain currently hosting the document, one of the first things I check on each submission,
2) the submission title (which is NOT the original article title, contrary to the Hacker News guidelines), which sounds much too serious for a title on a joke article,
or
3) the first few comments, which treated the article as though it reported historical facts. Good for you to let us know what the background is.
Yes, and it also lacks something more important - the original versions of the papers that were rejected!
While I certainly have my quibbles with my peer reviewers from time to time, in every case that I've had a paper reviewed, whether reject or accept the final paper that ended up making publication was a far, far better version than the original. For all we know, the original versions of those papers were poorly-motivated, typo-filled messes, potentially even with grievous errors that were later corrected.
That's certainly been true of some papers that I've provided negative reviews for and have later seen in another form as wonderfully polished gem.
While I certainly have my quibbles with my peer reviewers from time to time, in every case that I've had a paper reviewed, whether reject or accept the final paper that ended up making publication was a far, far better version than the original. For all we know, the original versions of those papers were poorly-motivated, typo-filled messes, potentially even with grievous errors that were later corrected.
That's certainly been true of some papers that I've provided negative reviews for and have later seen in another form as wonderfully polished gem.
Thank you.
I particularly like this, regarding RSA :
Finally, there is the question of the application. Electronic mail on the Arpanet is indeed a nice gizmo, but it is unlikely it will ever be diffused outside academic circles and public laboratories—environments in which the need to maintain confidentiality is scarcely pressing.
Finally, there is the question of the application. Electronic mail on the Arpanet is indeed a nice gizmo, but it is unlikely it will ever be diffused outside academic circles and public laboratories—environments in which the need to maintain confidentiality is scarcely pressing.
My favorites:
“An Axiomatic Basis for Computer Programming.” I am not sure I under- stand this article. It claims to be about programming, but it doesn’t contain a single line of code.
"On Computable Numbers with an Application to the Entsheidungs-problem." [...] Either these numbers are too big to be represented in the machine[...] or they are not, in which case a machine that can't compute them is simply broken!
“An Axiomatic Basis for Computer Programming.” I am not sure I under- stand this article. It claims to be about programming, but it doesn’t contain a single line of code.
"On Computable Numbers with an Application to the Entsheidungs-problem." [...] Either these numbers are too big to be represented in the machine[...] or they are not, in which case a machine that can't compute them is simply broken!