Is novel research worth doing? Evidence from peer review(pnas.org)
pnas.org
Is novel research worth doing? Evidence from peer review
https://www.pnas.org/doi/10.1073/pnas.2118046119
53 comments
Bullshit. Journals select for novelty, this is true. But this is novelty in a trifling fashion, like teaching a dog to roll over. The entire system, at least in the UK is set up to actively discourage risky research entirely. Doing something truly new is discouraged unless you are truly a senior or leader. By then, you've probably lost most of your imagination anyway.
My anecdata from the US agrees. The best part is when funding agencies require proposal reviewers to actually send their critiques back on research proposals. I've read through many reviewers replies to proposals designed for specifically set risky/novel research funding and the responses are sometimes so conservatively comical without a truly valid argument you can't help but laugh.
I've also worked at research organizations specifically touted as being highly independent with freedom to explore novel paths and the story is basically the same. Ultimately, someone somewhere holds the purse strings and conservatism kicks in. You at minimum need to spin your research to fit popular trend keyword language or make it apply to these areas.
Financial structures ultimately dictate the lack of support for novelty. Basically, novel research has freedom only when it's independently funded, which limits the scale of most novel research. No one wants to take risk, they want to market taking high risk while taking low risk and selling mediocrity. I think the fundamental underlying issue we have across multiple societies is that we no longer proportionately reward risk anymore and that's why people seek low risk everywhere. Taking high risk often has limited opportunity for reward, even if you are successful, so why bother? You'll net more success taking low risks and only fools take high risk.
I've also worked at research organizations specifically touted as being highly independent with freedom to explore novel paths and the story is basically the same. Ultimately, someone somewhere holds the purse strings and conservatism kicks in. You at minimum need to spin your research to fit popular trend keyword language or make it apply to these areas.
Financial structures ultimately dictate the lack of support for novelty. Basically, novel research has freedom only when it's independently funded, which limits the scale of most novel research. No one wants to take risk, they want to market taking high risk while taking low risk and selling mediocrity. I think the fundamental underlying issue we have across multiple societies is that we no longer proportionately reward risk anymore and that's why people seek low risk everywhere. Taking high risk often has limited opportunity for reward, even if you are successful, so why bother? You'll net more success taking low risks and only fools take high risk.
I've thought for a long time that this is why you see so much innovation during wartime. War overrides conservatism and gets risky things funded.
The same can apply to war-like scenarios. A lot of novel biological research got funded during COVID. We should expect to see a burst in advancement in biomedical technology as it bears fruit over the next 5-10 years.
The core problem is probably that conservatism makes the most sense at the individual level for most people. Even if you had a casino whose net payout was positive, it still may not make sense individually to gamble. If individual rewards are sporadic and concentrated on a power law distribution then statistically most players will lose.
An obvious solution is a less power-law type distribution, which tends toward some kind of socialism, but that has its own problems. Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome. Trying to suppress it doesn't work; it's like trying to totally suppress sexuality. Extreme socialist societies (e.g. Soviet Communism) often evolve into totalitarian mafia states because if there is no positive outlet for those urges they drive people into crime. Eventually you get a meritocracy concentrated in crime and the mob takes over everything.
The same can apply to war-like scenarios. A lot of novel biological research got funded during COVID. We should expect to see a burst in advancement in biomedical technology as it bears fruit over the next 5-10 years.
The core problem is probably that conservatism makes the most sense at the individual level for most people. Even if you had a casino whose net payout was positive, it still may not make sense individually to gamble. If individual rewards are sporadic and concentrated on a power law distribution then statistically most players will lose.
An obvious solution is a less power-law type distribution, which tends toward some kind of socialism, but that has its own problems. Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome. Trying to suppress it doesn't work; it's like trying to totally suppress sexuality. Extreme socialist societies (e.g. Soviet Communism) often evolve into totalitarian mafia states because if there is no positive outlet for those urges they drive people into crime. Eventually you get a meritocracy concentrated in crime and the mob takes over everything.
"Nobody's figured out yet how to provide a productive channel in a socialist system for humans' totally normal drives to compete and maximize individual outcome."
Nobody has figured out how to maximize individual outcome, or provide a channel for such pursuit. Period. Socialism has nothing to do with it. People are crap at managing more than a few dozen other people, even for the best managers who have ever lived. And the systems we have in place since agriculture took over are trying to manage many, many more people.
Most people in more developed countries live in partially socialist systems. This doesn't prevent them from getting private sector jobs, creating companies, and otherwise trying to pursue the maximization of their individual outcomes.
Nobody has figured out how to maximize individual outcome, or provide a channel for such pursuit. Period. Socialism has nothing to do with it. People are crap at managing more than a few dozen other people, even for the best managers who have ever lived. And the systems we have in place since agriculture took over are trying to manage many, many more people.
Most people in more developed countries live in partially socialist systems. This doesn't prevent them from getting private sector jobs, creating companies, and otherwise trying to pursue the maximization of their individual outcomes.
Or fake crime. Solving real crime is too hard. Why not fake it.
I have a hunch that this conservatism problem is why ADHD genes / traits remain so prevalent. The percentage of populations with it is around 5% and remarkably consistent across drastically different cultures.
The reason is that those with ADHD are often physiologically incapable of not pursuing an interest. Often those interest lie in novel areas. It’s unfortunate for those with the traits, but a net benefit for society.
So there’s a game theory equilibrium wherein successful societies will retain a certain percentage of those genes or become too conservative.
The reason is that those with ADHD are often physiologically incapable of not pursuing an interest. Often those interest lie in novel areas. It’s unfortunate for those with the traits, but a net benefit for society.
So there’s a game theory equilibrium wherein successful societies will retain a certain percentage of those genes or become too conservative.
Traits that are not beneficial to the individual will die out, the kind of group selection this process would require is implausible at best.
I would assume that the kind of personality your are referring to is adaptive for the individual as well, especially the mild variants.
I would assume that the kind of personality your are referring to is adaptive for the individual as well, especially the mild variants.
> Traits that are not beneficial to the individual will die out
Not necessarily, examples from several species where males get eaten after mating would seem to counter that. In some the males try to escape if possible to mate again.
> the kind of group selection this process would require is implausible at best.
There can certainly be group theory optimums that are less beneficial for individuals but beneficial enough to the whole group that traits bad for an individual will persist. I’d say bees dying after stinging an intruder would be an example. Well or even the hive principle where only the queen reproduces.
Still I would agree that the mild variations of adhd traits can be beneficial to some individuals. At least enough to largely eliminate the traits despite being largely maladaptive in non-nomadic societies given that we’re not hive creatures. One example I’ve read was a paper that showed increased offspring in individuals with adhd traits in nomadic tribes in Africa, with decreased reproductive rates in adjacent non-nomadic tribes.
Our modern society is odd in that it’s not like either traditional society. ADHD traits can be very beneficial for individuals in technology careers requiring creativity while simultaneously being painful due to society being largely optimized for non-adhd neurotypes.
Not necessarily, examples from several species where males get eaten after mating would seem to counter that. In some the males try to escape if possible to mate again.
> the kind of group selection this process would require is implausible at best.
There can certainly be group theory optimums that are less beneficial for individuals but beneficial enough to the whole group that traits bad for an individual will persist. I’d say bees dying after stinging an intruder would be an example. Well or even the hive principle where only the queen reproduces.
Still I would agree that the mild variations of adhd traits can be beneficial to some individuals. At least enough to largely eliminate the traits despite being largely maladaptive in non-nomadic societies given that we’re not hive creatures. One example I’ve read was a paper that showed increased offspring in individuals with adhd traits in nomadic tribes in Africa, with decreased reproductive rates in adjacent non-nomadic tribes.
Our modern society is odd in that it’s not like either traditional society. ADHD traits can be very beneficial for individuals in technology careers requiring creativity while simultaneously being painful due to society being largely optimized for non-adhd neurotypes.
That was my conclusion after publishing a paper during my BSc. Academia is full of petty politics. Professors that gatekeep whole fields on a "like"-basis, reviewers arguing with "not enough experiments done" and questioning uniqueness and novelty in ways that just don't make any sense (to me). It is a very bureaucratic process that cannot possibly be good for innovation and progress. Then you get incentives of fields that don't make any progress to keep the hype train going so their jobs don't become meaningless over night. It just keeps going. So, I obviously declined to do a PhD and went into industry. At the bare minimum, playing politics in industry leads to vastly better (pay) outcomes than in academia.
Dog whistles in academic reviewing; phrases that can be valid but are typically being used in an invalid way:
- Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper.
- Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition.
- Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources?
I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.
- Not novel: Novelty is ill-defined. Most journals and conferences have guidelines on this that come down to "will someone find it useful" which should make "not novel" a rare review, not the most common. Additionally, what's obvious post hoc is not obvious a priori (I actually saw an AC override reviewers because of this!). If something is well written it also comes across as easy and simple. Lack of novelty could be a true lack of novelty or just an indication of a well written paper.
- Incremental: All research is incremental. Research can be too incremental (not enough work done) but this is never expanded upon in those types of reviews. This kind of statement is meaningless in isolation. Incrementalism will always exist with a publish or perish paradigm. One cannot publish breakthroughs year over year. Those take significant periods of time or a lucky break. Neither is a common occurrence, by definition.
- Not enough experiments: "Money is all you need" has become a popular phrase. There may not be enough experiments and this can be true but perfection is the enemy of good. Especially in research, where the experiment space is non-exhaustive. Can a reviewer be satisfied in a finite amount of time and with finite resources?
I've started to change my mind about where blame falls for all this. It starts with the reviewers, but if this becomes the norm then the system is at fault. It has not removed the rotten apples and thus let the barrel spoil. All while we've had decades of discussion about this happening. I think a lot falls on the Area Chairs and Metareviewers, as they should be preventing these types of reviews to pass. But the whole incentive structure of publishing is wrong. There is high pressure to reject and zero pressure to let papers through. This especially hurts our junior researchers (grad students) as it becomes a lottery process for determining if they can graduate. After all, we can't have any wizards without any noobs.
It is absolutely a systemic problem. A problem that will not be changed or solved within our lifetimes, that's for sure.
> A problem that will not be changed or solved within our lifetimes
Honestly, I don't like opinions like this. That's just passing off the work that needs to be done to fix these problems. It wasn't even a hundred years ago when people published more in the open and the publish or perish paradigm didn't exist. The latter is a relatively tool, and collapsed because Goodhart's Law.
We can make the changes, but not with defeatist attitudes.
Honestly, I don't like opinions like this. That's just passing off the work that needs to be done to fix these problems. It wasn't even a hundred years ago when people published more in the open and the publish or perish paradigm didn't exist. The latter is a relatively tool, and collapsed because Goodhart's Law.
We can make the changes, but not with defeatist attitudes.
Fair point. I was just thinking that one sensible change could be to publish everything without significant constraints on open platforms. Then, in some way, the community and some moderate moderation would basically decide which papers and researchers get "famous". In the best case, that would result in more true "breakthroughs", but reflecting about how social media works (incentives to constantly push narratives, hype and simple "truths" that explain complex world issues) at the moment, that seems a bit too optimistic. That's why I peddled back to "not solvable within our lifetimes". Who knows. I'd like to do research, but if I go into academia I'd be immensely constrained and would have no freedom what to work on. The only viable option is to become sufficiently wealthy to fund myself and my ideas. Would be nice, even if a tiny bit unrealistic.
There's a double edged sword here. Currently CVPR has a social media ban that is extremely strict. I've definitely been seeing Twitter act differently than it normally does around this so we'll see if it works. But why this matters here is that there's an incredibly strong correlation between the popularity of the lab, the amount of eyes that can read the preprint (with authors names), and acceptance at top conferences (i.e. CVPR). I often say that double blind only exists for small fries. But even still, big labs will get more likes from twitter accounts that post arxiv papers. There are plenty of means to bypass this social media ban.
But even from this experiment we can see that there are people actively trying the fix the situation. Likely naive, but something is better than nothing. I wish there would be more push for ACs and Meta Reviewers having the highest standards, but this is where we are.
I am totally for us abandoning journals and conferences and just publishing to arxiv and open review. This is exactly what I would do if I was independently wealthy and could research without the constraints that I am in. But we also need to recognize the issues with this and that popularity and outreach ability greatly affect perceptions of quality of the paper. That this is not meritocratic.
But even from this experiment we can see that there are people actively trying the fix the situation. Likely naive, but something is better than nothing. I wish there would be more push for ACs and Meta Reviewers having the highest standards, but this is where we are.
I am totally for us abandoning journals and conferences and just publishing to arxiv and open review. This is exactly what I would do if I was independently wealthy and could research without the constraints that I am in. But we also need to recognize the issues with this and that popularity and outreach ability greatly affect perceptions of quality of the paper. That this is not meritocratic.
As a clinical researcher, I wholeheartedly agree. Even worse, meta analysis is comparatively so well cited (even shitty ones) that doing anything else, such as original research, will be suboptimal since the reduced time investment allows you to publish nonstop. Truly, meta analysis has become a blood-ducking parasite of medical science.
I know many young researchers who've specialized in meta analysis for that reason.
I know many young researchers who've specialized in meta analysis for that reason.
Isn't there tremendous real value in meta-analysis? The problem seems to lie in how credit is allocated (via citations).
I'd argue the problem is a combination of citations and short-termism. We don't give researchers the space to do something novel, because institutions demand results in the short term. If you give someone a tangible, quantifiable demand A and a nebulous, low-priority demand B, they will spend nearly 100% of their time chasing A. Truly novel results can take months or years of hard focus; every grant proposal, teaching duty, and routine publication is a distraction.
> Isn't there tremendous real value in meta-analysis?
There's also tremendous value in doing the dishes and mopping the floor. Without it we'd get diseases and live in filth. And when we do this, we should be respected and appreciated. But - not as brilliant trail-blazers.
There's also tremendous value in doing the dishes and mopping the floor. Without it we'd get diseases and live in filth. And when we do this, we should be respected and appreciated. But - not as brilliant trail-blazers.
>By then, you've probably lost most of your imagination anyway.
I've recently left academia after ~10 years to work on more blue-sky-stuff in industry, and this rings so true. It took me a few months to unlearn academia's focus on the minimal project we can do to get the next paper out (still unlearning!).
It's hard to get in, it's harder to get out.
I've recently left academia after ~10 years to work on more blue-sky-stuff in industry, and this rings so true. It took me a few months to unlearn academia's focus on the minimal project we can do to get the next paper out (still unlearning!).
It's hard to get in, it's harder to get out.
I worked as a permatemp for almost 7 years in industry and didn't have the ability to order my own reagents. It took 3 or 4 years of working in a DOE lab to break that habit and start ordering my own reagents to innovate with.
Ingrained work habits are hard to break.
Ingrained work habits are hard to break.
Certain fields are a lot easier to do novel work in. For a grad student in computer science/computational biology, you practically don't need any grant funding at all. You can download public data, work on the university hpc for basically free, publish whatever the hell you want, and pay your stipend with a TAship. Others, where you have to regularly buy reagents or maintain an animal model, you pretty much need grant support to buy these things.
I heard established profs talk about how they ask for funding for something already done or almost successfully done. Then they do something new with the money.
[citation needed]
My background is in humanities but the dynamics are the same: Chasing the goal of objectivity, easily measurable datapoints like number of publications and number of citations are used for quantifying impact of a researcher. This incentivizes researching/publishing low hanging fruit, because actually working on deep or unusual approaches means sinking more time into the endevour while risking less or no publishable results. For people trying to earn a living in academic institutions this would be stupid.
I can confirm the GP matches my experience in academia (PhD, published a paper based on my dissertation, left academia shortly after).
There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysis); etc. etc. This is how you rack up publications for your tenure review. It's also how you rack up publications to get funding.
Once you have tenure you have a bit more freedom, but you still need to worry about funding (generally). And by that time you've been in the habit of grabbing the lowest-hanging fruit for 1-1.5 decades, assuming you were ~well trained~ in graduate school. After so long reinforcing that habit it becomes hard and scary to build a ladder and go after the better, higher fruit.
There is immense pressure for early career researchers to work on low-hanging fruit that will give easy guaranteed publications: make a minor change to previous work and write a paper about it; write a paper about what other people wrote papers about (either a review or a meta-analysis); etc. etc. This is how you rack up publications for your tenure review. It's also how you rack up publications to get funding.
Once you have tenure you have a bit more freedom, but you still need to worry about funding (generally). And by that time you've been in the habit of grabbing the lowest-hanging fruit for 1-1.5 decades, assuming you were ~well trained~ in graduate school. After so long reinforcing that habit it becomes hard and scary to build a ladder and go after the better, higher fruit.
Habit is not the problem. Once you get tenure, a good part of your job is to ensure that the younger generation in your lab/uni publishes well, i.e encourage them to pursue the next low-hanging fruit... So no, you do not in fact have more freedom.
That's a fair point. I jumped ship before I got that far, obviously. From my perspective as a student it looked a lot like habit on the part of the senior/tenured people in the department, but I should not have been so quick to assert that.
I suggest the author cite the following works which I believe are related to this topic:
list of own publications
list of own publications
I think often of this jab by Rodney Brooks on my home field of human-robot interaction [1]:
"A group of people who have never used a computer are recruited for an experiment, using variations of Unix commands to manipulate files from a command line interface. The users have never heard of computer files before and they don’t in any sense have the utility of files internalized.
The experiment is to determine whether something the experimenters have called utility versus expressiveness is more highly valued. One set of users is taught commands like “rm,” “mkdir,” and “cat,” etc., while another is taught “delete,” “new_directory,” and “type_out,” etc. They are asked to do some housekeeping tasks on a file system. Their speed and accuracy is measured, and they fill in a questionnaire about how hard and how intuitive the task was. The paper itself about these experiments touts “statistically significant results,” with lots of p-values, etc.
No HCI academics bother to invent windows, icons, a mouse, and so on. Such work is viewed as too informal and unscientific."
I just finished my PhD and these lines were in my head since they were published. Between an advisor who would only greenlight plans if they immediately led to a paper and an embarrassing amount of impractical "safe" research that only makes for good tweets or headlines, I've given up hope on anything meaningful coming out of the Ivory Towers. The robot revolution will not be peer reviewed.
[1] https://dl.acm.org/doi/10.1145/3209540
"A group of people who have never used a computer are recruited for an experiment, using variations of Unix commands to manipulate files from a command line interface. The users have never heard of computer files before and they don’t in any sense have the utility of files internalized.
The experiment is to determine whether something the experimenters have called utility versus expressiveness is more highly valued. One set of users is taught commands like “rm,” “mkdir,” and “cat,” etc., while another is taught “delete,” “new_directory,” and “type_out,” etc. They are asked to do some housekeeping tasks on a file system. Their speed and accuracy is measured, and they fill in a questionnaire about how hard and how intuitive the task was. The paper itself about these experiments touts “statistically significant results,” with lots of p-values, etc.
No HCI academics bother to invent windows, icons, a mouse, and so on. Such work is viewed as too informal and unscientific."
I just finished my PhD and these lines were in my head since they were published. Between an advisor who would only greenlight plans if they immediately led to a paper and an embarrassing amount of impractical "safe" research that only makes for good tweets or headlines, I've given up hope on anything meaningful coming out of the Ivory Towers. The robot revolution will not be peer reviewed.
[1] https://dl.acm.org/doi/10.1145/3209540
There's an interesting tension in academia between the need to publish regularly, the fact that deep research often needs longer work periods before achieving impactful results, and the fact that sometimes quantity of output produces better net results.
I wonder if this could be alleviated by incentivizing the publishing of papers on work with no statistically significant results, as has been discussed here and elsewhere. That might increase quantity of output, while building a useful database of what not to spend time on.
I wonder if this could be alleviated by incentivizing the publishing of papers on work with no statistically significant results, as has been discussed here and elsewhere. That might increase quantity of output, while building a useful database of what not to spend time on.
I would like to see peer-to-peer discussion incentivized. To an extent it is with poster sessions counting for external value. But I think this could be improved by counting more informal discussions and personal communications.
As a youngster I got the idea that truly novel experimentation should be conducted alongside the most predictable incremental progress, in a unified way so that an overall structural approach which allows this would be an ideal thing to build.
Even if it takes decades.
Which is how long research labs take anyway.
Maybe this means something. Talk about HCI. Coming from a guy back in 1982 who thought pointing & clicking was pretty cool. Almost a decade before the mouse was widely recognized, the trackball/firing-button was in use industrially to select graphic items. At one of the control rooms of the most automated marine chemical installations at the time, they had consoles that looked like NASA Mission Control during the moon mission. Tied into their mainframes. And wired into their tank farms, pipelines, and ship docks. Well a somewhat humble oil company can have more money to spend on this kind of thing than NASA sometimes.
The oversize CRTs would have the sharp monochrome graphics showing little pictures of the available tanks on the left and available docks on the right. In between, connecting them, was the pipeline maze. Scattered across the pipelines were the little markers representing the control valves, which physically controlled where the product was going to go. They were like a little blank spot in the pipeline, just big enough so a cursor could go there and select it with a click so it would no longer be blank. Using a big console-mounted trackball like would appear years later on the Atari Millipede arcade game. And that would end up opening the valve so the product would flow through that segment. Until they had an unbroken line from tank to dock.
When your ship was coming in, you would co-ordinate the exact dock and tanks that were going to be involved, and after review the graphic layout would be executed physically in the field before measurement & testing could be directed for that vessel.
So I went home to my Atari computer, unplugged the Centipede cartridge and popped in the BASIC. Kept the home trackball with the regulation cueball plugged into a joystick port. Mocked up the console GUI in color since this is what really set new things like home Ataris apart from earlier plain-text machines, so besides basic graphics I got accustomed to pointing & clicking at home using my own code. No Mouse. Statically like I had seen. Really just one more step away from people intuitively pointing & clicking for targeting in Missile Command, which was the other main Atari game using the trackball.
I read a number of Atari books and experimented quite a bit and I could see that the truly fast games were really a product of deeper experience in things that I did not possess; immersion in the emerging microprocessor ecosystem, and very long hours of progress, which I could not compare to as a hobbyist. I wanted to make an accurate navigational Google-map type framework where the graphics would be first-person 3-D as you drove along realistic freeways. You would certainly want a racing mode, so that would be a bit of GTA style landmark navigation either way. Obvious idea, but that was really too much for me and my equipment.
Instead I did a much simpler slow-action logic game called Master Mind where you deduce the color pattern that the opposing player (or computer) has secretly encoded. You simply dragged & dropped the colors into position, and the computer answered.
So I was dragging & dropping quite a few years before most of the world had a mouse to point & click with in DOS. Heck, before most of them even had DOS. But it was really just one day's HCI work. What else are you going to do with the equipment? You're going to have to experiment for yourself. It's good not to be hampered by all the traditional academic obstacles.
I think nothing beats returns on inventions which have already been accomplished compared to research yet to be done, but the wisest investments are in the balanced performance of both. They go together so well.
Even if it takes decades.
Which is how long research labs take anyway.
Maybe this means something. Talk about HCI. Coming from a guy back in 1982 who thought pointing & clicking was pretty cool. Almost a decade before the mouse was widely recognized, the trackball/firing-button was in use industrially to select graphic items. At one of the control rooms of the most automated marine chemical installations at the time, they had consoles that looked like NASA Mission Control during the moon mission. Tied into their mainframes. And wired into their tank farms, pipelines, and ship docks. Well a somewhat humble oil company can have more money to spend on this kind of thing than NASA sometimes.
The oversize CRTs would have the sharp monochrome graphics showing little pictures of the available tanks on the left and available docks on the right. In between, connecting them, was the pipeline maze. Scattered across the pipelines were the little markers representing the control valves, which physically controlled where the product was going to go. They were like a little blank spot in the pipeline, just big enough so a cursor could go there and select it with a click so it would no longer be blank. Using a big console-mounted trackball like would appear years later on the Atari Millipede arcade game. And that would end up opening the valve so the product would flow through that segment. Until they had an unbroken line from tank to dock.
When your ship was coming in, you would co-ordinate the exact dock and tanks that were going to be involved, and after review the graphic layout would be executed physically in the field before measurement & testing could be directed for that vessel.
So I went home to my Atari computer, unplugged the Centipede cartridge and popped in the BASIC. Kept the home trackball with the regulation cueball plugged into a joystick port. Mocked up the console GUI in color since this is what really set new things like home Ataris apart from earlier plain-text machines, so besides basic graphics I got accustomed to pointing & clicking at home using my own code. No Mouse. Statically like I had seen. Really just one more step away from people intuitively pointing & clicking for targeting in Missile Command, which was the other main Atari game using the trackball.
I read a number of Atari books and experimented quite a bit and I could see that the truly fast games were really a product of deeper experience in things that I did not possess; immersion in the emerging microprocessor ecosystem, and very long hours of progress, which I could not compare to as a hobbyist. I wanted to make an accurate navigational Google-map type framework where the graphics would be first-person 3-D as you drove along realistic freeways. You would certainly want a racing mode, so that would be a bit of GTA style landmark navigation either way. Obvious idea, but that was really too much for me and my equipment.
Instead I did a much simpler slow-action logic game called Master Mind where you deduce the color pattern that the opposing player (or computer) has secretly encoded. You simply dragged & dropped the colors into position, and the computer answered.
So I was dragging & dropping quite a few years before most of the world had a mouse to point & click with in DOS. Heck, before most of them even had DOS. But it was really just one day's HCI work. What else are you going to do with the equipment? You're going to have to experiment for yourself. It's good not to be hampered by all the traditional academic obstacles.
I think nothing beats returns on inventions which have already been accomplished compared to research yet to be done, but the wisest investments are in the balanced performance of both. They go together so well.
Thanks for the long response and I agree that experimentation like yours needs to have more venues for exposure, which is hard to do in our contemporary content treadmill.
"The enemy of art is the absence of limitations." Seeing research budgets spent on equipment - that either never gets used or is poorly maintained - just to save a spot on a given funding bracket leaves a bad taste in my mouth.
"The enemy of art is the absence of limitations." Seeing research budgets spent on equipment - that either never gets used or is poorly maintained - just to save a spot on a given funding bracket leaves a bad taste in my mouth.
I know what you mean.
I built my own laboratory from surplus research equipment where I only had certain types of pieces but I ended up with stronger resources than any one original owner.
Was able to use more features over more years doing invoiceable routine work the research way, and between vessels explore applications that were not invoiceable. That's where the real progress is, and it's possible to make it eventually pay off. Got so much more out of the equipment for many more years than the original companies. And I made money making progress on the instruments while it cost them money every inch of the way.
So what if I could have gotten a whole lot more out of millions of dollars worth of new gear, I was just making the most of the limited stuff I could get affordably instead. I was also a lot more of a loose cannon when it didn't really cost me a million dollars either.
The large petrochemical labs, with all their PhDs and virtually unlimited budgets compared to mine, could not do what I could do with their old equipment. Even after they had gotten a head start on newer models they had acquired before declaring their older ones surplus. And it took even more time before I had access to the surplus.
Even though these were industrial labs they were hampered by a carryover from the academic laboratory background.
And I was welcome to the unexplored possibilities overflowing out the basement doors with the truckloads of "obsolete" electronics. So I can't complain.
I built my own laboratory from surplus research equipment where I only had certain types of pieces but I ended up with stronger resources than any one original owner.
Was able to use more features over more years doing invoiceable routine work the research way, and between vessels explore applications that were not invoiceable. That's where the real progress is, and it's possible to make it eventually pay off. Got so much more out of the equipment for many more years than the original companies. And I made money making progress on the instruments while it cost them money every inch of the way.
So what if I could have gotten a whole lot more out of millions of dollars worth of new gear, I was just making the most of the limited stuff I could get affordably instead. I was also a lot more of a loose cannon when it didn't really cost me a million dollars either.
The large petrochemical labs, with all their PhDs and virtually unlimited budgets compared to mine, could not do what I could do with their old equipment. Even after they had gotten a head start on newer models they had acquired before declaring their older ones surplus. And it took even more time before I had access to the surplus.
Even though these were industrial labs they were hampered by a carryover from the academic laboratory background.
And I was welcome to the unexplored possibilities overflowing out the basement doors with the truckloads of "obsolete" electronics. So I can't complain.
> We find that higher novelty is consistently associated with higher acceptance; submissions in the top novelty quintile are 6.5 percentage points more likely than bottom quintile ones to get accepted.
Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to look at the whole process.
Here’s a thought experiment to perhaps better explain what I mean: Let’s say you’re playing roulette, but with slightly modified rules. If you bet on a color (red or black) you have a chance to double your money as usual. But if you bet on a single number and get it right you only make 6.5% more, or 2.13 your money. Would you bet on a color (i.e. be conservative) or a single number (i.e. try to be “novel”)?
Ok, so “novel” research is 6.5% more likely to be accepted if submitted, but how likely is it to even be submitted? I would argue that doing “novel” research is much riskier at the earlier stages, and that you need to look at the whole process.
Here’s a thought experiment to perhaps better explain what I mean: Let’s say you’re playing roulette, but with slightly modified rules. If you bet on a color (red or black) you have a chance to double your money as usual. But if you bet on a single number and get it right you only make 6.5% more, or 2.13 your money. Would you bet on a color (i.e. be conservative) or a single number (i.e. try to be “novel”)?
I've heard some researcher saying that a way to manage this is to dedicate a part of your time to risky research, and the rest to less risky research.
So you still publish, but also get to explore "interesting" stuff (that also needs to be researched). You are covered by your less risky activity.
So, a single number 20%-30% of the time, and a color 70%-80% of the time. And your risky activity could become your comfortable zone if it takes.
Getting published is not the only worthy thing. The content itself and how you enjoy the thing too.
So you still publish, but also get to explore "interesting" stuff (that also needs to be researched). You are covered by your less risky activity.
So, a single number 20%-30% of the time, and a color 70%-80% of the time. And your risky activity could become your comfortable zone if it takes.
Getting published is not the only worthy thing. The content itself and how you enjoy the thing too.
Don't forget time for teaching, avoiding students (tried to say 'advising' but autocomplete had a better word choice), and grant applications...
I think it would be much more interesting to see a study between novel research produced under different education models. E.g. government funded vs industry funded vs user pays.
> we measure the novelty and conventionality of manuscripts by the atypicality of combinations of journals in their reference lists
Kind of an odd proxy for novelty of the research, but ok I guess. Maybe they’re going for cross-disciplinary work?
Kind of an odd proxy for novelty of the research, but ok I guess. Maybe they’re going for cross-disciplinary work?
Switch to another metric, make a different conclusion. This is modern science where you can prove whatever you want.
You're right about the metric, but this paper falls into the category of information analysis, not science. As you say, the results for this kind of work usually depend on the metric, and since those metrics are at the choice of authors, the result are open to debate. I'm not saying that the results are uninteresting, only that it is risky to make strong conclusions based on them.
For example, I work in a multidisciplinary field. Once I counted the citations per word of text in papers published in the separate disciplines. I found that biology-oriented research had a cite/word rate that was about 5 times that of physics-oriented research. I don't think that says anything really significant about the two fields, but it is something worth knowing, when judging tenure files.
For example, I work in a multidisciplinary field. Once I counted the citations per word of text in papers published in the separate disciplines. I found that biology-oriented research had a cite/word rate that was about 5 times that of physics-oriented research. I don't think that says anything really significant about the two fields, but it is something worth knowing, when judging tenure files.
Yeah, this was my first reaction as well. It's incredibly hard to tell how much signal is in a metric like this. I can imagine two papers or two grant proposal with significantly different degrees of novelty that have the exact same references.
In the end, trying to find a perfect system for new and original research (I like new and original much better than "novel") is not possible, because new and original research does not follow any rules. All you can do is to build enough slack into the system so that people can do whatever the hell they are interested in, in addition to their measured duties. But then how do you admit people into this system? If it is based on objective measurements, you will mostly get people who will do great at their measured duties, but will mostly not be able to use the slack to produce new and original research; instead they will turn the provided slack into objectively measurable activities. Bar unlimited resources, I don't see a good way out of this conundrum. Do you?
I believe the culprit is the assumption the grants/tenure-track systems make about applicants/assistant professors. That assumption is negative, as if new hires will try _not to do research_. It's also about the number of grad school offerings, it just seems so huge at the moment, which again forces introducing such metrics on what is considered "success" in academia.
Of course researchers will want to do research. The question is, how do you select for those whose research will ever amount to something truly interesting, and avoid giving resources to those whose research will not?
You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so criteria that worked in the past might not work now. Also, if you learn those criteria once, and then fix them, people will just game them.
You can try looking at the past, and derive criteria for it from that, via machine learning for example, but I would be hesitant to leave something like that up to a machine. Also times change, so criteria that worked in the past might not work now. Also, if you learn those criteria once, and then fix them, people will just game them.
>The question is, how do you select for those whose research will ever amount to something truly interesting
You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.
You can't, as "truly interesting" is context dependent, changes over time and something that everyone deems as futile may become interesting - that's the point of research. You just increase the bar of entry to get people who work very hard and leave it to them to decide.
Working very hard is not well-defined in this context. What does it even mean? You can work hard when you have a clear goal, let's say put those 10 barrels onto that truck over there. Or, let's write a new web browser within 2 years. Putting out 20 papers per year can also be considered working very hard.
You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly interesting research. HOW they do this is not up to you.
And there is a difference between truly interesting research, and just busy work research. It's not that easy to identify truly interesting research without the benefit of hindsight. It is somewhat easier to identify busy work research for an objective subject matter expert (but of course this is not 100% either, and personal preferences can definitely cloud the experts judgement).
You don't want people who work very hard. You want people who will EVENTUALLY put out new, original, and truly interesting research. HOW they do this is not up to you.
And there is a difference between truly interesting research, and just busy work research. It's not that easy to identify truly interesting research without the benefit of hindsight. It is somewhat easier to identify busy work research for an objective subject matter expert (but of course this is not 100% either, and personal preferences can definitely cloud the experts judgement).
You need people who got excellent grades in their undergrad program, who have somehow demonstrated that they like the field they’re going into academia for (as in clubs, extracurriculars, and whatnot), and who peers recommend as being likely to do novel research. The last one requires, gasp, talking to the applicant and seeing if they’re full of shit.
That's already done now. People in Academia have excellent grades, and they like what they do. How would peers know about their ability to do novel research, not having done any themselves?
Current system is like measuring programmer productivity by lines of code so I think it is hard to do worse than what we currently do.
Interestingly, this article defends traditionally established peer reviewing structures claiming newer reforms aren't needed
Is novel research worth doing?
It depends on why you are doing it, I suppose. The framing of the piece seems to have a lot of baked in assumptions that likely make sense if your goal is, I guess, successful career in academia.
That's not the driving force behind all research. Research pursued for other reasons may get little to no press but may be successful for the metrics/goals that drove the decision.
It depends on why you are doing it, I suppose. The framing of the piece seems to have a lot of baked in assumptions that likely make sense if your goal is, I guess, successful career in academia.
That's not the driving force behind all research. Research pursued for other reasons may get little to no press but may be successful for the metrics/goals that drove the decision.
Yes and no, but first we need to define novelty.
I'm in ML and all the academic issues are on full display here (comes with any hyped research area). Reviewer 2 will always knock you on novelty no matter what you do. That's because all research is to some extent derivative. As it should be. We shouldn't be reinventing the wheel. But on the other hand, all research is incremental (especially in a publish or perish system. You can't publish breakthroughs frequently). Leaps and bounds don't happen often and they generally do either by accident (someone doing something dumb) or because someone was spending a lot of time on that area (likely not publishing in main track or at all). But novelty is also hard to measure post hoc. Most things are non-obvious a priori but are obvious post hoc. Reviewers go after these all the time and all it encourages is researchers to make their works convoluted. I've literally been told that papers I've written are too simple and straight forward. I've also seen those papers get better reviews after complexifying the text. I don't feel good about this btw.
But I want to give a perfect example of all this in ML. Diffusion models. The stochastic noise process has been around for decades. People have been applying them to generative models for awhile too. But they were basically unknown until they beat GANs at image generation and within a few short months they are a household name (even more than GANs). This looks like a bigger breakthrough than it is because they were flying under the radar for so long and really just one team spent a lot of time with them AND had access to A LOT of compute resources. But still, I think this should make you think about how novelty works and if you're reviewing, maybe don't squash papers because you think they are dead ends. Diffusion was a "dead end" until it wasn't.
I'm in ML and all the academic issues are on full display here (comes with any hyped research area). Reviewer 2 will always knock you on novelty no matter what you do. That's because all research is to some extent derivative. As it should be. We shouldn't be reinventing the wheel. But on the other hand, all research is incremental (especially in a publish or perish system. You can't publish breakthroughs frequently). Leaps and bounds don't happen often and they generally do either by accident (someone doing something dumb) or because someone was spending a lot of time on that area (likely not publishing in main track or at all). But novelty is also hard to measure post hoc. Most things are non-obvious a priori but are obvious post hoc. Reviewers go after these all the time and all it encourages is researchers to make their works convoluted. I've literally been told that papers I've written are too simple and straight forward. I've also seen those papers get better reviews after complexifying the text. I don't feel good about this btw.
But I want to give a perfect example of all this in ML. Diffusion models. The stochastic noise process has been around for decades. People have been applying them to generative models for awhile too. But they were basically unknown until they beat GANs at image generation and within a few short months they are a household name (even more than GANs). This looks like a bigger breakthrough than it is because they were flying under the radar for so long and really just one team spent a lot of time with them AND had access to A LOT of compute resources. But still, I think this should make you think about how novelty works and if you're reviewing, maybe don't squash papers because you think they are dead ends. Diffusion was a "dead end" until it wasn't.