Lessons from a year of AI research(jetnew.notion.site)
jetnew.notion.site
Lessons from a year of AI research
https://jetnew.notion.site/100-Lessons-from-1-Year-of-AI-Research-cf431d0f43764750acb83462de06979b
13 comments
Love the timeline of failure and success (the latter of which is what one usually sees).
I think there may be some who breeze through grad school, likely by being in a strong research environment beforehand, or by having lots of support. I mean, they have to exist?
Have I genuinely met anyone like that? Nah. And, oh boy, can it be a struggle!
During a PhD, it's very easy to put yourself into an increasingly hopeless situation that is mentally tough, for years on end, and it is really hard to even convey to anyone outside of academia why you would do that.
It is especially hard to convey to people outside of academia how 5 (nowadays often 6,7, etc) years of work all come down to a few, often seemingly random decisions of hiring committees and other structures.
You start grad school thinking it's something you build, bit by bit. At then end, it's more of an ever shifting collection of you, put together in the hopes it will be evaluated well. But then you realize, it really has little to do with objective quality, and much of the situation is outside of your control. There is no short project in grad school. It's the entire thing!
I think the worst thing is the uncertainty about literally everything. Is your Professor ever telling the truth? Will you really get that support two years out? Will the funding persists? What are you even researching?
I have seen people had the work of the past five years ripped apart during their thesis defense (or wherever), I have seen people make a few missteps and ruin their chances on the job market. I have seen people invest a lot in work that never makes any impact. I have seen people who thought they'd make Professor, and then found they have zero motivation or talent for research and/or teaching.
You can never be sure that ain't you, until you have your PhD and your job.
It's always on your mind. Projects aren't short anymore, you work years on stuff before it ever becomes a paper. You work every waking hour and often every sleeping hour as well. And you lack the knowledge and experience to assess whether what you are doing will really go anywhere, if the network you are building is right, if the people you write to will be interested in your work. At the very least, you never know who else starts with you and will compete with you on the job market, which is increasingly filled with people willing to work for little money and little job security.
In the past two years, academic positions dropped 70% or more, at least in some fields. And we have 2-3 cohorts of applicants on the market due to Corona, and this will remain for years.
Academic research nowadays requires nerves of steel, and I have seen grad school ruin the health (mental and physical) of quite a few people, all of them very good at what they do.
So after all that doom and gloom, people need to realize that it is a struggle for probably everyone. But it is also rewarding, or it can be.
I think there may be some who breeze through grad school, likely by being in a strong research environment beforehand, or by having lots of support. I mean, they have to exist?
Have I genuinely met anyone like that? Nah. And, oh boy, can it be a struggle!
During a PhD, it's very easy to put yourself into an increasingly hopeless situation that is mentally tough, for years on end, and it is really hard to even convey to anyone outside of academia why you would do that.
It is especially hard to convey to people outside of academia how 5 (nowadays often 6,7, etc) years of work all come down to a few, often seemingly random decisions of hiring committees and other structures.
You start grad school thinking it's something you build, bit by bit. At then end, it's more of an ever shifting collection of you, put together in the hopes it will be evaluated well. But then you realize, it really has little to do with objective quality, and much of the situation is outside of your control. There is no short project in grad school. It's the entire thing!
I think the worst thing is the uncertainty about literally everything. Is your Professor ever telling the truth? Will you really get that support two years out? Will the funding persists? What are you even researching?
I have seen people had the work of the past five years ripped apart during their thesis defense (or wherever), I have seen people make a few missteps and ruin their chances on the job market. I have seen people invest a lot in work that never makes any impact. I have seen people who thought they'd make Professor, and then found they have zero motivation or talent for research and/or teaching.
You can never be sure that ain't you, until you have your PhD and your job.
It's always on your mind. Projects aren't short anymore, you work years on stuff before it ever becomes a paper. You work every waking hour and often every sleeping hour as well. And you lack the knowledge and experience to assess whether what you are doing will really go anywhere, if the network you are building is right, if the people you write to will be interested in your work. At the very least, you never know who else starts with you and will compete with you on the job market, which is increasingly filled with people willing to work for little money and little job security.
In the past two years, academic positions dropped 70% or more, at least in some fields. And we have 2-3 cohorts of applicants on the market due to Corona, and this will remain for years.
Academic research nowadays requires nerves of steel, and I have seen grad school ruin the health (mental and physical) of quite a few people, all of them very good at what they do.
So after all that doom and gloom, people need to realize that it is a struggle for probably everyone. But it is also rewarding, or it can be.
Your post rings true. On the contrary the linked article was written by an undergrad student having some introduction to research, for a short time (a year) with a supporting supervisor and network so of course his list is naive and idealistic. In reality research and methodology etc. doesn't matter much, it's baseline skill. 70-80% of a PhD come done to social things like relationship with advisor, ability to write papers that will be accepted for publication even if they're bullshit.
The one that matters is missing: how do you get the best random seeds? /s
# Sussman attains enlightenment
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
“What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
“Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said.
Minsky then shut his eyes.
“Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.”
At that moment, Sussman was enlightened.
http://www.catb.org/jargon/html/koans.html
In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.
“What are you doing?”, asked Minsky. “I am training a randomly wired neural net to play Tic-Tac-Toe” Sussman replied.
“Why is the net wired randomly?”, asked Minsky. “I do not want it to have any preconceptions of how to play”, Sussman said.
Minsky then shut his eyes.
“Why do you close your eyes?”, Sussman asked his teacher. “So that the room will be empty.”
At that moment, Sussman was enlightened.
http://www.catb.org/jargon/html/koans.html
I'm not actually sure what the morale is here
If you shut your eyes, you do not see anything in the room. This does not mean there is nothing in the room.
If you randomize your seeds, you do not deliberately bias your model. This does not mean your model is unbiased.
If you randomize your seeds, you do not deliberately bias your model. This does not mean your model is unbiased.
Intelligence does not start from a random state, but from a pretrained one. Random state is likely to roll an artificial overfit or if it just fits, you won't know why it is better than another state - or rather why your training method sucks from other initial states.
Proper methods of bias analysis are network compression or random surgery, and expanding test data set. Proper methods of training are robust to varied initial states.
Proper methods of bias analysis are network compression or random surgery, and expanding test data set. Proper methods of training are robust to varied initial states.
41. Remember to consider variance of results and the usage of random seeds.
Great one! Actual question: has anyone looked if pseudo randomnes is worse than full randomness?
I once (8 years ago) did a comparison between random, pseudorandom and quasirandom. Quasirandom worked ever so slightly better than the other two in speed and final performance, but not enough to warrant the additional complexity of implementation.
It is pseudoworse.
That site hijacks the CMD+LEFT and CMD+RIGHT Mac hotkeys for browser BACK/FORWARD and also has very strange behavior when scrolling with the keyboard. Why? Why? Why?
Hi, I'm the author of the post. I wrote it on Notion to seek feedback from a few friends, but seems like one friend shared the Notion link here :) Hope that my website would provide a better reading experience instead - https://jetnew.io/blog/2021/100-lessons/
This seems really like a browser problem, and I'd love to know a fix. Geogebra.org hijacks CMD-` in Safari and it is absolutely infuriating.
As well as up, down, pgup, pgdown, alt+left and alt+right on Chromium Linux. Stopped reading when I saw this enforcement of how they think their site should be experienced.
Find peace in doing trial and error statistical black box studies after x years of formal systems studies. I couldnt do that for my part.
The title here and on the blog differs, I don't know why submiters pick other titles. I mean I wouldn't quote a newspaper or paper with a wrong title, would I?
It is one of the site's guidelines.
>If the title contains a gratuitous number or number + adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
>If the title contains a gratuitous number or number + adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
Okay and why?
PG considered them a special case of linkbait. Which is the only other time you are supposed to modify the title.
I think there are many click bait titles other than "10 things ..." lists. Still don't understand why there's a special rule just for these.
"You Can Now Save Money with Y New Strategy”
"You Can Now Travel Abroad Without Having to…"
"The Last … You’ll Ever Need"
"You Won’t Believe… What Y has Found"
"Why You Should…"
"Why You’ve Never Heard of This Top Travel Destination"
"This is why you’re losing money"
Just a few examples.
"You Can Now Save Money with Y New Strategy”
"You Can Now Travel Abroad Without Having to…"
"The Last … You’ll Ever Need"
"You Won’t Believe… What Y has Found"
"Why You Should…"
"Why You’ve Never Heard of This Top Travel Destination"
"This is why you’re losing money"
Just a few examples.
Most of those would be rewritten to a more descriptive title if they hit the front page.
Agreed.
"Y"
Do AI researchers have a commonly accepted definition for intelligence?
There is also no specification for "AI".
There isn't even an accepted definition of intelligence in humans, let alone animals, never mind machines! We're pretty much at the "I know it when I see it" stage of definition.
How can that be true? If the author is doing AI research presumably they and others in the same field have have some foundational shared ideas that they build upon. If they didn't how would a person earning a PhD in this field ever be able to say if they are contributing something new to the study of AI? There has to be something that defines the work as work in AI rather than just computer science.
A large foundation for this field is statistical inference. To me, AI almost always means ML, and ML = algorithms that optimize themselves to make predictions.
You have correctly deduced why AI research is just a paper factory.
[deleted]
Not really, AI is more of a large collection of activities. I would distill it down to "problem-solving".
The irony is that once you solve a problem, it's not a problem, so people's natural reaction is to only call things AI when they're unsolved. Once it's solved, "that's not AI".
The irony is that once you solve a problem, it's not a problem, so people's natural reaction is to only call things AI when they're unsolved. Once it's solved, "that's not AI".
Ability to solve novel problems with little experience. Skill acquisition efficiency.
On the Measure of Intelligence, François Chollet https://arxiv.org/abs/1911.01547
On the Measure of Intelligence, François Chollet https://arxiv.org/abs/1911.01547
That's not really compatible with the history of AI, where designing systems to solve a problem has traditionally been considered in scope. Seems that, like many people, Chollet wants to think of AI in terms of AGI (artificial general intelligence).
Yes, that's what he focuses on. General intelligence, not narrow task-related intelligence. If a system uses too much experience (training samples) to learn a task then it's not intelligent. It just brute forces the problem. Intelligence requires quick learning from very little experience.
101st lesson - 100 is a nice number but no one is going to read a blog post with 100 lessons learned. Either focus on the 5-10 most important lessons or consolidate and summarize.
The author might find that this well thought out list will help when applying for work. I would suggest that they copy it over to GitHub in addition to their public projects.
Original post:
https://jetnew.io/blog/2021/100-lessons/
https://jetnew.io/blog/2021/100-lessons/
How do you formally verify your neural networks and the like? Or is formal verification possible? Does this limit the areas where it can be applied?
-> 87. Don't let yourself be too affected by the opportunity costs of doing research.
Why not? Is there a strong payoff later? How did you learn this lesson?
:)
Good listicle!
Why not? Is there a strong payoff later? How did you learn this lesson?
:)
Good listicle!
I reflected upon how I approached many past opportunities, and realised that I only truly enjoy the process if the nature of work is interesting to me because I'm mostly interest-driven. I also found that enjoying the process is already rewarding regardless of the outcome, and at the same time have much more control over the process than the outcome. :)
Thanks for the write up! As someone starting a doctoral degree early next year, I greatly appreciate it
Lol this is so aspirational it could only come from an undergrad.
Let me tell you that I've finally made it to the stressful part of the being a serious "AI" researcher, where I have a real project (as in difficult to achieve goals, not just "turn the crank" stuff) and real deadlines (deliverables on collaborators projects and my own conferences submissions) and the only thing I prioritize above doing the work itself is keeping my advisor (and other collaborators) up to date on what I'm doing so that when he reads my paper draft he's not completely lost. Everything like organizing papers, citations, logging infra, etc is meaningless when you're trying to piece together a solution. Like seriously somedays I barely have time to exercise and eat dinner with my wife (let alone organizing my bookmarks).
For example I'm trying to solve a particular compilers problem using integer programming (note that at a high this isn't that high level because this is a small cottage industry) and so I have like 50 paper tabs open that I bounce between when thinking/experimenting. The way it usually goes is I'll hack, get stuck, go back to the papers, find something, hack, and on. And usually the eureka moment comes some hours later because I connect something.
You might say that I'm a bad researcher but I know for a fact (external validation) that I'm not. And if you look at other highly productive researchers (like TT track profs at my "elite" school) this is indeed how they work. All of this zotero, notion, mlflow stuff is of the ilk of productivity porn for other flavors of knowledge workers (ie a mirage and/or snake oil). Let me put it this way: my advisor is a top 500 h-index person (the exact significance of that metric notwithstanding) and he doesn't have a bibtex of his own papers, let alone zotero for all of the papers he reads/comes across.
The only thing that matters is code/math/etc output (whatever your material output is) and your abilities are also highly correlated with it with the casualty flowing in t opposite direction (make more stuff and you'll get better at making stuff).
But I guess conversely do do some of these things when you're young and have the time (and I don't mean that condescendingly). E.g. reading outside of your area is probably the most valuable (from my own, admittedly a typical, experience, since I jumped domains many times); I very frequently can outpace even my senior colaborators very quickly on understanding a problem and solution simply because when I was younger I dabbled in ... all the things (physics, math, cs).
The other thing that I'll say is there's something obviously missing from this list but only if you've really made it this far: collaborators and interactions with collaborators. The only thing that matters aside from the produce is getting people to make use of it. That means writing, speaking, and getting buyin from your collaborators. If you really truly want to be successful then work on your people skills as it pertains to this area - that means learn to speak the language of your research community, learn to give good (engaging, interesting, useful) presentations, learn to write well (including making nice diagrams), and learn to explain things in ways that smart but busy people will understand. Besides all of this being key to being productive it's also what feeds you (i.e. the real #1 priority) since it gets you jobs, academic and industry.
Let me tell you that I've finally made it to the stressful part of the being a serious "AI" researcher, where I have a real project (as in difficult to achieve goals, not just "turn the crank" stuff) and real deadlines (deliverables on collaborators projects and my own conferences submissions) and the only thing I prioritize above doing the work itself is keeping my advisor (and other collaborators) up to date on what I'm doing so that when he reads my paper draft he's not completely lost. Everything like organizing papers, citations, logging infra, etc is meaningless when you're trying to piece together a solution. Like seriously somedays I barely have time to exercise and eat dinner with my wife (let alone organizing my bookmarks).
For example I'm trying to solve a particular compilers problem using integer programming (note that at a high this isn't that high level because this is a small cottage industry) and so I have like 50 paper tabs open that I bounce between when thinking/experimenting. The way it usually goes is I'll hack, get stuck, go back to the papers, find something, hack, and on. And usually the eureka moment comes some hours later because I connect something.
You might say that I'm a bad researcher but I know for a fact (external validation) that I'm not. And if you look at other highly productive researchers (like TT track profs at my "elite" school) this is indeed how they work. All of this zotero, notion, mlflow stuff is of the ilk of productivity porn for other flavors of knowledge workers (ie a mirage and/or snake oil). Let me put it this way: my advisor is a top 500 h-index person (the exact significance of that metric notwithstanding) and he doesn't have a bibtex of his own papers, let alone zotero for all of the papers he reads/comes across.
The only thing that matters is code/math/etc output (whatever your material output is) and your abilities are also highly correlated with it with the casualty flowing in t opposite direction (make more stuff and you'll get better at making stuff).
But I guess conversely do do some of these things when you're young and have the time (and I don't mean that condescendingly). E.g. reading outside of your area is probably the most valuable (from my own, admittedly a typical, experience, since I jumped domains many times); I very frequently can outpace even my senior colaborators very quickly on understanding a problem and solution simply because when I was younger I dabbled in ... all the things (physics, math, cs).
The other thing that I'll say is there's something obviously missing from this list but only if you've really made it this far: collaborators and interactions with collaborators. The only thing that matters aside from the produce is getting people to make use of it. That means writing, speaking, and getting buyin from your collaborators. If you really truly want to be successful then work on your people skills as it pertains to this area - that means learn to speak the language of your research community, learn to give good (engaging, interesting, useful) presentations, learn to write well (including making nice diagrams), and learn to explain things in ways that smart but busy people will understand. Besides all of this being key to being productive it's also what feeds you (i.e. the real #1 priority) since it gets you jobs, academic and industry.
"but", Zotero is really neat if you use the plugins which make it just one-click-to-save it in your Zotero from browser. Hardly any overhead with that.
Here's my 2 cents on the topic from a thing I wrote last year ('Lessons Learned the Hard Way in Grad School (so far)'): https://www.andreykurenkov.com/writing/life/lessons-learned-...