I've recently been in a rut and while I still have a lot further to go I've been feeling better over the last couple of days and here's what helped:
- going to therapy
- taking PTO
- going outside
- hanging out with friends
- cooking a meal
- going on a date
- going to LA and getting sun
- reconnecting with old friends
I'm not out yet but I feel like with some time and effort I can be. How did I do it? I think there are really two parts of this - helping others and asking for help. I've always enjoyed the first but never done the second and that was really holding me back.
ML training is not as easily parallelizable as the other problems that have been explored. I'm not familiar with SETI but I know this to be true for folding@home.
As you mentioned, ML training can be parallelized but this requires either model/data parallelism.
Data parallelism means spreading the data over many different compute units and then synchronizing gradients somehow. The heterogeneous nature of @home computing makes this particularly challenging, as you will be limited by the smallest compute unit. I've personally only ever seen data (and model) parallel done on a homogenous compute cluster (i.e. 8x GPUS)
For model parallelism, we split the model across different compute units. However, this means that you need to synchronize the different parts of the model together, which can get very expensive when you do it across the internet. If you have 8xGPUS on one machine, your latency is limited by PCIe instead of TCP/IP in a distributed @home cluster.
But I would say it's not impossible, someone clever could definitely figure it out.
I graduated in 2020 and was considering doing a PhD in NLP as I had already been doing research at a lab. I ended up joining a series A startup, trying to turn research into products.
I don't regret it much, as I was tired of being a broke student. Also I think the problems you run into in industry are more interesting. However, I do think my skills stagnated a bit and I learned more when doing research.
I like this post a lot, even if it's a somewhat contrived example. In particular I like his point about switch statements making it easier to pulled out shared logic vs. polymorphic code.
There's so much emphasis on writing "clean" code (rightly so) that it's nice to hear an opposing viewpoint. I think it's a good reminder to not be dogmatic and that there are many ways to solve a problem, each with their own pros/cons. It's our job to find the best way.
You negotiate around dilution by asking for additional equity grants, like you would for a raise / promotion. But in general dilution is a good problem to have because it means you are raising more money and growing.
In general I would say expert knowledge is more valuable. However, I think of knowledge as a pyramid - how high you can go is determined by how wide/strong your foundation is.
In my experience I've found that trying to develop deep expertise requires a solid understanding of many different fundamentals. For example, when I was doing NLP research, I had to learn about distributed systems in order to diagnose problems when training with multiple GPUs, or about dynamic libraries for fixing CUDA errors.
Very uncommon for people to do ML locally nowadays. Almost everyone uses either cloud or has an cluster they ssh into. NVIDIA is still king for ML, very few people use AMD / M1, although hopefully that will change.
I think it's really hard to think up of curriculum for a "programming" class. It'd be hard to standardize across different professors. Also while I do think that more programming experience would benefit most new grads, it's hard for me to say that it would be more beneficial than say better interpersonal working skills or even specialized knowledge.
FWIW at the uni I attended, projects were a huge part of some classes (usually ~50%) and there were some SWE specific classes where you learn best practices / tooling. You would need to be able to program in order to graduate. I feel like this is the norm and not the exception, but I may be wrong.
There were a lot of CS clubs at my school that hosted workshops and tried to teach people some of this stuff. Maybe you can find a club (or start one) at your school.
Startups are also not profitable yet and are reliant on fundraising in order to continue to operate. The fundraising market is bleak currently, so there's pressure to reduce burn.
I found reading more code helped me to become a better programmer.
It helps a lot that reading code (at least for me) is much faster than writing code, so you can do more of it.
Old techniques are just as good as the new ones :)
If you have a specific problem you are trying to solve, I'd recommend just reading papers and looking up things you don't understand.
Otherwise, if you're just looking for general knowledge, a class could be helpful.
My best advice is to find people who do what you want to do and try to learn as much as possible from them. If you're interested in doing ML/DL research I think the best way to get into the field is to reach out to professors. I studied ML/DL (books, projects, classes, reimplementing papers) for several years in undergrad, but discussing and debating ideas is the one thing that took my understanding to a much deeper level. A good professor will also point out gaps in your knowledge that you might be missing.
A second bit of advice: Programming (and execution) skills are IMO heavily undervalued by people looking to get into ML. The faster you can write code, debug, and implement new things, the easier it is to produce good research.
Some books I liked: PR & ML (Bishop), Deep Learning Book (Goodfellow), AI: A Modern Approach (Norving), Elements of Statistical Learning (Friedman)
Bay -> LA (Go Bruins!) -> NYC