How To Become An Open-Source Contractor(emiller.info)
emiller.info
How To Become An Open-Source Contractor
http://emiller.info/contracting-advice.html
7 comments
It seems every day someone or other discovers this, and the sheer magnitude of money there is to be made (and the bonus: having fun while doing it). I discovered it early this year, when I realized I was only one of two open-source developers in the entire world with my specific expertise... and the other apparently has been rather difficult to hire.
Great, congratulation. I'd be curious to hear more about your experiences. ;-)
Last summer, I started working on x264, the only open source H.264 video encoder. I knew basically nothing and was experimenting with the simplest things and harassing the only other active developer for explanations.
Slowly but surely, I learned the codebase. Eventually, I got my first major patch in: a modification to the subpixel motion search that improved efficiency a bit. But I still found myself blackboxing most of the code due to lack of understanding. Bit by bit, I made more and more complex changes. I taught myself x86 assembly, and then SIMD assembly. I finally learned the subtleties of C pointers. I finally realized that the best way to learn how things work is to identify some way to improve them, and then go and do it--the process of doing it will eventually teach you how it works, even if solely by forcing you to ask an enormous number of questions.
By late winter 2007-8, I probably knew the codebase better than anyone else in the world except the main developer, Loren Merritt, who had been the sole maintainer for almost three years. I was still learning new things all the time, of course, and his knowledge vastly eclipsed mine. But this limited knowledge of mine got me a contract from Avail Media, who wanted an adaptive quantization algorithm implemented, primarily to stop the problem of flat textured areas being blurred out (such as grass in football games). Bit by bit I tried various ideas, and finally got a working algorithm in January that did the job tremendously well. This was committed into x264 in April.
I then got another contract from them to improve the VBV compliance (buffering for constant and limited max bitrate encoding), since they're interested in x264 for television broadcast purposes. I worked on this, which was eventually later improved upon by Gabriel (an employee of Joost, a company that uses x264) and turned into a complete overhaul to the buffering system, which resulted in much improved compliance.
Finally, I got hired for a summer internship there at a rather decent salary.
In the meantime, Adobe had released the latest version of Flash 9, which supported H.264 video. Suddenly, thousands upon thousands of websites were interested in switching--the old FLV1 video format was simply awful, and VP6 was only mildly better and, unlike FLV1 and H.264, very expensive to license. There was a massive flood of people looking for assistance integrating x264 into their toolchains--there simply was no practical proprietary alternative for the vast majority of these websites. Most of the low-cost proprietary H.264 encoding apps were non-commercial only (and much inferior to x264) and the higher-cost ones were simply way out of their price range (and arguably mostly inferior too...).
So, as a developer of x264, I was in demand. I knew the application better than almost anyone else. Everyone wanted me to help them. I got contract offers from Criterion Collection, Facebook, and others. There were also loads of smaller sites looking for more minor help, or even just quick guidance on IRC. x264 is now far beyond a tool for DVD backups, anime, and Scene releases; it was a standard tool used far and wide. And again, I am one of only two active developers. Furthermore, after over a year of being involved in development, I basically made myself an expert in both x264 and the H.264 standard, now knowing most of the code backwards and forwards and having read most of the spec at one point in time or another.
And what about the other developers I mentioned?
Well, the main developer was very difficult to motivate. He didn't seem to care much about money, and had a habit of blowing off past contracts. If you can convince him that some of his assembly code is suboptimal though, he'll be right on it ;)
There were many other minor developers--mostly people who submitted bugfixes or worked on one portion of the code. For example, Gabriel worked primarily on ratecontrol and buffering, which was the most important thing given his job.
And thus I was left as the single developer who was willing to take all sorts of contracts, available to do them, and with enough expertise to deal with almost any part of the application, code-wise, algorithm-wise, or simply usage-wise.
Slowly but surely, I learned the codebase. Eventually, I got my first major patch in: a modification to the subpixel motion search that improved efficiency a bit. But I still found myself blackboxing most of the code due to lack of understanding. Bit by bit, I made more and more complex changes. I taught myself x86 assembly, and then SIMD assembly. I finally learned the subtleties of C pointers. I finally realized that the best way to learn how things work is to identify some way to improve them, and then go and do it--the process of doing it will eventually teach you how it works, even if solely by forcing you to ask an enormous number of questions.
By late winter 2007-8, I probably knew the codebase better than anyone else in the world except the main developer, Loren Merritt, who had been the sole maintainer for almost three years. I was still learning new things all the time, of course, and his knowledge vastly eclipsed mine. But this limited knowledge of mine got me a contract from Avail Media, who wanted an adaptive quantization algorithm implemented, primarily to stop the problem of flat textured areas being blurred out (such as grass in football games). Bit by bit I tried various ideas, and finally got a working algorithm in January that did the job tremendously well. This was committed into x264 in April.
I then got another contract from them to improve the VBV compliance (buffering for constant and limited max bitrate encoding), since they're interested in x264 for television broadcast purposes. I worked on this, which was eventually later improved upon by Gabriel (an employee of Joost, a company that uses x264) and turned into a complete overhaul to the buffering system, which resulted in much improved compliance.
Finally, I got hired for a summer internship there at a rather decent salary.
In the meantime, Adobe had released the latest version of Flash 9, which supported H.264 video. Suddenly, thousands upon thousands of websites were interested in switching--the old FLV1 video format was simply awful, and VP6 was only mildly better and, unlike FLV1 and H.264, very expensive to license. There was a massive flood of people looking for assistance integrating x264 into their toolchains--there simply was no practical proprietary alternative for the vast majority of these websites. Most of the low-cost proprietary H.264 encoding apps were non-commercial only (and much inferior to x264) and the higher-cost ones were simply way out of their price range (and arguably mostly inferior too...).
So, as a developer of x264, I was in demand. I knew the application better than almost anyone else. Everyone wanted me to help them. I got contract offers from Criterion Collection, Facebook, and others. There were also loads of smaller sites looking for more minor help, or even just quick guidance on IRC. x264 is now far beyond a tool for DVD backups, anime, and Scene releases; it was a standard tool used far and wide. And again, I am one of only two active developers. Furthermore, after over a year of being involved in development, I basically made myself an expert in both x264 and the H.264 standard, now knowing most of the code backwards and forwards and having read most of the spec at one point in time or another.
And what about the other developers I mentioned?
Well, the main developer was very difficult to motivate. He didn't seem to care much about money, and had a habit of blowing off past contracts. If you can convince him that some of his assembly code is suboptimal though, he'll be right on it ;)
There were many other minor developers--mostly people who submitted bugfixes or worked on one portion of the code. For example, Gabriel worked primarily on ratecontrol and buffering, which was the most important thing given his job.
And thus I was left as the single developer who was willing to take all sorts of contracts, available to do them, and with enough expertise to deal with almost any part of the application, code-wise, algorithm-wise, or simply usage-wise.
Thanks for posting that - I found it really inspiring. I'd just like to ask what it was that got you interested in the x264 project in the first place?
I found myself using it for encoding game videos of EVE Online, and had noticed that it got massively better quality at the same bitrate as compared to the more popular WMV9--at least on those videos, it could get 2-3x better compression at the same quality. So I promoted and used it, and eventually I decided to learn how it worked and see if I could make it better.
Since then, I've moved on from EVE Online; in fact, I would say the primary cause is I found that x264 development was more interesting (if college itself wasn't enough of a reason!) I still use it for all sorts of video compression, of course, and every once in a while I find myself quite literally improving it because it gave suboptimal results on some video I wanted to post online.
By the way, I didn't note it in my above post, but I'm a sophomore in college (CS major), which puts it quite into perspective, I think.
I always tell people that they don't necessarily need lots of background knowledge or experience to get involved in something as complex, as, say, video encoder development. Almost none of my prior knowledge was at all useful in learning how x264 worked. At most my linear algebra and Fourier knowledge from my education went to some use on a theoretical level, but that's about it, and it was only "nice to have"; I could have done as well without. It seems to be this way with most jobs I've taken, too; 90% of what you need to know ends up being learned on the job. What really matters is your ability to learn and adapt, not your prior knowledge base.
Since then, I've moved on from EVE Online; in fact, I would say the primary cause is I found that x264 development was more interesting (if college itself wasn't enough of a reason!) I still use it for all sorts of video compression, of course, and every once in a while I find myself quite literally improving it because it gave suboptimal results on some video I wanted to post online.
By the way, I didn't note it in my above post, but I'm a sophomore in college (CS major), which puts it quite into perspective, I think.
I always tell people that they don't necessarily need lots of background knowledge or experience to get involved in something as complex, as, say, video encoder development. Almost none of my prior knowledge was at all useful in learning how x264 worked. At most my linear algebra and Fourier knowledge from my education went to some use on a theoretical level, but that's about it, and it was only "nice to have"; I could have done as well without. It seems to be this way with most jobs I've taken, too; 90% of what you need to know ends up being learned on the job. What really matters is your ability to learn and adapt, not your prior knowledge base.
That within a year you've become extremely proficient with a non-trivial, math-heavy video compression standard says a lot for your baseline intelligence. Congratulations for sticking at it, although I doubt most of us in the lower 95% could have done the same! :)
I doubt most people on HN are in the lower 95% ;)
I'd say its more a matter of dedication and the sheer amount of time I've spent on it, plus the fact that I was able to skip a whole lot of it by basing my earlier insights off those of others rather than trying the (utterly hopeless) strategy of learning it all myself from the spec.
I'd also say the "math-heavy" is rather exaggerating it; the entire spec has not an ounce of math in it. All the numerical computations are written in pseudocode, not formulas; its basically written as if a computer was reading it instead of a human. Personally I have found this to be a rather terrible attribute, as it makes some of it nearly completely incomprehensible.
I also find this approach is often the kind of thing that leads people to assume they cannot do something; they think that only "really smart" people can possibly do some particular thing, and refuse to try as a result. Of course, it can also go the other way--because someone does something hard, they insist that they must be really smart, or else they couldn't have done it. This only reinforces the problem.
There also seems to be the rather misleading assumption that younger people are somehow less smart on average, and thus if a younger person does something hard, they must be even smarter than they would be assumed to be otherwise. I find this to be completely false; I don't think I've gotten one ounce better at math than I was in middle school, for example. People get more experienced, wiser and more knowledgeable, but I don't think they get much smarter.
Though, ironically, I don't actually think I am that smart; if grades are any indication, my last semester is clear proof that I'm not ;)
I'd say its more a matter of dedication and the sheer amount of time I've spent on it, plus the fact that I was able to skip a whole lot of it by basing my earlier insights off those of others rather than trying the (utterly hopeless) strategy of learning it all myself from the spec.
I'd also say the "math-heavy" is rather exaggerating it; the entire spec has not an ounce of math in it. All the numerical computations are written in pseudocode, not formulas; its basically written as if a computer was reading it instead of a human. Personally I have found this to be a rather terrible attribute, as it makes some of it nearly completely incomprehensible.
I also find this approach is often the kind of thing that leads people to assume they cannot do something; they think that only "really smart" people can possibly do some particular thing, and refuse to try as a result. Of course, it can also go the other way--because someone does something hard, they insist that they must be really smart, or else they couldn't have done it. This only reinforces the problem.
There also seems to be the rather misleading assumption that younger people are somehow less smart on average, and thus if a younger person does something hard, they must be even smarter than they would be assumed to be otherwise. I find this to be completely false; I don't think I've gotten one ounce better at math than I was in middle school, for example. People get more experienced, wiser and more knowledgeable, but I don't think they get much smarter.
Though, ironically, I don't actually think I am that smart; if grades are any indication, my last semester is clear proof that I'm not ;)
I also find this approach is often the kind of thing that
leads people to assume they cannot do something; they think
that only "really smart" people can possibly do some
particular thing, and refuse to try as a result. Of course,
it can also go the other way--because someone does
something hard, they insist that they must be really smart,
or else they couldn't have done it. This only reinforces
the problem.
Of course its a problem, it means I have fewer minions to assign to various x264 development tasks. ;)
- Code exactly what your client wants, not what you think the world wants. -
I had to point this out. Having been a leader on a good amount of projects that involved programming, I have completely lost count of the number of times someone didn't do their job and coded something completely different because they thought they were superman and could save the world by coding something else or coding it differently than what we thought. It turns out when their submission goes to be implemented, it doesn't work at all with the rest of the system.
I had to point this out. Having been a leader on a good amount of projects that involved programming, I have completely lost count of the number of times someone didn't do their job and coded something completely different because they thought they were superman and could save the world by coding something else or coding it differently than what we thought. It turns out when their submission goes to be implemented, it doesn't work at all with the rest of the system.
As someone looking to hire people like this, I whole-heartedly agree. The one thing - make sure that you write for both the technical and the suit audiences.
It's my job to find resources and present them to the technical team, who make the final decisions. But they decide from what I present to them. If you can't speak suit, you can't get to the techs, who make the hiring call. (But the check has my signature on it, so make sure that you're nice to both of us.)
It's my job to find resources and present them to the technical team, who make the final decisions. But they decide from what I present to them. If you can't speak suit, you can't get to the techs, who make the hiring call. (But the check has my signature on it, so make sure that you're nice to both of us.)
What areas are ripe for this sort of consulting? (Ignoring the fact that if you go into an area to make money rather than because you're rabidly interested in it, it's probably a bad idea.)
It seems there are two characteristics that would make an area ripe for this -- companies needing it badly, and some (real or perceived) barrier to entry that keeps legions of geeks from learning it.
Any other likely candidates?
It seems there are two characteristics that would make an area ripe for this -- companies needing it badly, and some (real or perceived) barrier to entry that keeps legions of geeks from learning it.
Any other likely candidates?
[deleted]
An inspiring article - so are the comments from darkshikari!