Why developers should care about system packages(morethanseven.net)
morethanseven.net
Why developers should care about system packages
http://morethanseven.net/2011/01/16/Why-developers-should-care-about-system-packages.html
8 comments
* The software was usually built against specific versions (often modified) of 3rd party libraries. *
That sounds like a disaster waiting to happen... maintaining a different version of libraries outside of their main development trunk means that:
1. Extra steps when a new main version comes out - you have to repatch your differences in.
2. #1 incurs a delay, thus a longer time that your customer are running without patches, which may be a security issue.
3. You won't do #1 at all. Thus you become reliant on old, unmaintained, buggy, security-holey libraries.
The only time I can think this is acceptable is to fix problems that upstream won't fix, or are platform specific (ie, not with your custom software, but with the OS underneath everything).
That sounds like a disaster waiting to happen... maintaining a different version of libraries outside of their main development trunk means that:
1. Extra steps when a new main version comes out - you have to repatch your differences in.
2. #1 incurs a delay, thus a longer time that your customer are running without patches, which may be a security issue.
3. You won't do #1 at all. Thus you become reliant on old, unmaintained, buggy, security-holey libraries.
The only time I can think this is acceptable is to fix problems that upstream won't fix, or are platform specific (ie, not with your custom software, but with the OS underneath everything).
Patching in new versions isn't usually a huge deal, it's just not worth the trouble. Let's say we're using version 1.2, and we fixed a few bugs in it over the past year. Now version 1.3 is out. Should we upgrade?
Maybe 1.3 fixes a few bugs we haven't encountered yet. But that's unlikely. We've been using it for a year. Far more likely, as in practically a certainty, is that 1.3 will introduce some regression that won't be discovered until it affects some customer.
Not every bug in every program is a security vulnerability, and framing the issue that way distorts things.
Maybe 1.3 fixes a few bugs we haven't encountered yet. But that's unlikely. We've been using it for a year. Far more likely, as in practically a certainty, is that 1.3 will introduce some regression that won't be discovered until it affects some customer.
Not every bug in every program is a security vulnerability, and framing the issue that way distorts things.
Well, if you had uptreamed your fixes against 1.2, they would be in 1.3, wouldn't they ? I find it a poor excuse for not upgrade assuming API and ABI are stable.
It's just laziness. It's easier because that way you don't have to argue with Management. But you also don't educate on the necessity of fixing security issues. Hooray for 1.4 JVM still in production for instance. And it's perfectly ok. Until some breach happens .....
It's just laziness. It's easier because that way you don't have to argue with Management. But you also don't educate on the necessity of fixing security issues. Hooray for 1.4 JVM still in production for instance. And it's perfectly ok. Until some breach happens .....
Not every upstream will take your patches. Not every upstream can take your patches.
Getting my fixes into 1.3 doesn't prevent 1.3 from having regressions.
I'm not the OP, but I will respond anyway:
1. We have made changes to the 3rd party source code that we want to ship now, not when they finally let the patch into their source tree.
2. We get new updates from the latest release, patch in our stuff and we are set, quick test on the CI servers and we can ship. Yes, customers may be without the fix for a little while longer...
3. It is actually in our best interest to keep 3rd party libraries at their latest release, if we want secure software. Which is why it is part of our release documents and steps that are required to be followed before we release.
1. We have made changes to the 3rd party source code that we want to ship now, not when they finally let the patch into their source tree.
2. We get new updates from the latest release, patch in our stuff and we are set, quick test on the CI servers and we can ship. Yes, customers may be without the fix for a little while longer...
3. It is actually in our best interest to keep 3rd party libraries at their latest release, if we want secure software. Which is why it is part of our release documents and steps that are required to be followed before we release.
Then what you have is a fork of the library. There is no reason you can't distribute forks of the library (to either your own paths or with your own modified names) using the system package manager.
If you are expecting the person taking your patches to install them to the system, however, that's a whole 'nother ball game, and gets back to the original point: as a system administrator, I have no reason to believe your "better" versions of those libraries are going to interoperate with everything else on the system, and I am then going to have to do a bunch of work doing that isolation myself.
If you are expecting the person taking your patches to install them to the system, however, that's a whole 'nother ball game, and gets back to the original point: as a system administrator, I have no reason to believe your "better" versions of those libraries are going to interoperate with everything else on the system, and I am then going to have to do a bunch of work doing that isolation myself.
And if you allow the customer to upgrade libraries themselves (or install any version they desire) you'll risk breaking the application. That can be even worse.
> I'm not sure why it's the author's responsibility to go setup a dozen different VMs
To make sure it compiles and works under dozen different distributions (in practice centos + ubuntu/debian usually cover most cases). If you don't test it, how do you know it's usable at all? Besides, as long as you don't invent some crazy custom build system, the only thing you really have to write is metadata and dependencies (rules file can be 3 lines long for autotools software).
Also, this way you can just push your software to a build farm like launchpad ppa and let them do the dirty work. If you include self-contained tests with the sources, you can even run them after the build to verify correctness for that environment. This is much better than doing it by hand.
Also keep in mind that the first person to deploy your software automatically to a couple of places will either use your/distro's package or will have to make their own. Building from source works only for one-person uncontrolled environment.
To make sure it compiles and works under dozen different distributions (in practice centos + ubuntu/debian usually cover most cases). If you don't test it, how do you know it's usable at all? Besides, as long as you don't invent some crazy custom build system, the only thing you really have to write is metadata and dependencies (rules file can be 3 lines long for autotools software).
Also, this way you can just push your software to a build farm like launchpad ppa and let them do the dirty work. If you include self-contained tests with the sources, you can even run them after the build to verify correctness for that environment. This is much better than doing it by hand.
Also keep in mind that the first person to deploy your software automatically to a couple of places will either use your/distro's package or will have to make their own. Building from source works only for one-person uncontrolled environment.
Please add FreeBSD to the list of OS's you use for testing of compilation.
You have no idea how frustrating it is to have an open source project claim to run on unix like operating systems and not compile correctly on FreeBSD or Mac OS X...
You have no idea how frustrating it is to have an open source project claim to run on unix like operating systems and not compile correctly on FreeBSD or Mac OS X...
As much as I agree... I'm not sure how much it's worth the effort until someone points out that your project definitely doesn't work. Even looking at silly stats like: http://distrowatch.com/stats.php?section=popularity - covering one popular rpm and one deb distribution, you're covering majority already (with only slight modifications if necessary). Mac users will expect their own packages anyways and you can't run Mac OSX legally without owning the hardware, which many developers don't own.
Regarding FreeBSD... lack of any good result in the first 3 pages of "freebsd packaging guide" search should explain a lot.
Regarding FreeBSD... lack of any good result in the first 3 pages of "freebsd packaging guide" search should explain a lot.
Generally if it compiles on FreeBSD it compiles on Mac OS X, at least that has been my experience in the past. I am in the unique position that I develop on Mac OS X, then do my tests on FreeBSD and when it works on those attempt it on Linux.
The counter-argument is that testing on more platforms tends to expose bugs that it's worth fixing because they would have affected you anyway, you just didn't know it yet.
search for "freebsd porting guide".
Ok - I wasn't clear on that one. I know they're called ports and would probably look for stuff like port creation / porting / port-something-else. Yet, it seems a bit weird to me that looking for freebsd packaging guide doesn't give you anything even remotely useful. (consider some developer who has a couple hours to spend, thinks about making a freebsd port, looks for packaging information and concludes that it's not even documented / not worth learning right now)
In FreeBSD to create a package you create a portfile, the packages come free at that point ...
But I understand your concern.
But I understand your concern.
try using openbsd sometime... :)
Imagine I'm your average, workaday software developer.
If I've written some software to scratch a personal itch and I push it out to github because I reckon someone, somewhere might benefit at some arbitrary future point, or to show off to a future employer, or whatever, then I really do not care about making sure it compiles and works on a dozen distributions. Just getting it working to fix my problem, then publishing it, ticks all the boxes I'm interested in. After all, I've got fifteen other problems I need to solve this week, none of which involve spooling up VMs for distributions I never use.
If a packager then decides that their favourite distro would benefit from having my code packaged nicely, at what point does that become my responsibility? It Works For Me(tm), and beyond reviewing and applying patches other people might send me to improve it, I don't necessarily have any interest whatsoever in what happens outside my repository as long as it doesn't stop the code from solving my problem. It's possible that I might be both developer and packager for a distribution or two, but that's the exception, not the rule.
Personally, I think the right way is for a clear path from language-package to distro-package to be provided by my environment such that I don't have to think about it, and I think that the bulk of that responsibility should be shouldered by the distro - but it's insane to think that the language environment shouldn't help, because there is a mutual benefit there. I think we're nearly there with RVM and rubygems, but that's about as far from autotools as you can get.
If I've written some software to scratch a personal itch and I push it out to github because I reckon someone, somewhere might benefit at some arbitrary future point, or to show off to a future employer, or whatever, then I really do not care about making sure it compiles and works on a dozen distributions. Just getting it working to fix my problem, then publishing it, ticks all the boxes I'm interested in. After all, I've got fifteen other problems I need to solve this week, none of which involve spooling up VMs for distributions I never use.
If a packager then decides that their favourite distro would benefit from having my code packaged nicely, at what point does that become my responsibility? It Works For Me(tm), and beyond reviewing and applying patches other people might send me to improve it, I don't necessarily have any interest whatsoever in what happens outside my repository as long as it doesn't stop the code from solving my problem. It's possible that I might be both developer and packager for a distribution or two, but that's the exception, not the rule.
Personally, I think the right way is for a clear path from language-package to distro-package to be provided by my environment such that I don't have to think about it, and I think that the bulk of that responsibility should be shouldered by the distro - but it's insane to think that the language environment shouldn't help, because there is a mutual benefit there. I think we're nearly there with RVM and rubygems, but that's about as far from autotools as you can get.
> I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job.
If you're in the enviable position of having lots of people willing to put in the time to maintain your package, and a distro that thinks your software is sufficiently noteworthy to include in their distro. But if your software is relatively new and you want to encourage wide adoption, one way to help is to make the install process friendly and familiar for sysadmins as well as fellow developers. And if you speak their language that goes a long way towards building that trust.
If you're in the enviable position of having lots of people willing to put in the time to maintain your package, and a distro that thinks your software is sufficiently noteworthy to include in their distro. But if your software is relatively new and you want to encourage wide adoption, one way to help is to make the install process friendly and familiar for sysadmins as well as fellow developers. And if you speak their language that goes a long way towards building that trust.
I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job.
Agreed. I do, however, expect upstream to accept reasonable patches from package maintainers to fix building bugs or make it easier for people to build packages themselves.
For the few times I've done it, I've had no trouble getting upstream to accept an updated .spec file (or even a barebones .spec file) that they can distribute in their targz to make it easier for the people who want to build their software as a package if their distribution doesn't provide one.
Agreed. I do, however, expect upstream to accept reasonable patches from package maintainers to fix building bugs or make it easier for people to build packages themselves.
For the few times I've done it, I've had no trouble getting upstream to accept an updated .spec file (or even a barebones .spec file) that they can distribute in their targz to make it easier for the people who want to build their software as a package if their distribution doesn't provide one.
In lots of cases the separate package maintainer's do a good but ultimately thankless task. The example I gave between the rabbitmq packages which are built as part of the development effort and made available on the rabbitmq site and the solr packages which are sometimes out of date and not referenced by the site highlights the point I'm trying to make.
I consider it a significant black mark on a product, open source or otherwise, if it doesn't have native packages for the most popular Linux distros. It's even better if they provide a yum or apt repository.
It is a challenge to do so, but if we can do it for our products (which have vast and complicated dependency chains and need to perform a huge amount of post-configuration in order to be easy for non-technical users to use), damned near any other project ought to be able to do it.
The vast majority of really ugly security issues I've seen have been due to people having installed something from source, sometimes years ago, and not realizing they're running exploitable software because their package manager tells them they're up to date. This reason alone should be enough to keep people using the native package manager for as much as possible. But there are many other good reasons, a few of which have been touched on in this article.
It is a challenge to do so, but if we can do it for our products (which have vast and complicated dependency chains and need to perform a huge amount of post-configuration in order to be easy for non-technical users to use), damned near any other project ought to be able to do it.
The vast majority of really ugly security issues I've seen have been due to people having installed something from source, sometimes years ago, and not realizing they're running exploitable software because their package manager tells them they're up to date. This reason alone should be enough to keep people using the native package manager for as much as possible. But there are many other good reasons, a few of which have been touched on in this article.
Collaborating with distribution packagers is a lot better than maintaining your own repositories. Packagers should always be up to speed with the latest guidelines and best practices for their distribution. I have often seen badly built packages by upstreams.
That's not feasible in a lot of cases. Commercial software, for instance, is not generally welcome in distribution repositories.
No, but running a public repository for a given distribution is the way forward in that case, and that does involve knowing the lay of the land.
Which is exactly what I said in my previous comment.
Which is what I was agreeing with. Vociferously :-)
Your post boils down to something like "developers should ship Debian packages because I use Debian and all my favorite software packages are out of date or non-existent".
Maybe you should move away from a distro that freezes for 6 months to roll a stable release which won't get anything more than security/crash fixes for 2+ years.
Maybe you should move away from a distro that freezes for 6 months to roll a stable release which won't get anything more than security/crash fixes for 2+ years.
Many Debian developers agree: http://lwn.net/Articles/406301/ OTOH, it's nice to know that you can run in the same environment without changes for 2 years and not care about incompatible updates.
That wasn't my aim. In fact I'm not using Debian at the moment as I indicated, I just know the commands better to give examples. This isn't about a specific distro, it's about the huge benefits of system packages and their toolchains in general. I'm not even really concerned with official packages and the official blessed package repos.
I've created hundreds of packages for OpenBSD and you just seem pretty gullible about the whole process.
You can't just write about the packaging process and hope people come forward to volunteer to do the actual work. There's a large amount of thankless sweat equity when you provide packages. People whine when things are broken and rarely send praise when it just works as it should.
If you really want to change the world, start with creating a Ruby apt repository for Debian/Ubuntu/whatever you use that people can actually deploy in production.
You can't just write about the packaging process and hope people come forward to volunteer to do the actual work. There's a large amount of thankless sweat equity when you provide packages. People whine when things are broken and rarely send praise when it just works as it should.
If you really want to change the world, start with creating a Ruby apt repository for Debian/Ubuntu/whatever you use that people can actually deploy in production.
For my money, here's what needs to happen (from a Ruby perspective; salt to taste):
1. Debian and Ubuntu stop packaging any Ruby libraries which aren't dependencies of an actual end-user application. If rubygems itself is a library dependency, package it but take away the gem command.
2. Make a public policy saying "If you want to do development, use RVM," and have metapackages with the dependencies for building each RVM ruby (so ruby-dev-mri, ruby-dev-jruby, and so on). Make sure that users know that if they think they need to run "gem install foo", then they need RVM.
It'd be nice for RVM to be able to build system packages, but I don't think it's necessary: there are already systems (nascent, but improving) for converting gems to system packages which handle dependencies nicely.
Is there anything wrong with this picture?
1. Debian and Ubuntu stop packaging any Ruby libraries which aren't dependencies of an actual end-user application. If rubygems itself is a library dependency, package it but take away the gem command.
2. Make a public policy saying "If you want to do development, use RVM," and have metapackages with the dependencies for building each RVM ruby (so ruby-dev-mri, ruby-dev-jruby, and so on). Make sure that users know that if they think they need to run "gem install foo", then they need RVM.
It'd be nice for RVM to be able to build system packages, but I don't think it's necessary: there are already systems (nascent, but improving) for converting gems to system packages which handle dependencies nicely.
Is there anything wrong with this picture?
RVM is great. But by design it's a solution to compiling software on every machine. This simply doesn't work for lots of sysadmins, especially with lots of machines - it means installing a complete build environment which opens up much more evil.
Well, yes. And that's the effective situation we've got at the moment.
What I'm proposing is a full acknowledgement of that: if you want to live on the bleeding edge, then that should be supported as well as possible, but a very clear delineation should be drawn between the responsibilities of the distribution and those of the developer/sysadmin. RVM does that very cleanly, even if you only use it with the system ruby.
What I'm proposing is a full acknowledgement of that: if you want to live on the bleeding edge, then that should be supported as well as possible, but a very clear delineation should be drawn between the responsibilities of the distribution and those of the developer/sysadmin. RVM does that very cleanly, even if you only use it with the system ruby.
"If rubygems itself is a library dependency, package it but take away the gem command."
Might be better to un-gem any required gems and package them with the app. Then add some code that fakes out any calls in the app to 'gem' or 'rubygems'.
OTOH, rubygems is part of ruby now, so leaving it out might be a problem.
Might be better to un-gem any required gems and package them with the app. Then add some code that fakes out any calls in the app to 'gem' or 'rubygems'.
OTOH, rubygems is part of ruby now, so leaving it out might be a problem.
If you bundle one gem with five different apps (as could easily happen with, say, rack), then any security updates to that gem mean that five distro security updates have to get pushed out. That's a lot more work than it needs to be.
As far as calls to the "gem" method go, I have in mind a distro-specific replacement which would check that the right system package is installed instead of the right gem. That would, I think, be handy for packagers.
Having rubygems the library being part of ruby is not something I can see a problem with. Having the gem command available is a problem because it overlaps so much with functionality which should be provided by the distro. That being said, ~/.gem is useful; maybe it should only be neutered for root/sudo.
As far as calls to the "gem" method go, I have in mind a distro-specific replacement which would check that the right system package is installed instead of the right gem. That would, I think, be handy for packagers.
Having rubygems the library being part of ruby is not something I can see a problem with. Having the gem command available is a problem because it overlaps so much with functionality which should be provided by the distro. That being said, ~/.gem is useful; maybe it should only be neutered for root/sudo.
One of the reasons software gets released in scripts rather than packages is because the root package installation mechanism is root accessible only. Most companies restrict root access and package installation is blocked by proxies. The combination of which means that scripts that can be downloaded separately and run are much easier for enterprise developers to use and get started with.
Until the average enterprise realises that packaging and installation needs to be done better we will always have the need for mechanisms that hack around the problem that system admins cause for the users of the servers.
Until the average enterprise realises that packaging and installation needs to be done better we will always have the need for mechanisms that hack around the problem that system admins cause for the users of the servers.
I agree, but developers have enough on their plate and knowing how to administer your workstation isn't the same as administrating a server.
That said, allowing user system wide installation could be possible. I think, at least under Linux, combining selinux and cgroup to provide sandboxed system wide user install could be possible but you'd possibly need to bypass classic unix directory ownership and permissions. That might require to much work or not even be impossible.
Anyone here with enough knowledge to confirm or refute my feeling ?
That said, allowing user system wide installation could be possible. I think, at least under Linux, combining selinux and cgroup to provide sandboxed system wide user install could be possible but you'd possibly need to bypass classic unix directory ownership and permissions. That might require to much work or not even be impossible.
Anyone here with enough knowledge to confirm or refute my feeling ?
Is there a decently written guide for creating ubuntu packages? The package maintainers guide confuses the heck out of me and it's way too long and disorganized.
https://wiki.ubuntu.com/PackagingGuide - a little bit better than Debian's official one. Unfortunately the more you do it, the more you actually need to refer to the Debian's guide and see why it was written that way...
I also recommend checking out https://wiki.ubuntu.com/PbuilderHowto - pbuilder is "the" way to build packages, which I unfortunately discovered very late.
I also recommend checking out https://wiki.ubuntu.com/PbuilderHowto - pbuilder is "the" way to build packages, which I unfortunately discovered very late.
To my knowledge no. I think this is one of the main problems in getting people to create system packages. I think this fosters the split between package maintainers and developers. It's not that the documentation is all bad, it's just not aimed at the audience I think it should be.
System packages just aren't what web savvy developers grew up on and the documentation feels out of step. But the capabilities and tools around system packages are awesome.
I might take a go a writing a simple guide, but I'd be more than happy to see other people have a go that know more than I do. Let me know if this sounds of interest.
System packages just aren't what web savvy developers grew up on and the documentation feels out of step. But the capabilities and tools around system packages are awesome.
I might take a go a writing a simple guide, but I'd be more than happy to see other people have a go that know more than I do. Let me know if this sounds of interest.
Theres a great programme called checkinstall that gets you 90% there. Rather than running 'make install' run 'checkinstall', or rather than 'sudo python installscript.pt', run 'sudo checkinstall python installscript.py'
Checkinstall is brilliant: http://checkinstall.izto.org/
It's also guilty of being unattractive to the developers I'd love to see use it. The site says "Recent News: Dec 26th, 2009" and then concentrates on a NOTE TO SLACKWARE 8.0 USERS.
Showing the hello world of checkinstall would be much better imho, and it's what developers from the web world expect.
It's also guilty of being unattractive to the developers I'd love to see use it. The site says "Recent News: Dec 26th, 2009" and then concentrates on a NOTE TO SLACKWARE 8.0 USERS.
Showing the hello world of checkinstall would be much better imho, and it's what developers from the web world expect.
Checkinstall of Python and VirtualEnv is the only sane way I've found to do python development and deployment when you're developing on Maverick and deploying to Lenny.
devs, esp polygot/startup devs, need sysadmin xp/knowledge.
That was for commercial software of course, but even for open source software, I'm not sure why it's the author's responsibility to go setup a dozen different VMs. That really is the package maintainer's job.