Better Package Management for python(nvie.com)
nvie.com
Better Package Management for python
http://nvie.com/posts/better-package-management/
5 comments
Agreed on pip-sync, but I find pip-compile to be a compelling idea (this is the first time I've heard of it).
Even small projects get cluttered requirements.txt files very fast, causing unneeded packages to spread like viruses from venv to venv, requiring coordination from the whole team to definitely eradicate, because one member's sloppy hygiene spreads the contagion back to everyone.
I don't know if pip-compile is the right answer to that problem, but it certainly seems to go in the right direction, providing living documentation of your direct requirements in the requirements.in file.
Even small projects get cluttered requirements.txt files very fast, causing unneeded packages to spread like viruses from venv to venv, requiring coordination from the whole team to definitely eradicate, because one member's sloppy hygiene spreads the contagion back to everyone.
I don't know if pip-compile is the right answer to that problem, but it certainly seems to go in the right direction, providing living documentation of your direct requirements in the requirements.in file.
I personally don't feel that pip-compile is that useful. All you can achieve with it can be achieved with a "pip install" + freeze in a clean venv. OK, error detection may have some use, but you still end with almost the same effort (packages are downloaded, parsing happens and you get an error).
IMHO we'd first need to standardize how dependencies are managed (requirements.txt, setup.py ...)and then build a solution off that. if you have requirements.txt-based setup, is easy to e.g. recursively download/parse to get the full list of dependencies.
The requirements.in file reminds me of automake.
IMHO we'd first need to standardize how dependencies are managed (requirements.txt, setup.py ...)and then build a solution off that. if you have requirements.txt-based setup, is easy to e.g. recursively download/parse to get the full list of dependencies.
The requirements.in file reminds me of automake.
Agreed that this is doable, although I would find rebuilding a virtualenv every time I wanted to change a dependency pretty annoying. Perhaps pip-compile is just a stopgap until a better solution comes along, but I think I'd find it helpful.
I'd be more for a "pip diff" in that case, to show me what are the differences between the current venv and the changed layout.
I don't really like the idea of consolidating dependencies locally because you'll end up with a big compiled requirements.txt which then you have to maintain yourself. IMHO it'll be a drag when you'll update a 3rd party dependency which has its own changed dependencies...
I don't really like the idea of consolidating dependencies locally because you'll end up with a big compiled requirements.txt which then you have to maintain yourself. IMHO it'll be a drag when you'll update a 3rd party dependency which has its own changed dependencies...
So are you suggesting that you don't track "meta"-dependencies (dependencies of dependencies)? I've been bitten by issues from this before, which I think is kind of the point of pip-compile.
I don't really see how it's much different from Gemfile.lock for Ruby or npm-shrinkwrap.json for Node.
I don't really see how it's much different from Gemfile.lock for Ruby or npm-shrinkwrap.json for Node.
The gemfile.lock (and I assume the node equivalent) are reactive things in the sense that they're generated for the final configuration (much like pip freeze).
I'm not really against of tracking dependencies recursively, but not for the final purpose of building a grand requirements file (which people will end up using and tweaking). I'm more interested if there are conflicts between different 3rd party libs (e.g. if one needs package==1.0.0 and the other needs package=1.2.0) and if they're not necessarily backwards compatible (e.g. v1.4 vs v1.6 of django).
Now, if you want to change the version of a 3rd party which, in turn, has altered dependencies, you'd need to clean up the venv anyway to be on the safe side. Currently I end up just rebuilding the venv and performing a pip install. On distribution, you could build an equivalent of a gemfile.lock via pip freeze (again). Hence, no real benefit of a pip-compile. Still, a pip diff (e.g. between 2 requirements.txt, tracking recursively dependencies) would be of more value imho, particularly if the devs would agree on a pattern/best practice.
There's also the matter of adding yet another file (requirements.in) to the setup framework (setup.py, metafiles...) and somewhat changing the meaning of requirements.txt.
I'm not really against of tracking dependencies recursively, but not for the final purpose of building a grand requirements file (which people will end up using and tweaking). I'm more interested if there are conflicts between different 3rd party libs (e.g. if one needs package==1.0.0 and the other needs package=1.2.0) and if they're not necessarily backwards compatible (e.g. v1.4 vs v1.6 of django).
Now, if you want to change the version of a 3rd party which, in turn, has altered dependencies, you'd need to clean up the venv anyway to be on the safe side. Currently I end up just rebuilding the venv and performing a pip install. On distribution, you could build an equivalent of a gemfile.lock via pip freeze (again). Hence, no real benefit of a pip-compile. Still, a pip diff (e.g. between 2 requirements.txt, tracking recursively dependencies) would be of more value imho, particularly if the devs would agree on a pattern/best practice.
There's also the matter of adding yet another file (requirements.in) to the setup framework (setup.py, metafiles...) and somewhat changing the meaning of requirements.txt.
Okay, that's a fair point. It just seems like Ruby and Node have sucessfully adopted the use of two separate files for dependency tracking, so it seems like it would work with Python as well.
I guess what you're suggesting is that requirements.txt should be the equivalent of a Gemfile and there should be a separate "requirements.lock" (or whatever) which tracks the output of pip freeze.
On top of that, I understand that you're suggesting that the generation of this lock file should be part of the process of setting up a virtualenv as opposed to a separate tool such as pip-compile.
If my interpretation is correct, then I totally agree. Although I still think pip-compile may be useful until such a tool exists :)
I guess what you're suggesting is that requirements.txt should be the equivalent of a Gemfile and there should be a separate "requirements.lock" (or whatever) which tracks the output of pip freeze.
On top of that, I understand that you're suggesting that the generation of this lock file should be part of the process of setting up a virtualenv as opposed to a separate tool such as pip-compile.
If my interpretation is correct, then I totally agree. Although I still think pip-compile may be useful until such a tool exists :)
There are times when I wish pip was slightly better, but it gets the job done pretty well.
Right at this moment, with Docker coming to the fore, it's easy to imagine a world where it doesn't matter anymore.
Right at this moment, with Docker coming to the fore, it's easy to imagine a world where it doesn't matter anymore.
Gems by themselves don't work right, but how about with bundler as well?
The sad part is that I end up having multiple requirements files anyway to match the different environments (test, production, development) -- see the django applications for this. IMHO (again) we'd need some sort of structured requirements for different environments.
If I install an app/package for development, then I'd need development dependencies (like test-related packages etc.). I'm not supposed to know/care what the third party code needs for testing; just when I start testing the relevant code should "just work". Also, if I'm in production the testing-related packages should not pollute my environment...
If I install an app/package for development, then I'd need development dependencies (like test-related packages etc.). I'm not supposed to know/care what the third party code needs for testing; just when I start testing the relevant code should "just work". Also, if I'm in production the testing-related packages should not pollute my environment...
Would it be possible to create a solid, language independent package manager so that it's not necessary to create a new one for each language/community?
I'd love to see Pip, Bundler/Rubygems, NPM, Cabal, Maven, etc. replaced by a single tool (or maybe two: dependency management and package installation).
I'd love to see Pip, Bundler/Rubygems, NPM, Cabal, Maven, etc. replaced by a single tool (or maybe two: dependency management and package installation).
Different languages work different ways, it would be really difficult.imho,I would not use a package manager that is not written in the language I use.So what language should it be ? Even if someone comes up with a generic protocol, there is no way it would answer all needs.
The language doesn't matter if it's portable. One giant benefit of a single tool is in projects using multiple languages. Using multiple similar tools side by side is a pain. Even for a simple web app with JS it's already necessary to use two different package managers (client JS + server side code).
I'm not savvy enough to understand the complications - why would it be so difficult?
Usually people code in one language (python in this case). learning another language for package management would require extra effort. Also, setting up the package is something you do once per project and forget about it (ok, you update it but far less frequent than your real code).
To be more exact, how many times have you customised your setup.py file? edited its dependencies and so on..? Imagine doing that in a totally different system (like debian packaging).
You can even now manage your dependencies and install scripts with something like automake or a homemade solution. Not really portable but if you don't care about your code being a module part of a bigger system...
To be more exact, how many times have you customised your setup.py file? edited its dependencies and so on..? Imagine doing that in a totally different system (like debian packaging).
You can even now manage your dependencies and install scripts with something like automake or a homemade solution. Not really portable but if you don't care about your code being a module part of a bigger system...
I think rvm guys are trying something similar in their next iteration of RVM
I don't see why people keep on with virtualenv when all you need to create a sandbox is to set the PYTHONPATH environment variable...?
Because it's easier, and why not?
I have been slightly disappointed after setting up my project in a virtualenv and upgrading some stuff on my system when some virtualenv packages broke. I thought that is what virtualenv was supposed to prevent.
(Maybe I need to read the entire virtualenv documentation, to have a solid idea of how it works, and what it does exactly, but hen that will take time, and the PYTHONPATH option starts to look more appealing, as at least I do know how environmental variables work in Linux).
(Maybe I need to read the entire virtualenv documentation, to have a solid idea of how it works, and what it does exactly, but hen that will take time, and the PYTHONPATH option starts to look more appealing, as at least I do know how environmental variables work in Linux).
That's odd, in the many years I've been using venv for, I can't remember many breakages, and none that weren't fixed by either just running "virtualenv env" again or just removing and recreating it (a two-minute affair).
How can it be easier to install more packages then just setting an environment variable?
In this case I don't think either tool is needed - pip-sync is really trying to avoid tearing down and rebuilding cleanly a virtual env - which is the whole point of virtual environments anyway.
Anyway - yes we need to fix python package management. No there is not a clear solution. But then apt-get, Tuesday updates and gems don't work "right" either.