Show HN: Vex, a new way to run things in Python's virtualenvs(pypi.python.org)
pypi.python.org
Show HN: Vex, a new way to run things in Python's virtualenvs
https://pypi.python.org/pypi/vex
11 comments
> everyone finds 'workon' invaluable, a position I do understand.
At first I wanted to praise virtualfish [1] and its ability to automatically activate a virtualenv when I cd into it, but then I realized that vex's way does indeed make more sense. What do we do in a virtualenv? We basically just run pip to do various things, or run the program (in case it's an application and not a library) or test suite.
So yeah, I think I'm gonna switch. The added portability is a nice bonus.
[1] https://github.com/adambrenecki/virtualfish
At first I wanted to praise virtualfish [1] and its ability to automatically activate a virtualenv when I cd into it, but then I realized that vex's way does indeed make more sense. What do we do in a virtualenv? We basically just run pip to do various things, or run the program (in case it's an application and not a library) or test suite.
So yeah, I think I'm gonna switch. The added portability is a nice bonus.
[1] https://github.com/adambrenecki/virtualfish
Am I the only one who activates the virtualenv by hand? I don't understand what workon and other things provide. These days, I just have a single virtualenv that gets sourced when my shell starts up, all my projects run in virtual machines (I mostly work with Django). Projects that conflict with others get their own virtualenv, but sourcing it once at the start of the day is still not that much hassle.
You aren't, I do it all the time. But it does get slightly tedious to type out the whole path. (when switching virtualenvs frequently, unlike your use case)
What I use that is analogous to your single virtualenv sourced at shell startup, is pip install --user. I tend to use that for command line tools like style checkers.
What I use that is analogous to your single virtualenv sourced at shell startup, is pip install --user. I tend to use that for command line tools like style checkers.
Awesome, thank you for trying this. Please make a github issue or email me if you have any suggestions informed by the experience :)
edit: That one feature reminded me of Kenneth Reitz's autoenv (https://github.com/kennethreitz/autoenv).
I can't do anything to futz with cd, but it could be possible to reverse that and cd to a project directory when a virtualenv is activated, I am just not sure yet mechanically the right way to choose which one. Worth considering.
edit: That one feature reminded me of Kenneth Reitz's autoenv (https://github.com/kennethreitz/autoenv).
I can't do anything to futz with cd, but it could be possible to reverse that and cd to a project directory when a virtualenv is activated, I am just not sure yet mechanically the right way to choose which one. Worth considering.
Well, I use the postactivate hook to set environment variables (for convenience) and connect to project-specific tmux session.
This solution has a lot of flaws, though: it isn't robust and doesn't make sense for non-Python-based projects (my work is not limited to Django). In fact, I'm working on switching to dedicated VMs for my projects—looking for a nice solution currently. I suspect it may involve Vagrant, Docker, perhaps Chef.
export PROJECT=`cat $VIRTUAL_ENV/.project`
export SRC=$PROJECT/src
[ -z $TMUX ] && cdproject && envopen
This solution has a lot of flaws, though: it isn't robust and doesn't make sense for non-Python-based projects (my work is not limited to Django). In fact, I'm working on switching to dedicated VMs for my projects—looking for a nice solution currently. I suspect it may involve Vagrant, Docker, perhaps Chef.
> pydoc still doesn't work
Are you using `pydoc` on the command line? If so, try pdoc: https://github.com/BurntSushi/pdoc. I wrote it as a tool to replace epydoc (strictly automatic documentation), but it works on the command line just like pydoc. (It can also generate HTML documentation with cross module linking that isn't crap. See an example here: http://pdoc.burntsushi.net/nfldb)
Are you using `pydoc` on the command line? If so, try pdoc: https://github.com/BurntSushi/pdoc. I wrote it as a tool to replace epydoc (strictly automatic documentation), but it works on the command line just like pydoc. (It can also generate HTML documentation with cross module linking that isn't crap. See an example here: http://pdoc.burntsushi.net/nfldb)
I do sometimes, more as a matter of muscle memory than volition. I'm going to make a try to use your tool as a replacement for a while and see if I can generate any issues for you :)
However, I still would like to eventually unbreak pydoc for users who might have bad habits like I do.
The issue is that virtualenv itself had an opportunity to fix this (with multiple patches submitted IIRC) and they decided to punt by implementing it as yet another shell function. So for me to unpunt without coupling to the shell, I pretty much have to dump a script into bin/. I guess maybe not that many people use pydoc anyway?
However, I still would like to eventually unbreak pydoc for users who might have bad habits like I do.
The issue is that virtualenv itself had an opportunity to fix this (with multiple patches submitted IIRC) and they decided to punt by implementing it as yet another shell function. So for me to unpunt without coupling to the shell, I pretty much have to dump a script into bin/. I guess maybe not that many people use pydoc anyway?
> I guess maybe not that many people use pydoc anyway?
That would be my guess. IMO, pydoc isn't very discriminating with what it produces. The HTML is pretty ugly and the command line version just kind of vomits everything in the module to stdout. My idea with pdoc was to try and discover a module's public interface, and then display that.
That would be my guess. IMO, pydoc isn't very discriminating with what it produces. The HTML is pretty ugly and the command line version just kind of vomits everything in the module to stdout. My idea with pdoc was to try and discover a module's public interface, and then display that.
I've never bothered with virtualenvwrapper - I create virtualenvs alongside my projects, and call `$virtualenv/bin/python` or `$virtualenv/bin/pip` etc. directly. Sometimes I activate the virtualenv, sometimes I don't.
That's what I do too. I mostly activate them by hand nowadays, just to avoid calling python. Fish is smart enough that it knows that, if I type a period in a directory that contains a virtualenv, I want to activate it.
I'm very happy more people are coming around to this idea, despite multiple independent discoveries. :)
Here's another project that attempts something similar: https://github.com/berdario/invewrapper
Here's a sortof-blog-post and some more discussion about the superiority of modifying a subshell rather than sourcing a bunch of janky shell-functions that modify your current shell: https://gist.github.com/datagrok/2199506
Here's another project that attempts something similar: https://github.com/berdario/invewrapper
Here's a sortof-blog-post and some more discussion about the superiority of modifying a subshell rather than sourcing a bunch of janky shell-functions that modify your current shell: https://gist.github.com/datagrok/2199506
I did not know about invewrapper, thanks. pew is another decent name.
edit: after looking at the doc, my impression is that invewrapper seems to follow virtualenvwrapper's design more closely with a large number of subcommands with names similar to virtualenvwrapper's, most of virtualenvwrapper's options and features (except hooks and maybe some of the project stuff). I would describe the options as "comprehensive." I specifically wanted something with a dramatically simpler interface and feature set.
invewrapper also seems primarily or only intended to run a shell. I get personal use out of running arbitrary commands under vex as if it were sudo or something.
It is an improvement not to modify the current environment and couple tightly to specific shells either way, though.
One thing I do envy is the PowerShell prompt ;)
edit: after looking at the doc, my impression is that invewrapper seems to follow virtualenvwrapper's design more closely with a large number of subcommands with names similar to virtualenvwrapper's, most of virtualenvwrapper's options and features (except hooks and maybe some of the project stuff). I would describe the options as "comprehensive." I specifically wanted something with a dramatically simpler interface and feature set.
invewrapper also seems primarily or only intended to run a shell. I get personal use out of running arbitrary commands under vex as if it were sudo or something.
It is an improvement not to modify the current environment and couple tightly to specific shells either way, though.
One thing I do envy is the PowerShell prompt ;)
(invewrapper/pew author here)
I just discovered your project, so I haven't tried it yet, but since the approach seems similar, it should work just fine on Windows (maybe with some tweaks)... and you can just add the Powershell prompt to your docs, no need to envy it :)
I agree that pew is intended as a virtualenvwrapper replacement, obviously you can get something similar to `vex env cmd` with `pew in env cmd`...
Unfortunately I've neglected a little bit the project lately: the most serious thing I want to do is rewrite the test suite, to be able to run it on windows.
Since you mention PowerShell maybe you use it quite often? I'd like to get some feedback on Windows (I added Windows support out of completeness, but I seldom use that OS), since I don't know anyone who uses pew on windows :)
BTW, I discovered some new tools to manage environments in the last year:
pyenv (I actually already knew rbenv), modules[1] and I even started to use Nix
These have not the same scope, and in fact I appreciate the latter 2 idea of using the same system to manage dependencies and versions for all different kind of tools. I don't have an opinion yet on what's the correct way to do these things so, in the meanwhile, even tools that are python-specific (like ours) have a niche to fill
[1] https://archive.fosdem.org/2014/schedule/event/hpc_devroom_e...
I just discovered your project, so I haven't tried it yet, but since the approach seems similar, it should work just fine on Windows (maybe with some tweaks)... and you can just add the Powershell prompt to your docs, no need to envy it :)
I agree that pew is intended as a virtualenvwrapper replacement, obviously you can get something similar to `vex env cmd` with `pew in env cmd`...
Unfortunately I've neglected a little bit the project lately: the most serious thing I want to do is rewrite the test suite, to be able to run it on windows.
Since you mention PowerShell maybe you use it quite often? I'd like to get some feedback on Windows (I added Windows support out of completeness, but I seldom use that OS), since I don't know anyone who uses pew on windows :)
BTW, I discovered some new tools to manage environments in the last year:
pyenv (I actually already knew rbenv), modules[1] and I even started to use Nix
These have not the same scope, and in fact I appreciate the latter 2 idea of using the same system to manage dependencies and versions for all different kind of tools. I don't have an opinion yet on what's the correct way to do these things so, in the meanwhile, even tools that are python-specific (like ours) have a niche to fill
[1] https://archive.fosdem.org/2014/schedule/event/hpc_devroom_e...
Ha. I always thought something was unnecessarily awkward about virtualenv, but I couldn't quite put my finger on it. Virtualenvwrapper was an improvement in many ways, but its lack of portability could be a pain. Reading the docs, Vex makes perfect sense. We should use virtualenv basically the same way we sudo. It's kind of amazing it took this long for someone to do it. Thanks!
Vex seems very convenient, but it is also possible to avoid the "magic" with just standard virtualenv.
If you provide the full path to the python binary in the virtualenv, it executes with the virtualenv loaded:
If you provide the full path to the python binary in the virtualenv, it executes with the virtualenv loaded:
/home/acjohnson/virtualenvs/foo/bin/python myscript.py
Similarly, anything you've installed in that virtualenv that installs itself in a bin folder (pip, mercurial, django-admin.py, etc) will be in that bin folder, and calling them from that directory will execute them in the context of that virtualenv as well.
Fully agreed, and the README for vex suggests this if you are worried about overhead from the wrapper especially for non-interactive purposes where you are repeatedly incurring python startup.
I usually do this just by cd'ing to the project directory, which has the virtualenv inside it and running './venv/bin/python'.
That eliminates the possibility that I'll accidentally run the wrong version of python - e.g. a system python or a python from another environment that didn't properly deactivate.
Most other issues are easy to deal with (e.g. non-existent virtualenv, being in the wrong folder), but the issue of accidentally using the wrong environment is awful because it can manifest itself with subtle bugs or weird exceptions rather than an obvious error.
That eliminates the possibility that I'll accidentally run the wrong version of python - e.g. a system python or a python from another environment that didn't properly deactivate.
Most other issues are easy to deal with (e.g. non-existent virtualenv, being in the wrong folder), but the issue of accidentally using the wrong environment is awful because it can manifest itself with subtle bugs or weird exceptions rather than an obvious error.
I mostly do this, along with a little hack for certain commands I need, such as mercurial: I have a virtualenv under ~/opt/venv/py-2.7-misc, and make a symbolic link from ~/opt/venv/py-2.7-misc/bin to ~/opt/pybin -- and then my .bashrc checks for a ~/pybin and adds it to the PATH if it exists ( [[ -d "${HOME}/pybin" ]] ...).
So I can just go "pip install -u mercurial", or "pip install <some useful utility>" without worrying too much. And (my) "hg" is in my PATH -- and I can still run stuff in custom virtualenvs on a project basis, just as you say, by specifying the full path (./proj/venv/bin/python ...).
I've never understand why people use anything more complicated.
I do see occasions where activate is useful -- certain packages, I think scipy and pygame are among them -- are not only rather difficult to install via pip, but also tend to be very picky about their environment. Normally I get by by just installing those (and other packages with strong c/c++ library dependencies) via the package manager though.
So I can just go "pip install -u mercurial", or "pip install <some useful utility>" without worrying too much. And (my) "hg" is in my PATH -- and I can still run stuff in custom virtualenvs on a project basis, just as you say, by specifying the full path (./proj/venv/bin/python ...).
I've never understand why people use anything more complicated.
I do see occasions where activate is useful -- certain packages, I think scipy and pygame are among them -- are not only rather difficult to install via pip, but also tend to be very picky about their environment. Normally I get by by just installing those (and other packages with strong c/c++ library dependencies) via the package manager though.
This is cool, I like tools that focus on one thing.
Just like the author, I never used all virtualenvwrapper's features and was always happy to create and delete VEs on my own. The only thing missing from standard virtualenv workflow is easy way to activate the VE; so I wrote that on my own and been using it since ever. It's trivially small shell script but it does the work well, you can check it out at pypi:
https://pypi.python.org/pypi/ave/
Just like the author, I never used all virtualenvwrapper's features and was always happy to create and delete VEs on my own. The only thing missing from standard virtualenv workflow is easy way to activate the VE; so I wrote that on my own and been using it since ever. It's trivially small shell script but it does the work well, you can check it out at pypi:
https://pypi.python.org/pypi/ave/
What is truly horrible is that I wrote my shell function, named it ave, thought it was a great name and then months later (edit: after I released vex) realized you had written a package named ave. Oh well.. :)
This is also worth checking out if you just want 'workon' type functionality without the fuss.
This is also worth checking out if you just want 'workon' type functionality without the fuss.
I guess I was faster picking ave :) but vex is good too.
I like this. I've used both virtualenvwrapper and pew before, but they both have lots of features I don't use. Using this plus some small shell functions like the below to create, list and delete virtualenvs is enough for me.
function mkvirtualenv {
virtualenv "$HOME/.virtualenvs/$1"
if [ $# -eq 2 ]; then
vex "$1" pip install -r "$2"
fi
}
function lsvirtualenv {
ls "$HOME/.virtualenvs"
}
function rmvirtualenv {
rm -rf "$HOME/.virtualenvs/$1"
}It looks like there is a lot of demand for --make/--remove options to vex so I will do those (and that should nearly approach the end of the new features I will do on vex).
Probably good to do some error checking on the arguments to mkvirtualenv and rmvirtualenv (sleepily hitting enter too early, etc.)
Probably good to do some error checking on the arguments to mkvirtualenv and rmvirtualenv (sleepily hitting enter too early, etc.)
Looks good.
I currently use http://tudb.org/articles/2014/03/31/vpython/ for pretty much all of my python projects (also, i am the author of vpython).
I currently use http://tudb.org/articles/2014/03/31/vpython/ for pretty much all of my python projects (also, i am the author of vpython).
Well huh.
> Vpython is a tiny(-ish) bash script to help with your day to day virtualenv needs.
> * You don't have to worry about sourcing the activate script.
> * You don't have to point to your virtualenv path.
> just use vpython instead of python to invoke your scripts.
That's basically why I wrote vpython: I'm the author of this vpython[1], which isn't the same as yours, nor what the article is. (I suspect mine is a little more immature, as I've not worked on it for as long.) I guess similar needs beget similar utilities.
> Vpython is a tiny(-ish) bash script to help with your day to day virtualenv needs.
> * You don't have to worry about sourcing the activate script.
> * You don't have to point to your virtualenv path.
> just use vpython instead of python to invoke your scripts.
That's basically why I wrote vpython: I'm the author of this vpython[1], which isn't the same as yours, nor what the article is. (I suspect mine is a little more immature, as I've not worked on it for as long.) I guess similar needs beget similar utilities.
[1]: https://github.com/thanatos/vpythonHa. How fun :)
We take different aproaches though.
Repo link for reference: https://github.com/tbug/vpython
Mine expects you to know virtualenv and how it works, and also that you explicitly install it. Same goes for dependencies.
My needs where to avoid sourcing the virtualenv, so i wrote the tool to fix just that. Also, i built mine to resolve symlinks and be usable as the shebang in your python scripts, which is pretty neat when writing command line tools.
We take different aproaches though.
Repo link for reference: https://github.com/tbug/vpython
Mine expects you to know virtualenv and how it works, and also that you explicitly install it. Same goes for dependencies.
My needs where to avoid sourcing the virtualenv, so i wrote the tool to fix just that. Also, i built mine to resolve symlinks and be usable as the shebang in your python scripts, which is pretty neat when writing command line tools.
Ah, discovering it from the directory. I considered this in analogy with the way vagrant works.
Downside is, ofc, that compared to just running something inside a virtualenv shell, vpython takes forever (up to 100ms-ish on my machine) to find the virtualenv, and set the correct env variables.
Most of my projects are long running though (webservers, number crunchers, etc), so the extra miliseconds it takes to load is really not that important.
Most of my projects are long running though (webservers, number crunchers, etc), so the extra miliseconds it takes to load is really not that important.
one of the things I do in virtualenvwrapper is cdsitepackages to do some patch/workaround on the installed libraries. Not a lot though but its a nifty command to have.
This looks great. Any way to have a system-wide .vexrc?
Yes, there CAN be a way :) but would you do me a favor and write a spec for how you want that to work and jam that in a github issue? https://github.com/sashahart/vex/issues
Cool! That's the tool I always need!
virtualenvwrapper is very powerful with all kinds of hooks but I just never used all that power and needed things like speed more. virtualenvwrapper provided a lazy loader but that would have various problems from time to time as you'll see on the issue tracker.
So then for a long time I just ran 'virtualenv' and source wherever/bin/activate and found that I really didn't miss virtualenvwrapper all that much, which was a surprise because everyone finds 'workon' invaluable, a position I do understand.
Later my wife taught herself virtualenv and then wrote like 5 lines of shell to do the same thing as virtualenvwrapper (the classic environment-modifying thing) which I found hilarious, and I wrote a version of that in zsh with a little more error checking to make me happy and just jammed it in my dotfiles. Then after some months of using that I decided this is nice enough to publish so I worked on rewriting it as portable shell functions with more error checking and was going to put it on PyPI.
Toward the end of writing that I was feeling pretty finished, when some last detail that I forget made me hacked off about modifying the current shell environment, so on a lark I wrote a prototype of the new idea of only modifying a spawned process's environment in about an hour (most of which was learning further about portable shell) and once it was working I looked at implementing more error checking for all the weird conditions that may arise in shell and I said, you know what, I have no reason not to do this in Python and totally cut the dependency on which shell is being used, who is going to use this without Python anyway?
Then I ended up figuring out how to do prompts and completion and pretty much had a publishable project, which was surprising.
It's still faster to directly run the virtualenv's Python or to use your own small shell function but while YMMV, for my uses virtualenvwrapper is a lot of code I will never use and I use vex every day.
Highlights of my todos: pydoc still doesn't work (the reasons for this are lame) and probably it will be nice to implement --make/--remove to substitute mkvirtualenv/rmvirtualenv (which would also let me dump a pydoc script in virtualenv's bin/ or Script\ dir to fix that problem...)