Using pyenv for easy environments

This setup works really well to manage Python virtual environments when working over many projects.

If you want to:

• Automatically switch to your virtualenv when entering a project folder
• Have the correct version of Python installed with a single command
• Be able to recreate your virtualenv without thinking about it

Then read on.

History

I have various projects running on physical servers and virtual servers, and I use various versions of Ubuntu and MacOS for development.

Trying to keep my setup as simple as possible, I kept to using simple venvs in the project folder for my python projects. However, this had several problems:

• I had to use deadsnakes to install older python versions some projects.
• I had to switch to the virtualenv manually (not a big problem, since I had a ve alias to do that, but still)
• If I had to rebuild the virtualenv, I had to sort out which python version I needed.

I had a look at virtualenvwrapper and some other things over the years, but managing my virtualenvs in the same folder seemed easier.

Enter pyenv and pyenv-virtualenv.

Installation

Pyenv and pyenv-virtualenv

I’m not going into the details of how to install those, apart from saying that I followed the instructions for installing them:

• pyenv
• pyenv-virtualenv

That gives you the ability to install any version of Python and, using some magic in your dotfiles and a .python-version file in your project root, to automatically switch to the correct virtualenv.

rebuild_venv and remove_venv

The above installation solves some of the problems, but still required me to keep too much context in my head.

Enter some shell functions which you can put in your .bashrc or .bash_aliases:

 1rebuild_venv () {
 2    # Get the virtualenv name
 3    PYEV=`cat .python-version|xargs basename`
 4    # Get the python version
 5    PYEP=`cat .python-version|xargs dirname|xargs dirname`
 6    # Delete the virtualenv
 7    rm -rf ~/.pyenv/versions/$PYEP/envs/$PYEV
 8    rm -rf ~/.pyenv/versions/$PYEV
 9    # Install python version, skip if already installed
10    pyenv install $PYEP -s
11    # Recreate the virtualenv
12    pyenv virtualenv $PYEP $PYEV
13    pip install --upgrade pip setuptools wheel
14}
15
16remove_venv () {
17    # Get the virtualenv name
18    PYEV=`cat .python-version|xargs basename`
19    # Get the python version
20    PYEP=`cat .python-version|xargs dirname|xargs dirname`
21    # Delete the virtualenv
22    rm -rf ~/.pyenv/versions/$PYEP/envs/$PYEV
23    rm -rf ~/.pyenv/versions/$PYEV
24}
25
26export PYENV_ROOT="$HOME/.pyenv"
27export PATH="$PYENV_ROOT/bin:$PATH"
28eval "$(pyenv init --path)"
29eval "$(pyenv virtualenv-init -)"

How does it work?

pyenv relies on a file called .python-version in your project root to decide which virtualenv to use. If your virtualenv name is “testproject”, this could be as simple as testproject. pyenv-virtualenv simply looks up ~/.pyenv/versions/testproject and inserts a shim so that python refers to that virtualenv python.

Here’s the trick: for every virtualenv, a more specific name with the python version included is also created. Say you are using python 3.9.9, this would be .pyenv/versions/3.9.9/envs/testproject.

Using this, we can pin both the virtualenv name and python version in one go. This is what the shell functions do.

To use it, your .python-version file would look like this:

13.9.9/envs/testproject

The rebuild_pyenv function is designed to be run in the root of your project. It checks for a .python-version file in the current folder and gets the virtualenv from that. It will then delete the virtualenv if it exists already and recreate it, downloading the specified Python version if needed. As part of this process, it also upgrades pip.

How well does it work?

Solves 96% of use cases with no effort

For me, this makes it so much easier to work with virtualenvs. Simply cd into a project folder, and if the virtualenv is not automatically activated, run rebuild_pyenv to make it work. A full scorched-earth rebuild is typically rebuild_pyenv;make init;make test. A clean checkout in a temporary folder already activates the correct virtualenv, and if I want to change it, I just remove or edit the .python-version file and run rebuild_pyenv again.

Plays well with other tools

Used in combination with tools like pip-compile for version-pinning of packages, this provides a solid way to make sure everything is repeatable.

Yes, I know that pipenv, tox and some other python package tools allow you to specify Python versions and might provide some of this functionality, but this method removes any cognitive load regarding package management. One simple command to rule them all.

Plays well with other developers

Simply gitignore the file. Other developers on your project can then decide on their own preferred virtualenv method, including what python version to use.

Conclusion

I could have used a different way to specify the python version, but this is easiest to parse.