A hygienic Python setup for Linux, Mac, and WSL

Python dependency management is known to be bad. Over time, I’ve decided the only way I’m willing to live is to push my Python environment hygiene to the max. As I’ve recommended my setup to a lot of people, I figured I should write it up as a reference.

Tenets

This is what I aim to accomplish in my Python setup. You don’t have to agree with these tenets, and if you don’t, feel free to ignore any of the advice that follows as it flows from them.

  • Never install anything in system python installs

Implementation

  • Use pyenv to setup and manage multiple python installations (including anaconda)

pipenv and poetry

Why use a virtualenv management tool like pipenv or poetry? Each virtualenv gets a file defining its state (Pipfile or pyproject.toml), making them easy to manage and recreate, and it’s then that much easier for each project to have its own managed virtualenv.

Which to choose? It’s mostly your choice for which to use; I actually use both. I think they are good at different things, and good enough at those separate things to make it worth knowing and using both.

pipenv is better for specifying the environment for an application (e.g., a thing you deploy), which should have a very specific known python setup. It also lets you have .env files for project environment variables, which tend to be useful for applications (e.g., its deployment environment) and less so for libraries.

poetry is better for building libraries and CLI tools, which need to get put in PyPI and be flexible about the Python environment they end up in. Poetry’s PEP 518-compliant pyproject.toml is way easier than writing a setup.py file (which is still required if you’re using pipenv). It allows multiple virtualenvs/python versions (important for libraries). And, it can build wheels and publish to PyPI.

Both tools allow you to specify that the virtualenv directory for a project should go in the project directory itself (as .venv/), rather than in the central store of virtualenvs. I like this a lot, as it makes that virtualenv easier to find+destroy if it gets corrupted, and also gets inherently cleaned up if you remove the project directory. This feature is enabled with environment variables; instructions are included below.

pipx

pipx installs each package into its own virtualenv and then links the executables so they are on your $PATH

Isolated installs

To get isolated installations of these tools, I turn to homebrew, even on Linux (it’s good now!). As mentioned above, pip-installing requires selecting a pyenv-installed Python version to install into. You could pipx-install, but how do you install pipx? Additionally, brew formulas are kept much more up-to-date than most OS package managers (e.g., apt).

pyenv, pipenv, and pipx should be brew installed, but I recommend using poetry’s only standalone installation (for now), because the brew formula is currently broken.

homebrew

Website

mac

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

linux

sudo apt-get install build-essential curl file git/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >> ~/.bashrc

pyenv

Website

Note: currently on linux, you should patch brew-installed pyenv for readline support (see this GitHub issue) so the REPL gets tab completion.

brew install pyenv# get latest versionsPY27=$(pyenv install --list | grep "^\s*2.7" | tr -d ' ' | tail -1)
PY35=$(pyenv install --list | grep "^\s*3.5" | tr -d ' ' | tail -1)
PY36=$(pyenv install --list | grep "^\s*3.6" | tr -d ' ' | tail -1)
PY37=$(pyenv install --list | grep "^\s*3.7" | tr -d ' ' | tail -1)
PY38=$(pyenv install --list | grep "^\s*3.8" | tr -d ' ' | tail -1)
# set global python version
pyenv install $PY38
pyenv global $PY38

You can also configure pip to refuse to install outside of a virtualenv, see instructions here (thanks to Theron Luhn for this tip).

pipenv

Website

brew install pipenv# Optional, if you want virtualenvs for a projects to go inside the project rather than centrally. Then when you delete a project, the virtualenv doesn't stick around. I like this a lot!
# YOUR_DOTFILE should be .profile on mac, and on linux, your .bashrc or whatever for your shell
echo "export PIPENV_VENV_IN_PROJECT=1" >> YOUR_DOTFILE

poetry

Website

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
poetry completions bash | sudo tee /etc/bash_completion.d/poetry.bash-completion > /dev/null
# Optional, if you want virtualenvs for a projects to go inside the project rather than centrally. Then when you delete a project, the virtualenv doesn't stick around.
# YOUR_DOTFILE should be .profile on mac, and on linux, your .bashrc or whatever for your shell
echo "export POETRY_VIRTUALENVS_IN_PROJECT=1" >> YOUR_DOTFILE

Notes

This is based originally on Jacob Kaplan-Moss’s recommendations in this blog post.

If you think this is excessive, that’s totally acceptable, but you can keep it to yourself. I have found that this works for me, and I’m sharing it in case other people find it useful.

If you think that Linux system package managers should be used in place of homebrew, you can continue to use them and I will not try to stop you.

If you think piping curl’d content from GitHub into your shell is a bad idea, I agree in principle but I think it’s an acceptable risk for these programs.

If you’re on Windows and don’t want to use WSL (fair, but give it a try!), I’m sorry that I don’t have something for you.

If you think Docker is a better answer, you might be right! I have not gone far enough down that path to have a strong opinion about it, but it seems plausible, though it seems to get fiddly around IDE/GUI integration and integration with common resources from your host (e.g., dotfiles). I’m also unsure about whether I’d want to go with the traditional “container-as-process” model or the “container-as-vm” model of LXD.

If you think I have not gone far enough and there are ways to make this even more hygienic, please let me know here or on Twitter.

Cloud Robotics Research Scientist at @iRobot

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store