A hygienic Python setup for Linux, Mac, and WSL

Python dependency management is known to be bad. Over time, I’ve decided the only way I’m willing to live is to push my Python environment hygiene to the max. As I’ve recommended my setup to a lot of people, I figured I should write it up as a reference.

Tenets

This is what I aim to accomplish in my Python setup. You don’t have to agree with these tenets, and if you don’t, feel free to ignore any of the advice that follows as it flows from them.

  • Always use a virtualenv
  • virtualenvs are better when their state is managed (vs. direct pip installs into them), because then you can recreate them at will
  • CLI tools written in python shouldn’t be treated like python packages
  • Don’t pip-install tools that work across python versions into a specific python version (give them their own isolated install)

Implementation

  • Use pyenv to setup and manage multiple python installations (including anaconda)
  • Use pipenv and/or poetry to manage virtualenvs
  • Use pipx to install CLI tools written in python
  • Instead of pip installing (or pipx-installing) these tools, get isolated installs for each. pipenv and pipx can be installed with pip, but which pyenv python version would you install them in?
  • With this setup, nothing touches system python, and nothing ever actually gets pip installed into the pyenv python installs directly

pipenv and poetry

Why use a virtualenv management tool like pipenv or poetry? Each virtualenv gets a file defining its state (Pipfile or pyproject.toml), making them easy to manage and recreate, and it’s then that much easier for each project to have its own managed virtualenv.

pipx

pipx installs each package into its own virtualenv and then links the executables so they are on your $PATH

Isolated installs

To get isolated installations of these tools, I turn to homebrew, even on Linux (it’s good now!). As mentioned above, pip-installing requires selecting a pyenv-installed Python version to install into. You could pipx-install, but how do you install pipx? Additionally, brew formulas are kept much more up-to-date than most OS package managers (e.g., apt).

homebrew

Website

mac

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

linux

sudo apt-get install build-essential curl file git/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >> ~/.bashrc

pyenv

Website

brew install pyenv# get latest versionsPY27=$(pyenv install --list | grep "^\s*2.7" | tr -d ' ' | tail -1)
PY35=$(pyenv install --list | grep "^\s*3.5" | tr -d ' ' | tail -1)
PY36=$(pyenv install --list | grep "^\s*3.6" | tr -d ' ' | tail -1)
PY37=$(pyenv install --list | grep "^\s*3.7" | tr -d ' ' | tail -1)
PY38=$(pyenv install --list | grep "^\s*3.8" | tr -d ' ' | tail -1)
# set global python version
pyenv install $PY38
pyenv global $PY38

pipenv

Website

brew install pipenv# Optional, if you want virtualenvs for a projects to go inside the project rather than centrally. Then when you delete a project, the virtualenv doesn't stick around. I like this a lot!
# YOUR_DOTFILE should be .profile on mac, and on linux, your .bashrc or whatever for your shell
echo "export PIPENV_VENV_IN_PROJECT=1" >> YOUR_DOTFILE

poetry

Website

curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
poetry completions bash | sudo tee /etc/bash_completion.d/poetry.bash-completion > /dev/null
# Optional, if you want virtualenvs for a projects to go inside the project rather than centrally. Then when you delete a project, the virtualenv doesn't stick around.
# YOUR_DOTFILE should be .profile on mac, and on linux, your .bashrc or whatever for your shell
echo "export POETRY_VIRTUALENVS_IN_PROJECT=1" >> YOUR_DOTFILE

Notes

This is based originally on Jacob Kaplan-Moss’s recommendations in this blog post.

Cloud Robotics Research Scientist at @iRobot