Python dependency management is known to be bad. Over time, I’ve decided the only way I’m willing to live is to push my Python environment hygiene to the max. As I’ve recommended my setup to a lot of people, I figured I should write it up as a reference.
Tenets
This is what I aim to accomplish in my Python setup. You don’t have to agree with these tenets, and if you don’t, feel free to ignore any of the advice that follows as it flows from them.
- Never install anything in system python installs
- Always use a virtualenv
- virtualenvs are better when their state is managed (vs. direct pip installs into them), because then you can recreate them at will
- CLI tools written in python shouldn’t be treated like python packages
- Don’t pip-install tools that work across python versions into a specific python version (give them their own isolated install)
Implementation
- Use pyenv to setup and manage multiple python installations (including anaconda)
- Use pipenv and/or poetry to manage virtualenvs
- Use pipx to install CLI tools written in python
- Instead of pip installing (or pipx-installing) these tools, get isolated installs for each. pipenv and pipx can be installed with pip, but which pyenv python version would you install them in?
- With this setup, nothing touches system python, and nothing ever actually gets pip installed into the pyenv python installs directly
pipenv and poetry
Why use a virtualenv management tool like pipenv or poetry? Each virtualenv gets a file defining its state (Pipfile or pyproject.toml), making them easy to manage and recreate, and it’s then that much easier for each project to have its own managed virtualenv.
Which to choose? It’s mostly your choice for which to use; I actually use both. I think they are good at different things, and good enough at those separate things to make it worth knowing and using both.
pipenv is better for specifying the environment for an application (e.g., a thing you deploy), which should have a very specific known python setup. It also lets you have .env files for project environment variables, which tend to be useful for applications (e.g., its deployment environment) and less so for libraries.
poetry is better for building libraries and CLI tools, which need to get put in PyPI and be flexible about the Python environment they end up in. Poetry’s PEP 518-compliant pyproject.toml is way easier than writing a setup.py file (which is still required if you’re using pipenv). It allows multiple virtualenvs/python versions (important for libraries). And, it can build wheels and publish to PyPI.
Both tools allow you to specify that the virtualenv directory for a project should go in the project directory itself (as .venv/), rather than in the central store of virtualenvs. I like this a lot, as it makes that virtualenv easier to find+destroy if it gets corrupted, and also gets inherently cleaned up if you remove the project directory. This feature is enabled with environment variables; instructions are included below.
pipx
pipx installs each package into its own virtualenv and then links the executables so they are on your $PATH
Isolated installs
To get isolated installations of these tools, I turn to homebrew, even on Linux (it’s good now!). As mentioned above, pip-installing requires selecting a pyenv-installed Python version to install into. You could pipx-install, but how do you install pipx? Additionally, brew formulas are kept much more up-to-date than most OS package managers (e.g., apt).
pyenv, pipenv, and pipx should be brew installed, but I recommend using poetry’s only standalone installation (for now), because the brew formula is currently broken.
homebrew
mac
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"linux
sudo apt-get install build-essential curl file git/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"
test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >> ~/.bashrcpyenv
Note: currently on linux, you should patch brew-installed pyenv for readline support (see this GitHub issue) so the REPL gets tab completion.
brew install pyenv# get latest versionsPY27=$(pyenv install --list | grep "^\s*2.7" | tr -d ' ' | tail -1)
PY35=$(pyenv install --list | grep "^\s*3.5" | tr -d ' ' | tail -1)
PY36=$(pyenv install --list | grep "^\s*3.6" | tr -d ' ' | tail -1)
PY37=$(pyenv install --list | grep "^\s*3.7" | tr -d ' ' | tail -1)
PY38=$(pyenv install --list | grep "^\s*3.8" | tr -d ' ' | tail -1)# set global python version
pyenv install $PY38
pyenv global $PY38
You can also configure pip to refuse to install outside of a virtualenv, see instructions here (thanks to Theron Luhn for this tip).
pipenv
brew install pipenv# Optional, if you want virtualenvs for a projects to go inside the project rather than centrally. Then when you delete a project, the virtualenv doesn't stick around. I like this a lot!
# YOUR_DOTFILE should be .profile on mac, and on linux, your .bashrc or whatever for your shellecho "export PIPENV_VENV_IN_PROJECT=1" >> YOUR_DOTFILE
poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python
poetry completions bash | sudo tee /etc/bash_completion.d/poetry.bash-completion > /dev/null# Optional, if you want virtualenvs for a projects to go inside the project rather than centrally. Then when you delete a project, the virtualenv doesn't stick around.
# YOUR_DOTFILE should be .profile on mac, and on linux, your .bashrc or whatever for your shellecho "export POETRY_VIRTUALENVS_IN_PROJECT=1" >> YOUR_DOTFILE
Notes
This is based originally on Jacob Kaplan-Moss’s recommendations in this blog post.
If you think this is excessive, that’s totally acceptable, but you can keep it to yourself. I have found that this works for me, and I’m sharing it in case other people find it useful.
If you think that Linux system package managers should be used in place of homebrew, you can continue to use them and I will not try to stop you.
If you think piping curl’d content from GitHub into your shell is a bad idea, I agree in principle but I think it’s an acceptable risk for these programs.
If you’re on Windows and don’t want to use WSL (fair, but give it a try!), I’m sorry that I don’t have something for you.
If you think Docker is a better answer, you might be right! I have not gone far enough down that path to have a strong opinion about it, but it seems plausible, though it seems to get fiddly around IDE/GUI integration and integration with common resources from your host (e.g., dotfiles). I’m also unsure about whether I’d want to go with the traditional “container-as-process” model or the “container-as-vm” model of LXD.
If you think I have not gone far enough and there are ways to make this even more hygienic, please let me know here or on Twitter.
