Managing Python Environments with direnv and pyenv
As Python developers, most of us are familiar with Virtual Environments. One of the first things we do when working on a new project is to create an environment. We commonly use virtualenv or venv exactly for that purpose.
Each project we work on uses different packages and may even be compatible with only one Python version.
Doing something repeatedly warrants automation. In this article, we'll see how
pyenv can help us do that.
As a side note, some modern IDEs already automated these steps. For example, PyCharm will create the Virtual environment when initializing a project:
Although automating all these steps is a great win if we use IDEs that support such functionalities, a more generic solution should be IDE-agnostic.
The Problems of virtualenv
Imagine we found a project on GitHub and we would like to play around with it. Pyweather is a simple script that requests the extended weather forecast for our location and prints it on the terminal. These are the steps we take in order to try the script on our machine:
$ git clone https://github.com/lcofre/pyweather.git $ cd pyweather
Then we create the virtual environment and install the packages the script uses:
$ virtualenv --python=python3 env $ source env/bin/activate (env) $ pip install requirements.txt
And only then, we can execute the script:
(env) $ ./pyweather.py
We created a virtual environment and saved it in the root folder of our project. While being on that folder we had to activate the environment with the
When we finish working, we need to leave the Virtual Environment by executing
(env) $ deactivate
All those steps are our responsibility. How many times we may have forgotten to activate an environment and installed a package globally!
Let's see how
direnv helps us automate this.
direnv was made mainly to load environment variables, depending on the current directory and has an extension for many shells.
In this example, we'll be using using
direnv supports many other shells as well. And what's more important for us, it allows us to manage Python Virtual Environments.
To install it we'll run the
bash installer they provide. We could use the package manager of our distribution, but the
bash installer will ensure we install the latest version available:
$ curl -sfL https://direnv.net/install.sh | bash
Now we need to hook
bash. We'll edit
~/.bashrc and then reload it:
$ echo 'eval "$(direnv hook bash)"' >> ~/.bashrc $ source ~/.bashrc
direnv will link itself to the shell and will be executed always before each prompt. We will never notice it's working on the background.
direnv will check if something needs to be loaded on the current folder. It checks the existence of a file named
.envrc, with instructions on what should be loaded.
To load Python Virtual Environments we run the
layout command, followed by the Python version:
$ echo 'layout python' > .envrc
Or if we want to use Python 3:
$ echo 'layout python3' > .envrc
Running these will tell
direnv to look for a
python3 executable on the path.
As soon as we create the
.envrc file we'll be warned that we need to allow
direnv to access that folder. Let's do that right now:
$ direnv allow
direnv: loading .envrc ... New python executable in /home/myuser/untitled/.direnv/python-3.6.9/bin/python3 ... Installing setuptools, pkg_resources, pip, wheel...direnv: done. direnv: export +VIRTUAL_ENV ~PATH
As we can see in the output, the Virtual Environment was immediately created. The prompt is not modified though, so we won't see the name of the environment written at the beginning. Now we can install the packages we need as we did on the environment we created in the previous section:
$ pip install -r requirements.txt
direnv will silently activate the environment in the background. Whenever we move out of the directory, the environment will be deactivated:
$ cd ..
If we can use any Python version that is installed on the system,
direnv is all we need.
Let's suppose now that our
pyweather script requires a very specific version though.
pyenv is a version management utility for Python. It allows, among other things, to change Python versions on a per-project basis.
direnv provides support for it since version
2.21.0, so together they can give us a higher level of control on the version we use in our environment.
Let's start by installing
$ curl -L https://pyenv.run | bash
And then ensuring it will always be accessible to our terminal:
$ echo 'export PATH="~/.pyenv/bin:$PATH"' >> ~/.bashrc $ echo 'eval "$(pyenv init -)"' >> ~/.bashrc $ echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc $ source ~/.bashrc
Now let's suppose our
pyweather script requires a very specific Python version,
First, we need to install that version of Python:
$ pyenv install 3.6.2
And now we can configure our project to use the specific version:
$ echo 'layout pyenv 3.6.2' > .envrc $ direnv allow
We can confirm all works as expected by checking the Python version in the environment:
$ python --version
If we ever need to change the Python version, it will be enough for us to change the layout in the
Thanks to both utilities we can change the layout to any Python version, and our virtual environment will be updated right away.
Another advantage of using both
pyenv is that we can version our
.envrc file in our project repository.
That way all contributors will be able to configure their environment as intended by the project, as long as they install the utilities and Python version needed.
Virtual Environments are in a way detached from the Python development workflow. We need to remember to configure and activate it before working with our project. Thanks to
pyenv we can automate all this, and entering the project folder will do all the work for us in the background.
Installation of both utilities is not straightforward, but after being done once we will save ourselves a lot of time. We will also always have the certainty that we are working with the right virtual environment and Python version.