Save yourself from the Python dependency conflict

Save yourself from the Python dependency conflict

Anyone that has worked with Python for more than a few days knows the struggle of keeping packages isolated. Multiple Python projects, that need to co-exist on the same computer, can have different and often conflicting requirements.

Here I want to discuss a Python tool that helps you to keep Python packages isolated: Virtualenv.

Virtualenv allows you:

  • to create a “virtual” isolated Python installation and
  • to install packages into that virtual installation.

Virtualenv

What is virtualenv and why you should use it

What: Virtualenv is a Python package that creates a folder where Python programs can run. Period.

Why: because you want to play with different versions of libraries and you don't want to mess with other projects and, most importantly, the System libraries. Horribly dangerous things happen when you try to tame the complicated, ever-growing, dependency tree of Python in a Linux machine.

How to install virtualenv

Using the Python package manager (pip)

root@65ded660d870:~# python3 -m pip install virtualenv

Or using the system package manager (e.g. apt):

root@65ded660d870:~# apt install virtualenv

Create a virtualenv

I will use the virtualenv pip package.

root@65ded660d870:~# python3 -m venv env

This command creates a virtual environment in the env folder. A virtualenv, as you can see, is just a bunch of directories needed for running Python programs (with a copy of python included)

root@65ded660d870:~# ls -a env/
./  ../  bin/  include/  lib/  lib64@  local/  pyvenv.cfg  share/

Work in a virtualenv

You have to activate the virtualenv in order to work in it. 

 root@65ded660d870:~# . env/bin/activate
 (env) root@65ded660d870:~# pip install ansible

Nice, the activation puts a friendly reminder in my prompt.

But... What is happening under the hood?

A bash session works by searching the executables in the $PATH environment variable, which is affected by the activation of the virtualenv. Activating the env means giving the env/bin folder the highest priority. 

 (env) root@65ded660d870:~# echo $PATH
 /root/env/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Let's wrap our head around it:

root@16abbefbfa05:~# echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@16abbefbfa05:~# which python
/usr/local/bin/python
root@16abbefbfa05:~# . env/bin/activate
(env) root@16abbefbfa05:~# echo $PATH
/root/env/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
(env) root@16abbefbfa05:~# which python
/root/env/bin/python

Virtualenv and pip

Virtualenv is the tool that creates the isolated environment. pip is a tool for installing Python packages from the Python Package Index. You'll find pip automatically installed in the virtual environment. You have to use pip to manage packages in a virtualenv.

Keep the versions under control

Now, can we keep the virtual environment under control? In order to make reproducible software we have to care about making reproducible environments. You can use the pip freeze command to write down the versions of the packages installed:

(env) root@16abbefbfa05:~# pip install ansible
(env) $ pip freeze
ansible==2.8.0
cffi==1.13.2
cryptography==2.8
enum34==1.1.6
ipaddress==1.0.23
Jinja2==2.10.3
MarkupSafe==1.1.1
pkg-resources==0.0.0
pycparser==2.19
PyYAML==5.2
six==1.13.0

Then you can redirect the output to a file and put it under some version control system, e.g. git:

(env) root@65ded660d870:~# mkdir my-app && cd my-app
(env) root@65ded660d870:~# git init
(env) root@65ded660d870:~# pip freeze > requirements.txt
(env) root@16abbefbfa05:~/my-app# git add requirements.txt && git commit -m 'The requirements'

Keep the versions updated

A thing that I find very neat is to add a git-hook to remind myself of updating the requirements file.

(env) root@16abbefbfa05:~/my-app# cat .git/hooks/pre-commit
#!/bin/bash
diff requirements.txt <(pip freeze)
if [ $? != 0 ]
then
  echo "Env changes not updated in requirements.txt. Run 'pip freeze > requirements.txt' to update."
  exit 1
fi

If the output of pip freeze and the requirements file are different, the action of git commit will be prevented by this message.

Can I roll back to a previous state?

No. Virtualenv is not aware of its story, so in order to roll back to a previous state you are bound to using a version control tool.

Can you uninstall packages?

Yes, you just run pip remove package command in the virtualenv.

archiviato sotto: , ,