Python libraries are collections of reusable code that provide functionality for a wide range of tasks, from data analysis and machine learning to web development and automation. Libraries are often hosted on the Python Package Index (PyPI) and can be easily installed using package managers like pip.

As you work on different Python projects, you may need different versions of the same module. Probably even a module that you already have installed system-wide. For this, we use virtual environments. These environments allow you to install specific module versions in a local directory and alters your shell's environment to prioritize using them. Switching between environments can be as easy as sourceing another setup script.

The problem with virtual environments is that they don't mesh well with apt. In stead of apt, we will use a Python module manager called pip3. Our suggestion is to use pip only in virtual environments. Yes, it can also install modules system-wide, but most modules can be found as apt packages anyway. Generally, it is not a good idea to mix package managers (and modern Linux distributions outright deny this with explicit error when trying to install Python packages outside a virtualenv)!

First things first, we need python3, the venv module, and pip. These, we can get with apt

$ sudo apt install python3 python3-venv python3-pip

Assuming that you are in your project's root directory already, we can set up the virtual environment:

$ python3 -m venv .venv

The -m flag specifies to the Python interpreter a module. python3 searches its known install paths for said module (in this case, venv) and runs it as a script. .venv is the script's argument and represents the name of the storage directory. Take a look at its internal structure:

$ tree -L 3 .venv

Notice that in .venv/bin/ we have both binaries and activation scripts. These scripts, when sourced, will force the current shell to prioritize using these. The modules you install will be placed in .venv/lib/python3.*/site-packages/. Try to activate your environment now. Once active, you will have access to the deactivate command that will restore your previous environment state:

$ source .venv/bin/activate
$ deactivate

VIRTUAL_ENV_DISABLE_PROMPT="yes"

Same as apt, pip used to have a search function for modules. Unfortunately, they removed this feature due to a high number of queries. Now, to search for modules, you will need to use the web interface.

Let us install the modules needed for this laboratory. After that, let us also check the versions of all modules (some will be added implicitly). Can you also find their installation path in the .venv/ directory?

$ pip3 install requests beautifulsoup4
$ pip3 list

$ pip3 freeze > requirements.txt
$ pip3 install -r requirements.txt

We can use the python3 interpreter in interactive mode to quickly test whether our modules installed correctly:

$ python3
Python 3.12.7 (main, Oct  1 2024, 11:15:50) [GCC 14.2.1 20240910] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> help(requests.get)

If you ever need to debug your python program, the best approach is to open up a shell and run your program a bit tedious (copy pasting each line of the program). However, there's a better method:

import IPython
 
IPython.embed(colors='neutral')

$ sudo apt install ipython3

This will stop the execution of your script and open an ipython shell that has access to a copy of all your local & global data. Use this to inspect the state of your variables, their types, etc. Exit the shell (Ctrl + D) to continue the execution.