Creating Python Virtual Environments
This document describes how to create your own Python environment on CCAST, for installing and managing third-party Python packages.
Many HPC users want to install third-party Python libraries in their home directory. The most popular solution to this problem is Python virtual environments. Virtual environments allow you to create a self-sufficient Python environment in which you can install packages using the pip package manager. The steps below describe how to do this.
Note: Lines beginning with $ indicate shell commands. You may type these manually, or copy/paste them from this document. Either way, do not include the $ when executing commands. Text below commands indicates expected output.
Load the latest version of Python available on CCAST. Currently, this is version 3.8.0. Note: You will not have to do this step every time you use your virtual environment; only the first time you set it up.
$ module load python/3.8.0
Use the venv module in Python3 to create your virtual environment. You may change myvenv in the command below to whatever name you like.
$ python -m venv myvenv
This will create an empty virtual environment with no packages installed except for the base version of Python. If you want your virtual environment to have access to all of the python packages in the CCAST version of Python, you have to add an additional flag.
$ python -m venv --system-site-packages myvenv
Note: There may or may not be additional packages installed system-wide. If you feel that a package is popular enough that it should be installed system-wide, please contact firstname.lastname@example.org with your request.
You should now see a directory named myvenv (or whatever you named it) in the current working directory. Now, it is safe to unload the CCAST Python module.
$ module unload python
In order to install packages, we need to activate the virtual environment.
$ source myvenv/bin/activate
If successful, you should see your prompt change from something like this:
(myvenv) [user.name@login0001 ~]$
to indicate that you are “inside” the virtual environment.
Now you can run "pip install" to install whatever packages you like. A good first step is to upgrade pip itself.
$ pip install --upgrade pip
Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
100% |████████████████████████████████| 1.4MB 4.4MB/s
Installing collected packages: pip
Found existing installation: pip 19.0.3
Successfully uninstalled pip-19.0.3
Successfully installed pip-19.3.1
Then, if we wanted to install numpy:
$ pip install numpy
Downloading https://files.pythonhosted.org/packages/9b/af/4fc72f9d38e43b092e91e5b8cb9956d25b2e3ff8c75aed95df5569e4734e/numpy-1.17.4-cp37-cp37m-manylinux1_x86_64.whl (20.0MB)
|████████████████████████████████| 20.0MB 102kB/s
Installing collected packages: numpy
Successfully installed numpy-1.17.4
You can see which packages are installed in your virtual environment by running the following command while inside the environment.
$ pip freeze
You can save the results of the above command to a requirements file, so that you can reproduce your environment on another system.
$ pip freeze > requirements.txt
Here is an example requirements.txt file:
To install all of these packages, we would create a virtual environment as described in the previous steps, and then run the following:
$ pip install -r requirements.txt
Using cached https://files.pythonhosted.org/packages/9b/af/4fc72f9d38e43b092e91e5b8cb9956d25b2e3ff8c75aed95df5569e4734e/numpy-1.17.4-cp37-cp37m-manylinux1_x86_64.whl
Installing collected packages: numpy, pytz, six, python-dateutil, pandas
Successfully installed numpy-1.17.4 pandas-0.25.3 python-dateutil-2.8.1 pytz-2019.3 six-1.13.0
To exit a virtual environment, run the following command while inside the environment. You will see your prompt change to indicate that you have exited the virtual environment.
Lastly, if you want to load a virtual environment automatically every time you log in, add the command from step 4 to your ~/.bashrc file, and log out/in to see the change take effect.
$ echo “source ~/myvenv/bin/activate” >> ~/.bashrc