Topics Map > Services > Research Computing and Support > CCAST

Creating Python Virtual Environments

Notice: If you are looking for resources on HyFlex teaching, please visit our Teach HyFlex Page

This document describes how to create your own Python environment on CCAST, for installing and managing third-party Python packages.

Many HPC users want to install third-party Python libraries in their home directory. The most popular solution to this problem is Python virtual environments. Virtual environments allow you to create a self-sufficient Python environment in which you can install packages using the pip package manager. The steps below describe how to do this. 

Note: Lines beginning with $ indicate shell commands. You may type these manually, or copy/paste them from this document. Either way, do not include the $ when executing commands. Text below commands indicates expected output.

Load the latest version of Python available on CCAST. Currently, this is version 3.8.0. Note: You will not have to do this step every time you use your virtual environment; only the first time you set it up.

$ module load python/3.8.0

Use the venv module in Python3 to create your virtual environment. You may change myvenv in the command below to whatever name you like.

$ python -m venv myvenv

This will create an empty virtual environment with no packages installed except for the base version of Python. If you want your virtual environment to have access to all of the python packages in the CCAST version of Python, you have to add an additional flag. 

$ python -m venv --system-site-packages myvenv

Note: There may or may not be additional packages installed system-wide. If you feel that a package is popular enough that it should be installed system-wide, please contact support@ccast.ndsu.edu with your request.

You should now see a directory named myvenv (or whatever you named it) in the current working directory. Now, it is safe to unload the CCAST Python module.

$ module unload python

In order to install packages, we need to activate the virtual environment.

$ source myvenv/bin/activate

If successful, you should see your prompt change from something like this:

[user.name@login0001 ~]$

to this:

(myvenv) [user.name@login0001 ~]$

to indicate that you are “inside” the virtual environment.

Now you can run "pip install" to install whatever packages you like. A good first step is to upgrade pip itself.

$ pip install --upgrade pip

Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
    100% |████████████████████████████████| 1.4MB 4.4MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.0.3
    Uninstalling pip-19.0.3:
      Successfully uninstalled pip-19.0.3
Successfully installed pip-19.3.1

Then, if we wanted to install numpy:

$ pip install numpy

Collecting numpy
  Downloading https://files.pythonhosted.org/packages/9b/af/4fc72f9d38e43b092e91e5b8cb9956d25b2e3ff8c75aed95df5569e4734e/numpy-1.17.4-cp37-cp37m-manylinux1_x86_64.whl (20.0MB)
     |████████████████████████████████| 20.0MB 102kB/s 
Installing collected packages: numpy
Successfully installed numpy-1.17.4

You can see which packages are installed in your virtual environment by running the following command while inside the environment.

$ pip freeze

numpy==1.17.4

You can save the results of the above command to a requirements file, so that you can reproduce your environment on another system. 

$ pip freeze > requirements.txt
Here is an example requirements.txt file:

numpy==1.17.4
pandas==0.25.3
python-dateutil==2.8.1
pytz==2019.3
six==1.13.0

To install all of these packages, we would create a virtual environment as described in the previous steps, and then run the following:

$ pip install -r requirements.txt

Collecting numpy==1.17.4
  Using cached https://files.pythonhosted.org/packages/9b/af/4fc72f9d38e43b092e91e5b8cb9956d25b2e3ff8c75aed95df5569e4734e/numpy-1.17.4-cp37-cp37m-manylinux1_x86_64.whl
.
.
.
Installing collected packages: numpy, pytz, six, python-dateutil, pandas
Successfully installed numpy-1.17.4 pandas-0.25.3 python-dateutil-2.8.1 pytz-2019.3 six-1.13.0

To exit a virtual environment, run the following command while inside the environment. You will see your prompt change to indicate that you have exited the virtual environment.

$ deactivate

Lastly, if you want to load a virtual environment automatically every time you log in, add the command from step 4 to your ~/.bashrc file, and log out/in to see the change take effect.

$ echo “source ~/myvenv/bin/activate” >> ~/.bashrc




Keywords:Python, virtual environments, CCAST, pip install   Doc ID:101958
Owner:Nick D.Group:IT Knowledge Base
Created:2020-05-11 15:47 CDTUpdated:2020-08-24 14:36 CDT
Sites:IT Knowledge Base
Feedback:  0   0