Topics Map > Services > Research Computing and Support > CCAST
Using Python on CCAST
This article discusses the different versions and types of Python installations available on CCAST, how to use them, and how to create custom environments with your own list of packages.
Python installations available on CCAST
There are several ways to run Python on CCAST. Below is a list of the different Python installations available and how to use them.
Base (operating system) Python
Python is installed standard as part of the operating system on all CCAST servers (login and compute nodes). These versions are available simply by logging into the system with a terminal and running python
. This will launch a Python interpreter from one of the following locations:
/usr/bin/python
/usr/bin/python2
/usr/bin/python3
On newer systems (e.g. Thunder Prime) running python
will default to Python version 3, whereas on older systems (e.g. Thunder) running python
will default to Python version 2. When in doubt, the best practice is to explicitly run the version you want by calling either python2
or python3
.
The default, operating system Python installation is good for light scripting, or for code that only uses what’s available in the base Python distribution. If you need to make use of third-party packages in your Python code, consider using one of the other Python installations mentioned below.
Basic versioned Python modules
There are several Python versions available via the modules
framework. To view Python versions available via the modules system, run module avail python
. Here are the stable versions available as modules on CCAST:
python2/2.7.18
python/3.8.6-gcc-2pmf
python/3.8.6-intel-uly7
To load one of these modules, e.g. python2/2.7.18
, run module load python2/2.7.18
in your CCAST terminal session.
Like the base operating-system Python versions, these only include a basic Python distribution with minimal third-party packages. The benefit, however, is that these are strictly versioned and are consistent across nodes. Whereas, the base operating-system Python distributions are subject to version changes if the systems are updated.
Anaconda3 Python modules
The third type of Python distributions available on CCAST is Anaconda3. Anaconda3 is both a Python distribution framework and package manager. (It can also manage other languages, like R and Julia.) There are several versions of Anaconda3 available via the modules system, which can be found by running module avail anaconda
:
anaconda3/2021.11
anaconda3/2022.05
One key difference between the Anaconda3 modules and the other two types of Python distributions mentioned previously, is that Anaconda3 comes pre-loaded with many (but not all) of the most popular third-party packages. Modules can be loaded the same way as before: module load anaconda3/2022.05
.
Another benefit of Anaconda3 is that it can be installed by users in their own home/project directories, and can then be used to create custom environments for installing extra, non-standard packages (see below).
Creating custom Python environments
There are two different ways CCAST users can create their own Python environments: virtual environments and conda environments. Instructions for each are provided below.
Python virtual environments
Python virtual environments use the pip
pacakge manager and a basic versioned Python distribution. Here are the basic steps:
1. Load a Python3 module
module load python/3.8.6-gcc-2pmf
2. Create the virtual environment
python -m venv myenv
This will create a directory named myenv
in your current working directory, containing a link to the Python interpreter as well as folders for libraries and other supporting files. For consistency, virtual environment folders should not be moved once created, so make sure you are in the directory you want before creating the environment.
If the above command succeeds, you can now unload the Python module:
module unload python/3.8.6-gcc-2pmf
3. Activate your virtual environment
source myenv/bin/activate
Upon activating the virtual environment, you should see your terminal prompt change from something like this:
[user.name@login0003 ~]$
to something like this:
(myenv) [user.name@login0003 ~]$
This is to remind you that you are “inside” the virtual environment.
4. Upgrade pip and install packages
With a new virtual environment, it is always a good idea to first update the pip
package manager:
pip install --upgrade pip
After this, you can install the packages you need. For example, to install numpy
:
pip install numpy
5. Exiting the virtual environment
When you are done working in the virtual environment, simply run:
deactivate
Your terminal prompt will revert to normal to indicate that you are no longer in the virtual environment.
To use the environment again in the future, rerun the source
command from step 3.
Installing Anaconda/Miniconda in your home directory
As mentioned previously, Anaconda3 can be installed in individual users’ home/project directories, and can be used to create custom Python environments similar to virtual environments. Before Anaconda can be used to create custom environments, it must be installed according to the following steps:
1. Choose between Anaconda and Miniconda
There are two options available:
Anaconda provides a standard Python distribution, the conda
package manager, and lots of popular third-party packages.
Miniconda only provides Python and the conda
package manager, and therefore, is lighter-weight than a full Anaconda installation.
2. Select the appropriate installer
Since we will be installing Anaconda/Miniconda on CCAST, we need the versions for Linux:
For the rest of this tutorials, we will assume installation of Anaconda3, but steps for Miniconda3 are nearly identical.
3. Download the installer to your home directory
Right-click and copy the download link for Linux 64-Bit x86 and paste it into your terminal on CCAST, along with the wget
command, like so:
wget https://repo.anaconda.com/archive/Anaconda3-2022.10-Linux-x86_64.sh
Then hit Enter to start the download. This will create the installer Anaconda3-2022.10-Linux-x86_64.sh
in your working directory.
4. Run the installer and follow the prompts
To execute the installer, run:
bash Anaconda3-2022.10-Linux-x86_64.sh
This will launch an interactive text menu that will walk you through the installation process. The first step is to read agree to the license. Hit Enter to open the license, then use Enter again to scroll through and read the license agreement. At the end you will see the following prompt:
Do you accept the license terms? [yes|no]
[no] >>>
Type yes
and hit Enter to continue the installation process.
Next, Anaconda will ask you to confirm the installation location. The default location is ~/anaconda3
but you may select another directory you have access to (e.g. a project directory).
Anaconda3 will now be installed into this location:
/mmfs1/home/user.name/anaconda3
- Press ENTER to confirm the location
- Press CTRL-C to abort the installation
- Or specify a different location below
[/mmfs1/home/user.name/anaconda3] >>>
For now, we will just accept the default and hit Enter. After this, Anaconda will unpack all of it’s assets and begin the install. This can take several minutes, so be patient.
Once the installation is finished, the installer will ask if you want to initialize conda:
Do you wish the installer to initialize Anaconda3
by running conda init? [yes|no]
[no] >>>
Most users will want to answer yes
to this question, as running conda init
makes the conda
set of commands available to your terminal for all future terminal sessions. However, this will also set conda to auto-activate the base environment. This means every time you login, your terminal will look like this:
(base) [user.name@login0003 ~]$
This has the effect of making the Anaconda-installed Python your default Python installation. In most cases, this doesn’t cause problems, but it can sometimes cause unpexpected behavior. If you would prefer not to have conda auto-activate on login, you can following command:
conda config --set auto_activate_base false
This change will take effect on logging out and logging back in. Then, in the future, to activate conda, simply run:
conda activate
Likewise, to deacticate conda:
conda deactivate
Anaconda is now installed and we can use it to create custom Python environments.
Creating custom Python environments with Anaconda
Users may install third-party packages in their base conda environment, but most times, it is recommended to create separate environments for each different project you are working on. Here are the steps:
1. Create a conda environment
conda create -n myenv
Answer y
to the prompt. This will create an environment inside the anaconda3 installation directory.
2. Activate the environment and install packages
conda activate myenv
Again, this will modify your terminal to show the environment name (myenv)
at the beginning of your prompt. Now you can install packages. We will again use numpy
as an example:
conda install numpy
Then respond y
to the prompt and hit Enter to start the installation.
Note: Although numpy
comes pre-installed with Anaconda, this is only in the base conda environment. Anytime you create a new environment, it is as if there are no packages installed.
When finished using your environment, you can deactivate it using conda deactivate
as before.
Batch job examples for different Python installations
Python example jobs are located at /mmfs1/projects/ccastest/examples/Python_examples
on Thunder Prime. You can copy this directory to your scratch directory with the following command:
cp -r /mmfs1/projects/ccastest/examples/Python_examples $SCRATCH/
The directory contains a hello.py
Python file and 4 job script templates:
- basic_python.pbs: Uses the
python/3.8.6-gcc-2pmf
basic Python distribution. - global_conda.pbs: Uses the
anaconda3/2022.05
Anaconda3 module. - personal_conda.pbs: Uses a personal Anaconda3 environment.
- Assumes Anaconda is installed in
$HOME
withbase
environment. Both can be modified.
- Assumes Anaconda is installed in
- virtual_env.pbs: Uses a virtual environment.
- Must supply path to
bin/activate
script.
- Must supply path to
After copying to scratch and entering the directory, any of these examples can be submitted using qsub
. For example:
qsub basic_python.pbs
To use personal_conda.pbs
and virtual_environment.pbs
, users must have installed Anaconda3 in their home directory or created a virtual environment, respectively.
Integrating custom Python environments with Jupyter Notebook
In addition to running Python jobs via the batch scheduler, CCAST users can run Python interactively through the Jupyter Notebook app in Open OnDemand. To launch a Jupyter session, login to CCAST’s Open OnDemand service and select “Jupyter” from the “Interactive Apps” menu.
By default, Jupyter launches with one of the Anaconda3 modules, in order to provide a standard environment with popular packages for beginners to get started quickly. More advanced users may want to integrate their own custom Python environments with the Jupyter app. To do so, follow these steps:
1. Activate your Python environment
For virtual environments:
source path/to/virtual_environment/bin/activate
For conda environments:
conda activate myenv
2. Install the jupyter package in your environment
In order to integrate with Jupyter, you must have the jupyter
package installed, alongside any other packages you’ve installed.
For virtual environments:
pip install jupyter
For conda environments:
conda install jupyter
Note: The base Anaconda3 environment may include Jupyter out-of-the-box. If you aren’t sure, there’s no harm in reinstalling it a second time.
3. Install the environment as a ipykernel
For both virtual and conda environments:
python -m ipykernel install --user --name=MyEnvName
Note: The name you assign with the --name
command will be the name that appears in the Jupyter app.
4. Launch a Jupyter Notebook session and create a new notebook with your kernel
In the Jupyter interface, select the “New” dropdown on the right and you should see your custom kernel. Selecting it will launch a new notebook session with your custom environment.
5. Managing Jupyter kernels
To see a list of configured kernels from the terminal:
jupyter kernelspec list
To remove a kernel:
jupyter kernelspec remove MyEnv