Topics Map > Services > Research Computing and Support > CCAST
CCAST User Guide
This User Guide provides essential information about advanced research computing resources at CCAST/NDSU and how to use them. A must-read document for all CCAST users.
- Introduction, Context, and Qualifications
- Getting Started
- Research Computing Resources
- Running Jobs
- Utilization Monitoring
1. Introduction, Context, and Qualifications
The Center for Computationally Assisted Science and Technology (CCAST; pronounced "c-cast") provides advanced cyberinfrastructure for computational research and education at NDSU and beyond. CCAST develops, manages, brokers, and operates high-performance (HPC), cloud, and interactive computing resources, and educates researchers on proper and efficient use of the resources and on other topics of interest to the computational science and engineering community. We use UNIX/Linux primarily. The basic level of services is FREE of charge to NDSU faculty, staff, and students as well as certain external collaborators (upon approval of CCAST's Executive Director). Additional services are available for a fee.
1.1 Acknowledging CCASTUsers are required to include the following statement (or a close variant) in all research outputs (papers, presentations, theses, etc.) that have used CCAST resources: "This work used resources of the Center for Computationally Assisted Science and Technology (CCAST) at North Dakota State University, which were made possible in part by NSF MRI Award No. 2019077."
1.2 Reporting requirementsUsers, usually through their Principal Investigators (PIs; i.e., sponsors of their CCAST accounts), are required to report any research outputs and activities that have been enabled by the use of CCAST resources. Reporting items often include publications, presentations, grant applications, patents, theses, etc.
1.3 CCAST usage policiesUsers are required to carefully read and comply with CCAST Usage Policies.
1.4 How can you get help?Read this User Guide carefully and check the CCAST website and related Knowledge Base articles before contacting us. If you still cannot find answers to your questions, send an e-mail to email@example.com. In the e-mail, describe the issues, clearly state your questions, and provide a copy of the error messages and job submission script, the IDs of your failed jobs, and any other info that may help debug the issues. Please do not directly contact CCAST individual staff for technical support as this bypasses our tracking system to avoid dropped calls.
1.5 About this documentThis document will be updated often since hardware specifications, system administration practice, and usage policies, etc. are subject to changes.
2. Getting Started
2.1 Applying for an accountTo be able to access to "Thunder" and "Thunder Prime"–the two HPC clusters at CCAST–you need to have an active account with us. Please apply for a CCAST account if you have not already done so. A link to the online application form is available on the CCAST website.
2.2 Connecting to CCAST's HPC clustersFrom a Windows computer: PuTTY, a free SSH and telnet client, should be used. Download and install it, then double-click to open the application. In the "Host Name (or IP address)" field, enter the hostname: thunder.ccast.ndsu.edu (for Thunder) or prime.ccast.ndsu.edu (for Thunder Prime). Select (or leave) 22 for "Port" and SSH for "Connection type". Click "Open", you will be asked to enter your username and password.
2.3 Transferring filesBetween a Windows computer and Thunder: WinSCP client should be used. Download (for free) and install it, then open the application. In the "WinSCP Login" window, enter the hostname thunder.ccast.ndsu.edu (Thunder) or prime.ccast.ndsu.edu (Thunder Prime) as well as your username and password, then click on "Login". Once logged in, you will see a screen with two panels: the left shows files on your computer and the right shows your files on Thunder or Thunder Prime (usually your HOME directory, but you can double-click on the address bar and change the location). You can then easily drag and drop files between your computer and Thunder or Thunder Prime.
2.4 Learning UNIX/Linux and HPC
3. Research Computing Resources
3.1 HardwareCCAST’s Thunder has ~130 compute nodes (with >3,700 Intel CPU cores and 22 Nvidia GPU cards in total, including 8 Tesla P100 SMX2 GPUs, 4 Quadro GP100 GPUs, and 10 GeForce RTX 2080 Ti GPUs); each node has either 20, 32, 40, or 44 cores. Thunder Prime currently has >44 computes nodes (with >4,200 AMD EPYC 7662 CPU cores and 10 Nvidia A100 and 12 Nvidia A10 GPU cards); each node has either 64 or 128 cores. There are several big-memory nodes on each cluster. To check which nodes are currently free or partially free on Thunder or Thunder Prime, execute the command freenodes. The information will help you make the right choice when you request computing resources for your jobs.
3.2 SoftwareThere are many software programs installed on Thunder and Thunder Prime. Most are available to all CCAST users; some, e.g., ANSYS, Gaussian, VASP, etc., available only to those who have valid licenses and other authorized users. Software are usually organized as modules; to check available modules, execute module avail. You can also install software for yourself. Contact CCAST at firstname.lastname@example.org if you need help.
3.3 Storage spaceOnce logged in, you are in your HOME directory: /mmfs1/thunder/home/username (Thunder) or /mmfs1/home/username (Thunder Prime). Data in HOME is backed up periodically to tape, so it is a reliable storage area. Do not use your HOME directory for data or job input/output. Running jobs out of HOME is prohibited as it affects the interactive use and other important tasks.
3.4 Compute CondominiumResearchers can purchase condo nodes using equipment purchase funds from their grants or other available funds. These PI-owned compute nodes are attached to CCAST’s Thunder Prime cluster to take advantage of the existing infrastructure. Contact CCAST if you have questions regarding the condominium model.
4. Running JobsOnce you logged in to a CCAST HPC cluster, you are on one of its login nodes. Login nodes have limited resources and are intended only for basic tasks such as transferring data, managing files, compiling software, editing scripts, and checking on or managing jobs. DO NOT run your jobs on the login nodes!
4.1 Sample input files and job scriptsIf you are new to running jobs on Thunder and/or Thunder Prime or if it has been a while since the last time you ran an application, it is highly recommended that you first run some sample jobs we provide before running your own jobs. Users can copy sample input files and job scripts for various applications from /mmfs1/thunder/projects/ccastest/examples (Thunder) or /mmfs1/projects/ccastest/examples (Thunder Prime). More job examples for more applications will be added as they become available. Please check this directory frequently for the latest version of the job scripts.
4.2 Queue policies on Thunder and Thunder PrimeDifferent types of queues on Thunder are given below. Users can also find info about the queues on Thunder or Thunder Prime by executing qstat -q.
|Route Queue||Execution Queue||Walltime (hours)||Authorized Group|
|condo01, condo02, etc.||-||condo owners|