• Created by Unknown User (_andy_wang), last modified on Aug 23, 2021

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Below are a number of sample scripts that can be used as a template for building your own SLURM submission scripts for use on SCC. These scripts are also located at: /opt/scc/generic/examples/SLURM/, and can be copied from there. If you choose to copy one of these sample scripts, please make sure you understand what each #SBATCH directive before before using the script to submit your jobs. Otherwise, you may not get the result you want and may waste valuable computing resources.

Before allocating hundreds of jobs to the SLURM queue, it is a good idea to test your submission script using a small subset of your input files. Make sure that SLURM arguments for the number of CPUs, amount of memory and etc. are specified adequately and will not harm other users.

Single-Threaded Job

This script can serve as the template for many single-processor applications. The mem-per-cpu flag can be used to request the appropriate amount of memory for your job. Please make sure to test your application and set this value to a reasonable number based on actual memory use. The %j in the --output line tells SLURM to substitute the job ID in the name of the output file. You can also add a -e or --error line with an error file name to separate output and error logs.

#!/bin/bash -l
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@camh.ca     # Where to send mail	
#SBATCH --ntasks=1                    # Run on a single CPU
#SBATCH --mem=1gb                     # Job memory request
#SBATCH --time=00:05:00               # Time limit hrs:min:sec
#SBATCH --output=serial_test_%j.log   # Standard output and error log
pwd; hostname; date

module load lang/Python
#check loaded module
ml

echo "Running python script on a single CPU core"

python /opt/scc/generic/examples/SLURM/helloworld.py

date


Multi-Threaded SMP Job

This script can serve as a template for applications that are capable of using multiple processors on a single server or physical computer. These applications are commonly referred to as threaded, OpenMP, PTHREADS, or shared memory applications. While they can use multiple processors, they cannot make use of multiple servers and all the processors must be on the same node.

These applications required shared memory and can only run on one node; as such it is important to remember the following:

  • You must set --ntasks=1, and then set --cpus-per-task to the number of OpenMP threads you wish to use.
  • You must make the application aware of how many processors to use. How that is done depends on the application:
    • For some applications, set OMP_NUM_THREADS to a value less than or equal to the number of cpus-per-task you set.
    • For some applications, use a command line option when calling that application.

#!/bin/bash
#SBATCH --job-name=parallel_job_test # Job name
#SBATCH --mail-type=END,FAIL         # Mail events (NONE, BEGIN, END, FAIL, ALL)
#SBATCH --mail-user=email@camh.ca    # Where to send mail	
#SBATCH --nodes=1                    # Run all processes on a single node	
#SBATCH --ntasks=4                   # Number of processes
#SBATCH --mem=1gb                    # Total memory limit
#SBATCH --time=01:00:00              # Time limit hrs:min:sec
#SBATCH --output=multiprocess_%j.log # Standard output and error log
date;hostname;pwd

module load lang/Python

python /opt/scc/generic/examples/SLURM/python_openmp.py

date

Message Passing Interface (MPI) Jobs

PMIx Versions

When launching applications linked against our OpenMPI libraries via srun, you must specify the correct version of PMIx using the "--mpi" srun option. Generally speaking you can determine the appropriate PMIx version to use by running the ompi_info command after loading the desired OpenMPI environment module. For example,

$ module load bio/NEURON/7.6.7_Python-2.7.17
$ ompi_info --param pmix all
                 MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v3.1.4)
                 MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v3.1.4)
                 MCA pmix: pmix2x (MCA v2.1.0, API v2.0.0, Component v3.1.4)

$ ml purge
$ module load bio/NEURON/7.8.2_LFPy-2.2_Python-3.8.5
$ ompi_info --param pmix all
                MCA pmix: flux (MCA v2.1.0, API v2.0.0, Component v4.1.0)
                MCA pmix: isolated (MCA v2.1.0, API v2.0.0, Component v4.1.0)
                MCA pmix: pmix3x (MCA v2.1.0, API v2.0.0, Component v4.1.0)

In the examples above, you would specify pmix_v2 (i.e. ext2x) for the bio/NEURON/7.6.7_Python-2.7.17   and pmix_v3 (ext3x) for the second set of modules.

Step-by-step guide


  • No labels