Installing and Running FDS on a Linux Cluster

A common platform for running FDS simulations is a Linux cluster which consists of multiple computers referred to as "compute nodes" which are controlled by a single "head node." You typically login to the head node and launch jobs on the various compute nodes using a batch queuing system.

There are two ways that you can use FDS on a Linux cluster. You can either download and install the pre-compiled FDS and Smokeview binaries, or you can clone the firemodels/fds repository following these instructions. If you are just interested in running FDS, you probably want to do the former. If, however, you are interested in doing research, or working with the FDS developers, you should do the latter.

Installing the Pre-Compiled FDS and Smokeview Programs

1. Open a terminal session.

2. "cd" to the directory where the downloaded bundle is located, typically your home directory.

3. Run the installer script using the bash shell:

   $ bash FDS_i.j.k-SMV_l.m.n_linux64.sh
   

   Note that the version number for the file that you downloaded might be different. When you execute this command, there are some options for installation that will follow.

4. Make sure you unlimit your stack size. This is a common problem in getting jobs to run with Linux.

The installer no longer updates your startup files. The installer gives two options for updating your startup file (it no no longer touches any startup files). If you have modules installed on your system then add the following lines

export MODULEPATH=$HOME/FDS/FDS6/bin/modules:$MODULEPATH
module load FDS6

If you do not have modules then add the following line

source $HOME/FDS/FDS6/bin/FDSVARS.sh

Be sure to change the paths accordingly to match where you actually installed FDS.

Testing the Installation

To make sure that FDS installed properly, just type

fds

at the command prompt. You should see information about the version and date of compilation. If you are working at a single computer that is running Linux, you can now use FDS as you would have on a Windows PC. The FDS User's Guide provides some more details.

It is more than likely, however, that you are working on a Linux cluster, and if you just type fds at the command line, you will only launch a single process on the head node, which is not the way you want to use the cluster, except if you just have a short run that you want to debug or if you are developing an input file. Once you are ready to start longer jobs, you need to invoke the MPI (Message Passing Interface) functionality, which is taken up in the next section.

Running with MPI (distributed memory computing)

Starting with FDS 6.6.0, the Linux version uses Intel MPI. The libraries needed to run your job are installed with the downloaded installation package. A typical job is launched using a bash script (call it script.pbs, for example) like the following:

#!/bin/bash
#PBS -N job_name
#PBS -e /home/userid/.../job_name.err
#PBS -o /home/userid/.../job_name.log
#PBS -l nodes=2:ppn=2
#PBS -l walltime=2:0:0
export OMP_NUM_THREADS=1
cd /home/userid/.../
mpiexec -np 4 /home/userid/FDS/FDS6/bin/fds job_name.fds

The job_name is the base name of the input file, the .err and .log files contain what is usually spilled onto the screen when you run FDS. These files are typically created when the job is done. You can assign them to any directory you want because some Linux clusters have specific work spaces that are separate from the user directories. The parameter nodes indicates the name of nodes you want to use, and ppn is the number of processes per node. Our practice at NIST is to assign 2 MPI processes to each node because the nodes of our cluster have 2 sockets (i.e. chips), and jobs run fastest when we do not fill up the entire node. The walltime in this case is 2 hours. The job is typically killed after that, so choose wisely. The setting of OMP_NUM_THREADS is intended to overwrite any existing environment variable. For this example, we are not going to invoke OpenMP. The cd command changes directory to where the input file is located.

To submit this job, type

qsub script.pbs

Monitor your job by typing

qstat -a

Kill your job by typing

qdel jobid

where the jobid is given by the qstat command. There are many more options for these commands. Just do an Internet search and you'll see that many computing centers have listed them in detail. The ones listed here are the most important.