5. Processing

This section describes the usage of the programs that are part of the PI-DSMC program package. There are separate programs: PIDS2geo, PIDS3geo, PIDSMC and PIDSMCfrontend. None of these programs has a graphical user interface. All programs should be started from the command line.

5.1 Pre-Processing with PIDS2geo/PIDS3geo

The programs PIDS2geo and PIDS3geo are used to generate 2D and 3D data sets. Both programs require a configuration file that can be created with PIDSMCconfig. The table below lists the available command line parameters of PIDS2geo and PIDS3geo.

name required / optional description
-g config_filerequiredName of the configuration file
-d directoryoptionalDirectory where the data set will be saved. The current directory will be used if this parameter is not specified.
-hoptionalDisplay help
-poptionalGenerate a new dataset.
-np xoptionalUse x CPUs for geometry generation.

If PIDS2geo is executed with the -g option only, a menu with the available options is displayed. The options are:
  1. Create new data set

  2. Display information about the data set

The option "display information" will show some useful information about the specified data set like the number of molecules in each domain, the grid size and the number of real molecules represented by each simulated molecule.

5.2 DSMC Processing

After the generation of the data set, the DSMC calculation can be started.

name required / optional description
-d directoryrequiredDirectory where the data is stored. The current directory will be used if this parameter is not specified.
-hoptionalDisplay help
-tl timeoptionalAbort the simulation when the specified flow time [s] was reached.
-dll filenameoptionalEnable the DLL interface. The DLL with the specified name will be loaded.
-shmoptionalUse shared memory to save memory.
-seoptionalSave data set before the program aborts.
-numa a boptionalBind a instances to each NUMA node starting with node b. This prevents the OS from moving processes around.
-cdtoptionalUse constant time step, i.e. classical DSMC.
-cleanoptionalRemove shared memory pages after an abnormal program termination.
-exp fileoptionalExport molecules to file.
-imp fileoptionalImport molecules from file.
-init_gasoptionalCreate molecules as defined by the reference gas.
-lboptionalEnable automatic load balancing.
-mpi_nn noptionalForce neighbour to neighbour communication among n neighbours, default value 1.
-rnoptionalRandom selection of collision partners, i.e. classical DSMC.
-voptionalEnable verbose mode, i.e. show output on screen.

The command to start a typical DSMC calculation with 4 domains looks like this:
mpiexec -l -n 4 PIDSMC -d DataSet -v

-l and -n are parameters for mpiexec. 4 is the number of MPI ranks that will be started by mpiexec. The number of MPI ranks must be equal to the number of domains. In this example the directory "DataSet" must contain a data set with 4 domains.

After the simulation was started, the program will begin to output status information to the console. This information includes for example the flow time, the number of molecules and events like the initialization of the sample.

PIDSMC uses a hybrid programming model. Each domain is processed by an individual instance of the DSMC program. All instances are communicating via MPI. Each instance uses 4 threads, i.e. 4 CPU cores in parallel. With this hybrid model, PI-DSMC can run efficiently on NUMA machines and computer clusters. In order to run the DSMC programs efficiently, the underlying structure of the computer must be known.

Current server CPUs consists of 1 or more NUMA nodes. The number of MPI ranks per NUMA node multiplied by 4 should be equal to the number of cores per NUMA node. The MPI ranks can be bound to NUMA nodes using the "-numa a b" option.

HINT: the number of NUMA nodes on a machine can be determined with the following command: "numactl -hardware"

HINT: screen1 is a useful tool for PI-DSMC. Screen provides virtual consoles that do not require a user to be logged in to the system. It is convenient to start a simulation in screen and to detach from the console. At a later time screen can be resumed and it is possible to control the program again.

5.2.1 Frontend

After PIDSMC was started, PIDSMC can only be controlled via a frontend called PIDSMCfrontend. The frontend must be started on the computer on which the first MPI rank is running. The table below lists the available command line parameters for PIDSMCfrontend.

name required / optional description
-coptionalcancel all pending commands
-foptionalsend command without asking for confirmation
-hoptionalShow the help screen.
-ioptionalshow information about current simulation
-lb_offoptionaldisable load balancing
-lb_onoptionalenable load balancing
-lb_onceoptionalperform load balancing only once
-qoptionalquit current simulation (does not imply saving the data set)
-rsoptionalreset sample
-soptionalsave data on next occasion
-se_onoptionalenable save on exit
-se_offoptionaldisable save on exit
-tl_on toptionaltime limit - quit at flow time t (does not imply saving the data set)
-tl_offoptionaldisable time limit

If you want to show the status of a simulation type:
PIDSMCfrontend -i

If you want to stop a simulation and save the data set type:
PIDSMCfrontend -se_on -q

After a command was sent by PIDSMCfrontend, PIDSMC will process the command on the next occasion.