PI-DSMC V3 (CPU)

5. Processing

This section describes the usage of the programs that are part of the PI-DSMC program package. There are separate programs: PIDS2geo, PIDS3geo, PIDSMC and PIDSMCfrontend. None of these programs has a graphical user interface. All programs should be started from the command line.

5.1 Pre-Processing with PIDS2geo/PIDS3geo

The programs PIDS2geo and PIDS3geo are used to generate 2D and 3D data sets. Both programs require a configuration file that can be created with PIDSMCconfig. The table below lists the available command line parameters of PIDS2geo and PIDS3geo.

name	required / optional	description
-g config_file	required	Name of the configuration file
-d directory	optional	Directory where the data set will be saved. The current directory will be used if this parameter is not specified.
-h	optional	Display help
-p	optional	Generate a new dataset.
-np x	optional	Use x CPUs for geometry generation.

If PIDS2geo is executed with the -g option only, a menu with the available options is displayed. The options are:

1. Create new data set


2. Display information about the data set

The option "display information" will show some useful information about the specified data set like the number of molecules in each domain, the grid size and the number of real molecules represented by each simulated molecule.

5.2 DSMC Processing

After the generation of the data set, the DSMC calculation can be started.

name	required / optional	description
-d directory	required	Directory where the data is stored. The current directory will be used if this parameter is not specified.
-h	optional	Display help
-tl time	optional	Abort the simulation when the specified flow time [s] was reached.
-dll filename	optional	Enable the DLL interface. The DLL with the specified name will be loaded.
-shm	optional	Use shared memory to save memory.
-se	optional	Save data set before the program aborts.
-numa a b	optional	Bind a instances to each NUMA node starting with node b. This prevents the OS from moving processes around.
-cdt	optional	Use constant time step, i.e. classical DSMC.
-clean	optional	Remove shared memory pages after an abnormal program termination.
-exp file	optional	Export molecules to file.
-imp file	optional	Import molecules from file.
-init_gas	optional	Create molecules as defined by the reference gas.
-lb	optional	Enable automatic load balancing.
-mpi_nn n	optional	Force neighbour to neighbour communication among n neighbours, default value 1.
-rn	optional	Random selection of collision partners, i.e. classical DSMC.
-v	optional	Enable verbose mode, i.e. show output on screen.

The command to start a typical DSMC calculation with 4 domains looks like this:
mpiexec -l -n 4 PIDSMC -d DataSet -v

-l and -n are parameters for mpiexec. 4 is the number of MPI ranks that will be started by mpiexec. The number of MPI ranks must be equal to the number of domains. In this example the directory "DataSet" must contain a data set with 4 domains.

After the simulation was started, the program will begin to output status information to the console. This information includes for example the flow time, the number of molecules and events like the initialization of the sample.

PIDSMC uses a hybrid programming model. Each domain is processed by an individual instance of the DSMC program. All instances are communicating via MPI. Each instance uses 4 threads, i.e. 4 CPU cores in parallel. With this hybrid model, PI-DSMC can run efficiently on NUMA machines and computer clusters. In order to run the DSMC programs efficiently, the underlying structure of the computer must be known.

Current server CPUs consists of 1 or more NUMA nodes. The number of MPI ranks per NUMA node multiplied by 4 should be equal to the number of cores per NUMA node. The MPI ranks can be bound to NUMA nodes using the "-numa a b" option.

HINT: the number of NUMA nodes on a machine can be determined with the following command: "numactl -hardware"

HINT: screen1 is a useful tool for PI-DSMC. Screen provides virtual consoles that do not require a user to be logged in to the system. It is convenient to start a simulation in screen and to detach from the console. At a later time screen can be resumed and it is possible to control the program again.

5.2.1 Frontend

After PIDSMC was started, PIDSMC can only be controlled via a frontend called PIDSMCfrontend. The frontend must be started on the computer on which the first MPI rank is running. The table below lists the available command line parameters for PIDSMCfrontend.

name	required / optional	description
-c	optional	cancel all pending commands
-f	optional	send command without asking for confirmation
-h	optional	Show the help screen.
-i	optional	show information about current simulation
-lb_off	optional	disable load balancing
-lb_on	optional	enable load balancing
-lb_once	optional	perform load balancing only once
-q	optional	quit current simulation (does not imply saving the data set)
-rs	optional	reset sample
-s	optional	save data on next occasion
-se_on	optional	enable save on exit
-se_off	optional	disable save on exit
-tl_on t	optional	time limit - quit at flow time t (does not imply saving the data set)
-tl_off	optional	disable time limit

If you want to show the status of a simulation type:
PIDSMCfrontend -i

If you want to stop a simulation and save the data set type:
PIDSMCfrontend -se_on -q

After a command was sent by PIDSMCfrontend, PIDSMC will process the command on the next occasion.

Read on...

• 1. Introduction
• 2. System requirements
• 3. Installation
• 4. Configuration
• 5. Processing
• 6. Post-processing
• 7. Getting started

Please note: This page applies to PI-DSMC V3 which runs on CPUs.