PIConGPU in 5 Minutes on RoSi

A guide to run, but not understand PIConGPU. It is aimed at users of the high performance computing (HPC) cluster “RoSi” at the HZDR, but should be applicable to other HPC clusters with slight adjustments.

This guide needs shell access (probably via ssh) and git (preinstalled on most systems including RoSi). The RoSi shell can also be used from a browser (currently there is no support for Firefox) at https://rosi.hzdr.de/ (accessible from the intranet). Consider getting familiar with the shell (command line, usually bash) and git. Please also read the tutorial for your local HPC cluster.

Get the Source

Use git to obtain the source and use the current dev branch and put it into ~/src/picongpu:

mkdir -p ~/src
git clone https://github.com/ComputationalRadiationPhysics/picongpu ~/src/picongpu

Note

If you get the error git: command not found load git by invoking module load git and try again. Attention: the example uses the dev branch instead of the latest stable release.

Setup

You need a lot of dependencies.

Luckily, other people already did the work and prepared a profile that you can use. Copy it to your home directory:

cp ~/src/picongpu/etc/picongpu/rosi-hzdr/gpu-v100_picongpu.profile.example ~/gpu-v100_picongpu.profile

This profile determines which part of the HPC cluster (partition, also: queue) – and thereby the compute device(s) (type of CPUs/GPUs) – you will use. This particular profile will use NVIDIA Volta V100 GPUs.

You can view the full list of available profiles on GitHub (look for NAME.profile.example). The system specific directories sometimes contain a README file with some further useful information.

For this guide we will add our scratch directory location to this profile. Edit the profile file using your favorite editor. If unsure use nano: nano ~/gpu-v100_picongpu.profile (save with Control-o, exit with Control-x). Go to the end of the file and add a new line:

export SCRATCH=/bigdata/hplsim/external/<username>

(Please replace <username> with your username.)

Note

This is the location where runtime data and all results will be stored. If you’re not on RoSi make sure you select the correct directory: Consult the documentation of your HPC cluster where to save your data. On HPC clusters this is probably not your home directory.

In the profile file you can also supply additional settings, like your email address and notification settings.

Now activate your profile:

source ~/gpu-v100_picongpu.profile

Warning

You will have to repeat this command every time you want to use PIConGPU on a new shell, i.e. after logging in.

Now test your new profile:

echo $SCRATCH

That should print your data directory. If that works make sure that this directory actually exists by executing:

mkdir -p $SCRATCH
ls -lah $SCRATCH

If you see output similar to this one everything worked and you can carry on:

total 0
drwxr-xr-x  2 <username>    fwt   40 Nov 12 10:09 .
drwxrwxrwt 17 root     root 400 Nov 12 10:09 ..

Create a Scenario

As an example we will use the predefined LaserWakefield example. Create a directory and copy it:

mkdir -p ~/picongpu-projects/laser-wakefield-example
pic-create $PIC_EXAMPLES/LaserWakefield ~/picongpu-projects/laser-wakefield-example/try01
cd ~/picongpu-projects/laser-wakefield-example/try01

Usually you would now adjust the files in the newly created directory ~/picongpu-projects/laser-wakefield-example/try01 – for this introduction we will use the parameters as provided.

Note

The command pic-create and the variable $PIC_EXAMPLES have been provided because you loaded the file ~/gpu-v100_picongpu.profile in the previous step. If this fails (printing pic-create: command not found), make sure you load the PIConGPU profile by executing source ~/gpu-v100_picongpu.profile.

Compile and Run

Now use a compute node. Your profile provides a helper command for that:

getDevice

(You can now run hostname to see which node you are using.)

Now build the scenario:

# switch to the scenario directory if you haven't already
cd ~/picongpu-projects/laser-wakefield-example/try01
pic-build -j 6

This will take a while, go grab a coffee. The -j 6 behind pic-build tells the compiler to compile in parallel. You can speed up the compile process by requesting more cores here. However this will also require more memory for the compile process. If this fails, read the manual or ask a colleague.

After a successful build, run (still on the compute node, still inside your scenario directory):

tbg -s bash -t $PICSRC/etc/picongpu/bash/mpiexec.tpl -c etc/picongpu/1.cfg $SCRATCH/laser-wakefield-example/try01/run01

tbg: tool provided by PIConGPU
bash: the “submit system”, e.g. use sbatch for slurm
$PICSRC: the path to your PIConGPU source code, automatically set when sourcing gpu-v100_picongpu.profile
$PICSRC/etc/picongpu/bash/mpiexec.tpl: options for the chosen submit system
etc/picongpu/1.cfg: runtime options (number of GPUs, etc.)
$SCRATCH/laser-wakefield-example/try01/run01: not-yet-existing destination for your result files

Note

Usually you would use the workload manager (SLURM on RoSi) to submit your jobs instead of running them interactively like we just did. You can try that with:

# go back to the login node
exit
hostname
# ...should now display one of rosi5, rosi4, ...

# resubmit your simulation with a new directory:
tbg -s sbatch -c etc/picongpu/1.cfg -t etc/picongpu/rosi-hzdr/gpu-v100.tpl $SCRATCH/laser-wakefield-example/try01/run02

This will print a confirmation message (e.g. Submitted batch job 3769365), but no output of PIConGPU itself will be printed. Using squeue -u $USER you can view the current status of your job.

Note that we not only used a different “submit system” sbatch, but also changed the template file to etc/picongpu/rosi-hzdr/gpu-v100.tpl. (This template file is directly located in your project directory.) Both profile and template file are built for the same compute device, the NVIDIA Volta “V100” GPU.

Examine the Results

Results are located at $SCRATCH/laser-wakefield-example/try01/run01.

To view pretty pictures from a linux workstation you can use the following process (execute on your workstation, not the HPC cluster):

# Create a “mount point” (empty directory)
mkdir -p ~/mnt/scratch

# Mount the data directory using sshfs
sshfs -o default_permissions -o idmap=user -o uid=$(id -u) -o gid=$(id -g) rosi5:DATADIR ~/mnt/scratch/

Substitute DATADIR with the full path to your data (scratch) directory, e.g. /bigdata/hplsim/scratch/alice.

Browse the directory using a file browser/image viewer. Check out ~/mnt/scratch/laser-wakefield-example/try01/run01/simOutput/pngElectronsYX/ for image files.

On RoSi, you can also use the in-browser file manager to access your files. Go to https://rosi.hzdr.de, log in, click on the Home Directory icon under Files, and navigate to /bigdata/hplsim/scratch/<username>/laser-wakefield-example/try01/run01/simOutput/pngElectronsYX/.

openPMD

The PNG output is great for having a quick look at the example results, but it is a rather inefficient way of generating and storing your simulation results. In general, we write simulation data into ADIOS2 (.bp, .bp5, .bp4 extensions) or HDF5 (.h5) files with parallel I/O that follow the structure defined by the community openPMD standard. The output can be configured using the openPMD plugin. The example you just run also provides some basic openPMD output that can be now found under simOutput/openPMD/. You should make yourself familiar with how to read and visualize such data. Here are some useful resources:

openPMD standard and the PIC extension to the standard
openPMD API docs API for reading and writing openPMD output. Powerful tool, its proper use requires some familiarity with the standard itself.
openPMD-viewer Data visualization Python library built on top of the API.
openPMD-scipp Easy visualization of mesh data (fields) using the plopp and scipp Python libraries.
openPMD pandas support Straightforward solution for working with particle data.
openpmd-ls Command for getting an overview over the content of an openPMD output via command line.
bpls Powerful tool for inspecting individual ADIOS2 files via command line.

We recommend trying out the openPMD-viewer and the openPMD-scipp tools first, followed by a quick read through the standard itself and the python part of the first read API example.

When directly using the openPMD API make sure you make yourself familiar with when you need to call series.flush(). See Flush Chunk and Deferred Data API Contract for more details. Forgetting it or modifying the data before flush was performed is a common user mistake that can lead to very confusing results.

In addition to the standard openPMD output PIConGPU offers some reduced diagnostics via other plugins including the very powerful flexible binning plugin.