Basics¶

Preparation¶

First, decide where to store input files, a good place might be $HOME (~) because it is usually backed up. Second, decide where to store your output of simulations which needs to be placed on a high-bandwidth, large-storage file system which we will refer to as $SCRATCH.

As in our compiling from source section, we need a few directories to structure our workflow:

# source code
mkdir $HOME/src
# temporary build directory
mkdir $HOME/build

# PIConGPU input files
mkdir $HOME/paramSets
# PIConGPU simulation output
mkdir $SCRATCH/runs

Step-by-Step¶

TL;DR¶

(“too long, didn’t read and know how compiling works”)

pic-create ~/paramSets/originalSet ~/paramSets/myLWFA

cd ~/build
pic-configure $HOME/paramSets/myLWFA
make -j install

cd ~/paramSets/myLWFA
tbg -s qsub -c submit/0016gpus.cfg -t submit/hypnos-hzdr/k20_profile.tpl $SCRATCH/runs/lwfa_001

1. Create an Input (Parameter) Set¶

# clone the LWFA example to $HOME/paramSets/myLWFA
pic-create $PICSRC/examples/LaserWakefield/ $HOME/paramSets/myLWFA

Now edit $HOME/paramSets/case001/include/simulation_defines/param/* to change the physical configuration of this parameter set.

Now edit $HOME/paramSets/case001/submit/*.cfg to adjust runtime parameters (simulation size, number of GPUs, plugins, …).

Hint: you can further create parameter sets from parameter sets.

2. Compile Simulation¶

New .param files in inputs or changes of parameters in excisting files require a re-compile of PIConGPU. Our script pic-configure is a wrapper for CMake to quickly specify which parameter set and source version of PIConGPU shall be used.

# go to an empty build directory
cd $HOME/build
# clean it if necessary
rm -rf ../build/*

# configure case001
pic-configure $HOME/paramSets/myLWFA

# compile PIConGPU with the current parameter set (myLWFA)
# - "make -j install" runs implicitly "make -j" and then "make install"
# - make install copies resulting binaries to parameter set
make -j install

We always configure one parameter set for one compilation. If you adjust .param input files just now, you can just go back to $HOME/build and run make -j install again without further need to clean the directory or configuration.

3. Run Simulation¶

# go to param set with up-to-date PIConGPU binaries
cd $HOME/paramSets/myLWFA

# example run for the HPC System "hypnos" using a PBS batch system
tbg -s qsub -c submit/0016gpus.cfg -t submit/hypnos-hzdr/k20_profile.tpl $SCRATCH/runs/lwfa_001

This will create the directory $SCRATCH/runs/lwfa_001 were all simulation output will be written to. tbg will further create a subfolder picongpu/ in the directory of the run with the same structure as myLWFA to archive your input files.

Further Reading¶

Individual input files, their syntax and usage are explained in the following sections.

See pic-create --help for more options during parameter set creation:

pic-create create a new parameter set for simulation input
merge default picongpu parameters and a given example's input

usage: pic-create [OPTION] [src_dir] dest_dir
If no src_dir is set picongpu a default case is cloned

-f | --force         - merge data if destination already exists
-h | --help          - show this help message

Dependencies: rsync

See pic-configure --help for more options during parameter set configuration:

configure create a cmake call for picongpu 
and get fast access to selected picongpu cmake options

usage: configure [OPTION] <parameter_DIRECTORY>
If no path_to_CMakeLists.txt is set the directory of this binary is used as source directory.

-i | --install       - path were picongpu should be installed (default is <parameter_DIRECTORY>)
-a | --arch          - set cuda architecture (semicolon separated list, e.g.: "20;35;37;52;60")
-c | --cmake         - overwrite options for cmake (e.g.: -c "-DPIC_VERBOSE=1")
-t <presetNumber>    - configure this preset from cmakeFlags
-h | --help          - show this help message

After running configure you can run ccmake . to set additional compile options (optimizations, debug levels, hardware version, etc.). This will influence your build done via make.

You can pass further options to configure PIConGPU directly instead of using ccmake ., by passing -c "-DOPTION1=VALUE1 -DOPTION2=VALUE2".

The picongpu/ directory of a run can also be reused to clone parameters via pic-create by using this run as origin directory or to create a new binary with configure: e.g. pic-configure -i $HOME/paramSets/myLWFA2 $SCRATCH/runs/lwfa_001.

See tbg --help for more information about the tbg tool.