First, decide where to store input files, a good place might be
~) because it is usually backed up.
Second, decide where to store your output of simulations which needs to be placed on a high-bandwidth, large-storage file system which we will refer to as
As in our compiling from source section, we need a few directories to structure our workflow:
# source code mkdir $HOME/src # temporary build directory mkdir $HOME/build # PIConGPU input files mkdir $HOME/paramSets # PIConGPU simulation output mkdir $SCRATCH/runs
(“too long, didn’t read and know how compiling works”)
pic-create ~/paramSets/originalSet ~/paramSets/myLWFA cd ~/build pic-configure $HOME/paramSets/myLWFA make -j install cd ~/paramSets/myLWFA tbg -s qsub -c submit/0016gpus.cfg -t submit/hypnos-hzdr/k20_profile.tpl $SCRATCH/runs/lwfa_001
1. Create an Input (Parameter) Set¶
# clone the LWFA example to $HOME/paramSets/myLWFA pic-create $PICSRC/examples/LaserWakefield/ $HOME/paramSets/myLWFA
$HOME/paramSets/case001/include/simulation_defines/param/* to change the physical configuration of this parameter set.
$HOME/paramSets/case001/submit/*.cfg to adjust runtime parameters (simulation size, number of GPUs, plugins, ...).
Hint: you can further create parameter sets from parameter sets.
2. Compile Simulation¶
.param files in inputs or changes of parameters in excisting files require a re-compile of PIConGPU.
pic-configure is a wrapper for CMake to quickly specify which parameter set and source version of PIConGPU shall be used.
# go to an empty build directory cd $HOME/build # clean it if necessary rm -rf ../build/* # configure case001 pic-configure $HOME/paramSets/myLWFA # compile PIConGPU with the current parameter set (myLWFA) # - "make -j install" runs implicitly "make -j" and then "make install" # - make install copies resulting binaries to parameter set make -j install
We always configure one parameter set for one compilation.
If you adjust
.param input files just now, you can just go back to
$HOME/build and run
make -j install again without further need to clean the directory or configuration.
3. Run Simulation¶
# go to param set with up-to-date PIConGPU binaries cd $HOME/paramSets/myLWFA # example run for the HPC System "hypnos" using a PBS batch system tbg -s qsub -c submit/0016gpus.cfg -t submit/hypnos-hzdr/k20_profile.tpl $SCRATCH/runs/lwfa_001
This will create the directory
$SCRATCH/runs/lwfa_001 were all simulation output will be written to.
tbg will further create a subfolder
picongpu/ in the directory of the run with the same structure as
myLWFA to archive your input files.
Individual input files, their syntax and usage are explained in the following sections.
pic-create --help for more options during parameter set creation:
pic-create create a new parameter set for simulation input merge default picongpu parameters and a given example's input usage: pic-create [OPTION] [src_dir] dest_dir If no src_dir is set picongpu a default case is cloned -f | --force - merge data if destination already exists -h | --help - show this help message Dependencies: rsync
pic-configure --help for more options during parameter set configuration:
configure create a cmake call for picongpu and get fast access to selected picongpu cmake options usage: configure [OPTION] <parameter_DIRECTORY> If no path_to_CMakeLists.txt is set the directory of this binary is used as source directory. -i | --install - path were picongpu should be installed (default is <parameter_DIRECTORY>) -a | --arch - set cuda architecture (semicolon separated list, e.g.: "20;35;37;52;60") -c | --cmake - overwrite options for cmake (e.g.: -c "-DPIC_VERBOSE=1") -t <presetNumber> - configure this preset from cmakeFlags -h | --help - show this help message
After running configure you can run
ccmake . to set additional compile options (optimizations, debug levels, hardware version, etc.).
This will influence your build done via
You can pass further options to configure PIConGPU directly instead of using
ccmake ., by passing
-c "-DOPTION1=VALUE1 -DOPTION2=VALUE2".
picongpu/ directory of a run can also be reused to clone parameters via
pic-create by using this run as origin directory or to create a new binary with
pic-configure -i $HOME/paramSets/myLWFA2 $SCRATCH/runs/lwfa_001.
tbg --help for more information about the