Misc

This is a catch-all section for general notes on PyPIConGPU.

Python Concepts

Some common concepts recurr throughout the code and are not explained in detail when used. This aims to give a non-exhaustive list of such patterns.

They are as close to “standard python” as possible, so using a search engine/asking a python expert should also help you understand them.

Type Checking with Typeguard

Use as defined in PEP 484 “Type Hints”. Briefly:

def greeting(name: str) -> str:
    return 'Hello ' + name

Note that these type annotations are not checked by python on their own.

Typechecks are enabled through typeguard, mostly using the annotation @typechecked for classes.

This does not check attribute type annotations, see next section.

Typesafe Class Attributes

Class attributes can use type annotations according to PEP 484, but this is not enforced by typeguard.

The solution is to use properties.

In the code you will see:

class SomeObject:
    attr = util.build_typesafe_property(int)

Now the following hold:

1. Accessing attr before it has been set will raise an exception

2. Assigning attr anything other than an int will raise a TypeError

Note: instead of types you can use specifications provided by import typing, e.g. typing.Optional[int]. Optional vars still have no default, so you have to set them to None explicitly if you don’t want to provide them.

Internally the declaration above is expanded to the equivalent of:

@typechecked
def getter(self) -> int:
    if not hasattr(self, "actual_name_for_var__attr"):
        raise AttributeError("variable is not initialized")
    return getattr(self, "actual_name_for_var__attr")

@typechecked
def setter(self, value: int) -> None:
    setattr(self, "actual_name_for_var__attr", value)

class SomeObject:
    attr = property(getter, setter)

Map-Filter-Reduce

map(), filter(), and reduce() are “higher-order functions” and a basis of the functional programming paradigm (as opposed to loops for iterative processing). You can find an an introduction here (but any other intro is probably fine too).

Boils down to:

[2, 4, 6] == list(map(lambda x: 2*x,
                      [1, 2, 3]))

map() does not return a list directly and must be cast again. (Python, why?)

map() does not work on dictionaries, to for this the dictionary is cast to a set of key-value tuples:

{"a": 3, "b": 0} == dict(map(
    lambda kv_pair: (kv_pair[0], len(kv_pair[1])),
    {"a": [0, 0, 0], "b": []}.items()))

Note: Do not feel obliged to follow this pattern. It is commonly used, because it allows a concise notation, yet it is very much not mandatory.

Tool Support

This is a short list of tools to aid you with development.

Formatting

PEP 8 (formatting guidelines) compliance:

run from repo root flake8 lib/python/ test/python/

Note: Please obey the line length from PEP 8 even if that is annoying at times, this makes viewing files side-by-side much simpler.

Run Tests

go into test/python/picongpu, execute python -m quick (for tests with compilation python -m compiling)

Test Coverage

0. install coverage.py: pip install coverage

1. go to tests: cd test/python/picongpu

2. execute tests, record coverage: coverage run --branch -m quick

3. view reports

   • for PyPIConGPU: find ../../../lib/python/picongpu/pypicongpu/ -name '*.py' | xargs coverage report -m

   • for PICMI: find ../../../lib/python/picongpu/picmi/ -name '*.py' | xargs coverage report -m

   • Goal is 100% coverage (missing sections are printed)

For further options see the coverage.py doc.

API Documentation

Document the API using docstrings, which will be rendered by Sphinx AutoAPI automatically. The result is available from the TOC, although the name is generated to be the python module name: picongpu.pypicongpu.

The configuration is placed in Sphinx’s conf.py, all classes are traversed by using __all__ defined in every __init__.py.

Additional checks (completeness etc.) are NOT employed (and not even possible). So pay attention that you document everything you need.

As everything is passed to Sphinx, docstring should be valid reStructuredText. (reStructuredText quick guide from Sphinx)

You may use Sphinx-exclusive directives too, e.g. :ref:`MARK`.

To document parameters, return values etc. make them compatible with Sphinx’s autodoc. For details please refer to the respective documentation section.

Note

If there is an reStructuredText syntax error during generation (printed as warning after sphinx invocation make html) to debug:

1. visit the respective page

2. click View page source at the top of the page

   (This might be only availble locally, the online version displays Edit on GitHub instead)

An example can be found in the official documentation. The types are implictly given by the type hints, so these can be omitted:

from typeguard import typechecked
import typing


@typechecked
def my_func(a_char: str, num: int) -> typing.List[str]:
    """
    build list with "triangle" of chars

    Accepts a single char and builds a list where the n-th element contains a
    string consisting of n times the given char.
    The list has the given total length.

    This is an example for a section of the PyPIConGPU doc,
    see: :ref:`pypicongpu-misc-apidoc`.
    to show that all Sphinx directives are valid.
    Because this is reStructuredText, (nested) lists require separate
    paragraphs:

    - an item
    - another item

        - nested 1
        - nested 2

    - continuation of top level

    :param a_char: character to use, e.g. "."
    :param num: max, e.g. "3"
    :raises AssertionError: on a_char being not a single character
    :raises AssertionError: if num < 0
    :return: [".", "..", "..."]
    """
    assert 1 == len(a_char)
    assert 0 <= num

    if 0 == num:
        return []

    return my_func(a_char, num - 1) + [num * a_char]

Note that :raises TypeError: is omitted too, because all functions are checked for types anyways.

This produces the following code section:

doc_example.my_func(a_char: str, num: int) → List[str]

build list with “triangle” of chars

Accepts a single char and builds a list where the n-th element contains a string consisting of n times the given char. The list has the given total length.

This is an example for a section of the PyPIConGPU doc, see: API Documentation. to show that all Sphinx directives are valid. Because this is reStructuredText, (nested) lists require separate paragraphs:

• an item

• another item

  • nested 1

  • nested 2

• continuation of top level

Parameters:

• a_char – character to use, e.g. “.”

• num – max, e.g. “3”

Raises:

• AssertionError – on a_char being not a single character

• AssertionError – if num < 0

Returns:

[“.”, “..”, “…”]