PSyData API

PSyclone provides transformations that will insert callbacks to an external library at runtime. These callbacks allow third-party libraries to access data structures at specified locations in the code. The PSyclone wrappers to external libraries are provided with the PSyclone installation. Some example use cases are:

Profiling:

By inserting callbacks before and after a region of code, performance measurements can be added. PSyclone provides wrapper libraries for some common performance profiling tools, see Profiling for details.

Kernel Data Extraction:

PSyclone provides the ability to add callbacks that provide access to all input variables before, and output variables after a kernel invocation. This can be used to automatically create tests for a kernel, or to write a stand-alone driver that just calls one kernel, which can be used for performance tuning. Two example libraries that extract input and output data into either a Fortran binary or a NetCDF file are included with PSyclone (see Extraction Libraries).

Access Verification:

The callbacks can be used to make sure a field declared as read-only is not modified during a kernel call (either because of an incorrect declaration, or because memory is overwritten). The implementation included in PSyclone uses a simple 64-bit checksum to detect changes to a field (and scalar values). See Read-Only Verification for details.

NAN Test:

The callbacks can be used to make sure that all floating point input and output parameters of a kernel are not a NaN (not-a-number) or infinite. See NAN Test for the full description.

In-situ Visualisation:

By giving access to output fields of a kernel, an in-situ visualisation library can be used to plot fields while a (PSyclone-processed) application is running. There is no example library available at this stage, but the API has been designed with this application in mind.

The PSyData API should be general enough to allow these and other applications to be developed and used.

PSyclone provides transformations that will insert callbacks to the PSyData API, for example ProfileTrans, GOceanExtractTrans and LFRicExtractTrans. A user can develop additional transformations and corresponding runtime libraries for additional functionality. Refer to psy_data for full details about the PSyData API.

Read-Only Verification

The PSyData interface is being used to verify that read-only variables in a kernel are not overwritten. The ReadOnlyVerifyTrans (in psyir.transformations.read_only_verify_trans, or the Transformation Reference Guide) uses the dependency analysis to determine all read-only variables (i.e. arguments declared to be read-only in metadata, most implicit arguments in LFRic, grid properties in GOcean). A simple 64-bit checksum is then computed for all these arguments before a kernel call, and compared with the checksum after the kernel call. Any change in the checksum causes a message to be printed at runtime, e.g.:

--------------------------------------
Double precision field b_fld has been modified in main : update
Original checksum:   4611686018427387904
New checksum:        4638355772470722560
--------------------------------------

The transformation that adds read-only-verification to an application can be applied for both the LFRic and GOcean API - no API-specific transformations are required. Below is an example that searches for each loop in an invoke (which will always surround kernel calls) and applies the transformation to each one. This code has been successfully used as a global transformation with the LFRic Gravity Wave miniapp (the executable is named gravity_wave):

def trans(psy):
    from psyclone.psyir.transformations import ReadOnlyVerifyTrans
    from psyclone.psyir.nodes import Loop
    read_only_verify = ReadOnlyVerifyTrans()

    for invoke in psy.invokes.invoke_list:
        schedule = invoke.schedule
        for node in schedule:
            if isinstance(node, Loop):
                read_only_verify.apply(node)

    return psy

Besides the transformation, a library is required to do the actual verification at runtime. There are two implementations of the read-only-verification library included in PSyclone: one for LFRic, and one for GOcean. Both libraries support the environment variable PSYDATA_VERBOSE. This can be used to control how much output is generated by the read-only-verification library at runtime. If the variable is not specified or has the value ‘0’, warnings will only be printed if checksums change. If it is set to ‘1’, a message will be printed before and after each kernel call that is checked. If the variable is set to ‘2’, it will additionally print the name of each variable that is checked.

Read-Only Verification Library for LFRic

This library is contained in lib/read_only/lfric and it must be compiled before compiling any LFRic-based application that uses read-only verification. Compiling this library requires access to the LFRic infrastructure library (since it must implement a generic interface for e.g. the LFRic field class).

The Makefile uses the variable LFRIC_INF_DIR to point to the location where LFRic’s field_mod and integer_field_mod have been compiled. It defaults to the path to location of the pared-down LFRic infrastructure located in a clone of PSyclone repository, <PSYCLONEHOME>/src/psyclone/tests/test_files/dynamo0p3/infrastructure, but this will certainly need to be changed for any user (for instance with PSyclone installation). The LFRic infrastructure library is not used in linking the verification library. The application which uses the read-only-verification library needs to link in the infrastructure library anyway.

Compilation of the library is done by invoking make and setting the required variables:

make LFRIC_INF_DIR=some_path F90=ifort F90FLAGS="--some-flag"

This will create a library called lib_read_only.a.

Read-Only-Verification Library for GOcean

This library is contained in the lib/read_only/dl_esm_inf directory and it must be compiled before linking any GOcean-based application that uses read-only verification. Compiling this library requires access to the GOcean infrastructure library (since it must implement a generic interface for e.g. the dl_esm_inf r2d_field class).

The Makefile uses the variable GOCEAN_INF_DIR to point to the location where dl_esm_inf’s field_mod has been compiled. It defaults to the relative path to location of the dl_esm_inf version included in PSyclone repository as a Git submodule, <PSYCLONEHOME>/external/dl_esm_inf/finite_difference/src. It can be changed to a user-specified location if required (for instance with the PSyclone installation).

The dl_esm_inf library is not used in linking the verification library. The application which uses the read-only-verification library needs to link in the infrastructure library anyway.

Compilation of the library is done by invoking make and setting the required variables:

make GOCEAN_INF_DIR=some_path F90=ifort F90FLAGS="--some-flag"

This will create a library called lib_read_only.a. An executable example for using the GOcean read-only-verification library is included in examples/gocean/eg5/readonly, see Example 5.3: Read-only-verification.

NAN Test

This transformation can be used for both LFRic and GOcean APIs. It will test all input and output parameters of a kernel to make sure they are not NaN or infinite. If they are, an error message like the following is printed, but the program is not aborted:

PSyData: Variable a_fld has the invalid value Inf at index/indices 1 1 in module 'main' region 'update'.

Is uses the function IEEE_IS_FINITE from the ieee_arithmetic module for this test. Note that only floating point numbers will be tested. Integer numbers do not have a bit pattern for ‘infinity’ or NaN.

The runtime libraries for GOcean and LFRic are based on a jinja-template contained in the directory <PSYCLONEHOME>/lib/nan_test. The respective API-specific libraries map the internal field structures to Fortran basic types and call the functions from the base class to handle those.

The relevant libraries for the LFRic and GOcean APIs are contained in the lib/nan_test/lfric and``lib/nan_test/dl_esm_inf`` subdirectories, respectively. For more information on how to build and link these libraries, please refer to the relevant README.md files.

An executable example for using the LFRic read-only-verification library is included in tutorial/practicals/LFRic/building_code/4_psydata directory, see this link for more information.