PSyData API#

PSyclone provides transformations that will insert callbacks to an external library at runtime. These callbacks allow third-party libraries to access data structures at specified locations in the code.

Introduction to PSyData Classes#

The PSyData API imports a user-defined data type from a PSyData module and creates an instance of this data type for each code region. It then adds a sequence of calls to methods in that instance. A simplified example:

USE psy_data_mod, ONLY: PSyDataType
TYPE(PSyDataType), target, save :: psy_data

CALL psy_data%PreStart("update_field_mod", "update_field_code", 1, 1)
...

In order to allow several different callback libraries to be used at the same time, for example to allow in-situ visualisation at the same time as checking that read-only values are indeed not modified, different module names and data types must be used.

PSyData divides its application into different classes. For example, the class “profile” is used for all profiling tools (e.g. TAU, Vernier, DrHook, or the NVIDIA profiling tools). This class name is used as a prefix for the module name, the PSyDataType and functions. So if a profiling application is linked the above code will actually look like this:

USE profile_psy_data_mod, ONLY: profile_PSyDataType
TYPE(profile_PSyDataType), target, save :: profile_psy_data

CALL profile_psy_data%PreStart("update_field_mod", "update_field_code", 1, 1)

Note

While adding the class prefix to the name of the instance variable is not necessary, it helps improves the readability of the created code.

The list of valid class prefixes is specified in the configuration file. It can be extended by the user to support additional classes:

[DEFAULT]
...
VALID_PSY_DATA_PREFIXES = profile, extract

The class prefixes supported at the moment are:

Class Prefix	Description
profile	All libraries related to profiling tools like TAU, DrHook, NVIDIA’s profiling tools, Vernier etc. See Profiling for details.
extract	For libraries used for kernel data extraction. See PSy Kernel Extractor (PSyKE) for details.
read_only_verify	Use a checksum to verify that variables that are read-only are not modified in a subroutine.

In the following documentation the string PREFIX_ is used to indicate the class prefix used (e.g. profile).

Note

The transformations for profiling or kernel extraction allow to overwrite the default-prefix (see Passing Parameters From the User to the Node Constructor). This can be used to link with two different libraries of the same class at the same time, e.g. you could use drhook_profile and nvidia_profile as class prefixes. However, this would also require that the corresponding wrapper libraries be modified to use this new prefix.

Full Example#

The following example shows the full code created by PSyclone for a kernel extraction (excluding declarations for user data). This code is automatically inserted by the various transformations that are based on PSyDataTrans, like ProfileTrans and GOCeanExtractTrans. More details can be found in PSyDataTrans.

USE extract_psy_data_mod, ONLY: extract_PSyDataType
TYPE(extract_PSyDataType), target, save :: extract_psy_data

CALL extract_psy_data%PreStart("update_field_mod", "update_field_code", 1, 1)
CALL extract_psy_data%PreDeclareVariable("a_fld", a_fld)
CALL extract_psy_data%PreDeclareVariable("b_fld", b_fld)
CALL extract_psy_data%PreEndDeclaration
CALL extract_psy_data%ProvideVariable("a_fld", a_fld)
CALL extract_psy_data%PreEnd

DO j=1,jstop+1
  DO i=1,istop+1
    CALL update_field_code(i, j, a_fld%data, b_fld%data)
  END DO
END DO

CALL extract_psy_data%PostStart
CALL extract_psy_data%ProvideVariable("b_fld", b_fld)
CALL extract_psy_data%PostEnd

A user defined type PREFIX_PSyDataType is imported from the module PREFIX_psy_data_mod.
A static variable of type PREFIX_PSyDataType is declared.
PreStart is called to indicate that a new that a new instrumented code region is started.
PreDeclareVariable is called for each variable passed to the data API either before or after the instrumented region.
PreEndDeclaration is called to indicate the end of the variable declarations.
ProvideVariable is called for each variable to be passed to the wrapper library before the instrumented region.
PreEnd is called to signal the end of all PSyData activity before the instrumented region.
Then the actual instrumented code region is entered.
After the instrumented region, a call to PostStart is added to indicate that all further data output occurs after the instrumented region.
For each variable to be passed on to the wrapper library after the instrumented region a call to ProvideVariable is added.
A call to PostEnd is added once all variables have been provided.

Note

Depending on the options provided to the PSyData transformation, some of the calls might not be created. For example, for a performance profiling library no variables will be declared or provided.

API#

This section describes the actual PSyData API in detail. It contains all functions, data types and methods that need to be implemented for a wrapper library.

The PSyData API includes two function calls that allow initialisation and shut down of a wrapper library. These two calls must be inserted manually into the program, and their location might depend on the wrapper library used - e.g. some libraries might need to be initialised before MPI_Init is called, others might need to be called after. Similarly the shutdown function might need to be called before or after MPI_Finalize.

Note

Not all PSyData libraries require these calls (for example, the NVIDIA profiling library does not need it, the data extraction libraries do not need it, … ), but any PSyData library should include (potentially empty) subroutines in order to avoid linking problems. It is the responsibility of the user to decide if the initialisation and shutdown calls are unnecessary.

Similarly, the PSyData API also includes two function calls that allow programmatic control of whether or not the underlying data-capture mechanism is active. If this functionality is required then these calls must be inserted manually into the program. They are intended to be used e.g. when profiling only a part of a program’s execution or perhaps to switch on/off output of data for on-line visualisation. As with the initialisation and shutdown subroutines, any PSyData library should include implementations of these routines, even if they are empty.

Note

Currently only the profiling wrapper libraries and read-only-verification libraries implement the Start and Stop routines. Wider support for all PSyData-based APIs will be addressed in Issue #824.

Init and Shutdown Functions#

PREFIX_PSyDataInit()#

Initialises the wrapper library used. It takes no parameters, and must be called exactly once before any other PSyData-related function is invoked. Example:

use profile_psy_data_mod, only : profile_PSyDataInit
...
call profile_PSyDataInit()

PREFIX_PSyDataShutdown()#

Cleanly shuts down the wrapper library used. It takes no parameters, and must be called exactly once. No more PSyData-related functions can be called after PSyDataShutdown has been executed. Example:

use profile_psy_data_mod, only : profile_PSyDataShutdown
...
call profile_PSyDataShutdown()

Start and Stop Functions#

PREFIX_PSyDataStart()#

Currently not implemented in the kernel extraction wrapper.

Starts or enables the PSyData library so that subsequent calls to the API cause data to be output. For instance, if we have a time-stepping application where timestep holds the value of the current time step then we could turn on profiling after the first 5 steps by doing:

use profile_psy_data_mod, only: profile_PSyDataStart
...
if(timestep == 6) call profile_PSyDataStart()

(Assuming that profiling was disabled at application start by the runtime environment or by a call to profile_PSyDataStop - see below.)

This routine may be called any number of times but must be after PSyDataInit() and before PSyDataShutdown() (if present).

PREFIX_PSyDataStop()#

Currently not implemented in the kernel extraction wrapper.

Stops or disables the PSyData library so that subsequent calls to the PSyData API have no effect. Continuing the above time-stepping example, we could turn off profiling after time step 10 by doing:

use profile_psy_data_mod, only: profile_PSyDataStop
...
if(timestep == 11) call profile_PSyDataStop()

This routine may be called any number of times but must be after PSyDataInit() and before PSyDataShutdown() (if present).

`PREFIX_PSyDataType`#

The library using the PSyData API must provide a user-defined data type called PREFIX_PSyDataType. It is up to the application how this variable is used. PSyclone will declare the variables to be static, meaning that they can be used to accumulate data from call to call. An example of the PSyDataType can be found in the example extraction code (see lib/extract/binary/dl_esm_inf, lib/extract/ascii/dl_esm_inf, lib/extract/netcdf/dl_esm_inf, or Extraction Libraries for a detailed description), any of the profiling wrapper libraries (all contained in lib/profiling) or the read_only wrappers (in lib/read_only).

PreStart(this, module_name, kernel_name, num_pre_vars, num_post_vars)#

This is the first method called when the instrumented region is executed. It takes 4 parameters (besides the implicit PREFIX_PSyDataType instance):

module_name: This is the name of the module in which the original Fortran source code is contained. Together with kernel_name it can be used to create a unique name for each instrumented region.
kernel_name: The name of the kernel that is being executed.
num_pre_vars: This is the number of variables that will be supplied using ProvideVar before the instrumented region is executed.
num_post_vars: This is the number of variables that will be supplied using ProvideVar after the instrumented region is executed. The sum num_pre_vars+num_post_vars is the number of variable declarations that will follow.

Typically the static PREFIX_PSyDataType instance can be used to store the module and kernel names if they are required later, or to allocate arrays to store variable data. This call is always created, even if no variables are to be provided.

PreDeclareVariable(this, name, value)#

This method is called for each variable that will be written before or after the user-instrumented region. If a variable is written both before and after the region, the transformations will add two calls to PreDeclareVariable (it can be useful to provide a variable using a different name before and after, see Extraction Libraries). If no variables are to be provided to the wrapper library, this call will not be created (and there is no need to implement this function in a wrapper library).

name: This is the name of the variable as a string.
value: This is the actual content of the variable.

The same call is used for different arguments, so a generic interface is recommended to distinguish between the data types provided. The kernel extraction writer (see Extraction Libraries) uses the following declaration (with types defined in the dl_esm_inf library):

generic, public :: PreDeclareVariable => DeclareScalarInteger, &
                                         DeclareScalarReal,    &
                                         DeclareFieldDouble
...
subroutine DeclareScalarInteger(this, name, value)
    implicit none
    class(extract_PSyDataType), intent(inout), target :: this
    character(*), intent(in) :: name
    integer, intent(in) :: value
...
subroutine DeclareScalarReal(this, name, value)
    implicit none
    class(extract_PSyDataType), intent(inout), target :: this
    character(*), intent(in) :: name
    real, intent(in) :: value
...
subroutine DeclareFieldDouble(this, name, value)
    use field_mod, only : r2d_field
    implicit none
    class(extract_PSyDataType), intent(inout), target :: this
    character(*), intent(in) :: name
    type(r2d_field), intent(in) :: value
...

PreEndDeclaration(this)#: Called once all variables have been declared. This call is only inserted if any variables are to be provided either before or after the instrumented region (thus this call is not created for performance profiling).

ProvideVariable(this, name, value)#

This method is called for each variable to be provided to the runtime library.

name: This is the name of the variable as a string.
value: This is the actual content of the variable.

The same method ProvideVariable is called to provide variable name and content before and after the user-instrumented region. Again it is expected that a library using the API will provide a generic interface to distinguish between the various possible data types, which will be different for each infrastructure library:

generic, public :: ProvideVariable => WriteScalarInteger, &
                                      WriteScalarReal,    &
                                      WriteFieldDouble

PreEnd(this)#: The method PreEnd is called after all variables have been provided before the instrumented region. This call is also not inserted if no variables are provided.

PostStart(this)#: This is the first call after the instrumented region. It does not take any parameters, but the static PREFIX_PSyDataType instance can be used to store the name and number of variables if required. This will be followed by calls to ProvideVariable, which is described above. This call is not used if no variables are provided.

PostEnd(this)#: This method is the last call after an instrumented region. It indicates that all variables have been provided. It will always be created, even if no variables are to be provided.

Note

Only the PreDataStart call takes the module- and region-name as parameters. If these names are required by the PSyData runtime library in different calls, they must be stored in the PSyData object so that they are available when required.

`PSyDataTrans`#

Any transformation that uses the PSyData API works by inserting a special node into the PSyclone tree representation of the program. Only at program creation time is the actual code created that implements the API. The PSyDataTrans transformation contained in psyir/transformations/psy_data_trans.py is the base class for other transformations like profiling and kernel data extraction. All derived transformations mostly add specific validations, and provide parameters to PSyDataTrans, including the class of the node to insert. After passing validation, PSyDataTrans creates an instance of the class requested, and inserts it into the tree.

class psyclone.psyir.transformations.PSyDataTrans(node_class=<class 'psyclone.psyir.nodes.psy_data_node.PSyDataNode'>)[source]

Create a PSyData region around a list of statements. For example:

>>> from psyclone.parse.algorithm import parse
>>> from psyclone.parse.utils import ParseError
>>> from psyclone.psyGen import PSyFactory
>>> api = "gocean"
>>> ast, invoke_info = parse(SOURCE_FILE, api=api)
>>> psy = PSyFactory(api).create(invoke_info)
>>>
>>> from psyclone.psyir.transformations import PSyDataTrans
>>> data_trans = PSyDataTrans()
>>>
>>> schedule = psy.invokes.get('invoke_0').schedule
>>> # Uncomment the following line to see a text view of the schedule
>>> # print(schedule.view())
>>>
>>> # Enclose all children within a single PSyData region
>>> data_trans.apply(schedule.children)
>>> # Uncomment the following line to see a text view of the schedule
>>> # print(schedule.view())
>>> # Or to use custom region name:
>>> data_trans.apply(schedule.children,
...                  {"region_name": ("module","region")})

Parameters:: node_class (psyclone.psyir.nodes.ExtractNode) – The Node class of which an instance will be inserted into the tree (defaults to PSyDataNode).

apply(nodes, options=None)[source]

Apply this transformation to a subset of the nodes within a schedule - i.e. enclose the specified Nodes in the schedule within a single PSyData region.

Note that if the nodes are within a routine that previously had the pure attribute, this attribute is removed.

Parameters:

nodes (psyclone.psyir.nodes.Node or list of psyclone.psyir.nodes.Node) – can be a single node or a list of nodes.
options (Optional[Dict[str, Any]]) – a dictionary with options for transformations.
options["prefix"] (str) – a prefix to use for the PSyData module name (PREFIX_psy_data_mod) and the PSyDataType (PREFIX_PSYDATATYPE) - a “_” will be added automatically. It defaults to “”.
options["region_name"] ((str,str)) – an optional name to use for this PSyData area, provided as a 2-tuple containing a location name followed by a local name. The pair of strings should uniquely identify a region unless aggregate information is required (and is supported by the runtime library).

get_unique_region_name(nodes, options)[source]

This function returns the region and module name. If they are specified in the user options, these names will just be returned (it is then up to the user to guarantee uniqueness). Otherwise a name based on the module and invoke will be created using indices to make sure the name is unique.

Parameters:

nodes (list of psyclone.psyir.nodes.Node) – a list of nodes.
options (Dict[str, Any]) – a dictionary with options for transformations.
options["region_name"] ((str,str)) – an optional name to use for this PSyData area, provided as a 2-tuple containing a location name followed by a local name. The pair of strings should uniquely identify a region unless aggregate information is required (and is supported by the runtime library).

property name

This function returns the name of the transformation. It uses the Python 2/3 compatible way of returning the class name as a string, which means that the same function can be used for all derived classes.

Returns:: the name of this transformation as a string.
Return type:: str

validate(nodes, options=None)[source]

Checks that the supplied list of nodes is valid, that the location for this node is valid (not between a loop-directive and its loop), that there aren’t any name clashes with symbols that must be imported from the appropriate PSyData library and finally, calls the validate method of the base class.

Parameters:

nodes ((list of) psyclone.psyir.nodes.Loop) – a node or list of nodes to be instrumented with PSyData API calls.
options (Optional[Dict[str, Any]]) – a dictionary with options for transformations.
options["prefix"] (str) – a prefix to use for the PSyData module name (PREFIX_psy_data_mod) and the PSyDataType (PREFIX_PSYDATATYPE) - a “_” will be added automatically. It defaults to “”.
options["region_name"] ((str,str)) – an optional name to use for this PSyData area, provided as a 2-tuple containing a location name followed by a local name. The pair of strings should uniquely identify a region unless aggregate information is required (and is supported by the runtime library).

Raises:

TransformationError – if the supplied list of nodes is empty.
TransformationError – if the PSyData node is inserted between an OpenMP/ACC directive and the loop(s) to which it applies.
TransformationError – if the ‘prefix’ or ‘region_name’ options are not valid.
TransformationError – if there will be a name clash between any existing symbols and those that must be imported from the appropriate PSyData library.
TransformationError – if the target nodes are within an ELEMENTAL routine.

`PSyDataNode`#

This is the base class for any node that is being inserted into PSyclone’s program tree to use the PSyData API. The derived classes will typically control the behaviour of PSyDataNode by providing additional parameters.

class psyclone.psyir.nodes.PSyDataNode(ast=None, children=None, parent=None, options=None)[source]

This class can be inserted into a schedule to instrument a set of nodes. Instrument means that calls to an external library using the PSyData API will be inserted before and after the child nodes, which will give that library access to fields and the fact that a region is executed. This can be used, for example, to add performance profiling calls, in-situ visualisation of data, or for writing fields to a file (e.g. for creating test cases, or using driver to run a certain kernel only). The node allows specification of a class string which is used as a prefix for the PSyData module name (prefix_psy_data_mod) and for the PSyDataType (prefix_PSyDataType).

Parameters:

ast (sub-class of fparser.two.Fortran2003.Base) – reference into the fparser2 parse tree corresponding to this node.
children (list[psyclone.psyir.nodes.Node]) – the PSyIR nodes that are children of this node. These will be made children of the child Schedule of this PSyDataNode.
parent (psyclone.psyir.nodes.Node) – the parent of this node in the PSyIR tree.
options (Optional[dict[str, Any]]) – a dictionary with options for transformations.
options["prefix"] (str) – a prefix to use for the PSyData module name (prefix_psy_data_mod) and the PSyDataType (prefix_PSyDataType) - a “_” will be added automatically. It defaults to “”, which means the module name used will just be psy_data_mod, and the data type PSyDataType.
options["region_name"] (tuple[str,str]) – an optional name to use for this PSyDataNode, provided as a 2-tuple containing a module name followed by a local name. The pair of strings should uniquely identify a region unless aggregate information is required (and is supported by the runtime library).

Raises:

InternalError – if a prefix is specified that is not listed in the configuration file.

add_psydata_class_prefix(symbol)[source]

Returns a string prefixed with the class-specific prefix, e.g. if the prefix string is “profile”, then the symbol “PSyDataType” will become “profilePSyDataType”. Typically the _prefix will contain a trailing “_”.

Parameters:: symbol (str) – the symbol name to add the prefix to.
Returns:: the symbol name with the class string as prefix.
Return type:: str

classmethod create(children, symbol_table, ast=None, options=None)[source]

Creates a new (sub-class of a) PSyData node with the supplied ‘children’ nodes as its children. The symbols used by the PSyData API are added to the supplied symbol table. This is a class method so that it acts as a factory for the various sub-classes of PSyDataNode.

Parameters:

children (list[psyclone.psyir.nodes.Node]) – the PSyIR nodes that will become children of the new PSyData node.
symbol_table (psyclone.psyir.symbols.SymbolTable) – the associated SymbolTable to which symbols must be added.
options (Optional[dict[str, Any]]) – a dictionary with options for transformations.
options[prefix"] (str) – a prefix to use for the PSyData module name (prefix_psy_data_mod) and the PSyDataType (prefix_PSyDataType) - a “_” will be added automatically. It defaults to “”, which means the module name used will just be psy_data_mod, and the data type PSyDataType.
options["region_name"] (tuple[str,str]) – an optional name to use for this PSyDataNode, provided as a 2-tuple containing a module name followed by a local name. The pair of strings should uniquely identify a region unless aggregate information is required (and is supported by the runtime library).

Parent ast:

reference to fparser2 parse tree for the routine being instrumented with PSyData calls.

Raises:

TypeError – if the supplied children or symbol table are not of the correct type.

property fortran_module

Returns:: name of the PSyData interface Fortran module.
Return type:: str

generate_symbols(symbol_table)[source]

Generate the necessary symbols to import and use this PSyDataNode in the provided symbol_table if they don’t already exist.

Parameters:: symbol_table (psyclone.psyir.symbols.SymbolTable) – the associated SymbolTable to which symbols must be added.

lower_to_language_level(options=None)[source]

Lowers this node (and all children) to language-level PSyIR. The PSyIR tree is modified in-place. This PSyDataNode is replaced by a pair of Fortran-specific CodeBlocks (representing the calls to the start and stop procedures) with the body (children) of the PSyDataNode inserted between them. This use of CodeBlocks means that currently only the Fortran backend is capable of producing code representing the PSyDataNode.

Parameters:

options (Optional[dict[str, Any]]) – dictionary of the PSyData generation options.
options["pre_var_list"] (list[tuple[str, str]]) – container- and variable-names to be supplied before the first child. The container names are supported to be able to handle variables that are imported from a different container (module in Fortran).
options["post_var_list"] (list[tuple[str, str]]) – container- and variable-names to be supplied after the last child. The container names are supported to be able to handle variables that are imported from a different container (module in Fortran).
options["pre_var_postfix"] (str) – an optional postfix that will be added to each variable name in the pre_var_list.
options["post_var_postfix"] (str) – an optional postfix that will be added to each variable name in the post_var_list.

Returns:

the lowered version of this node.

Return type:

psyclone.psyir.node.Node

Raises:

GenerationError – if the node is not inside a Routine.

property module_name

Returns the _module_name of this PSyDataNode.

Returns:: the _module_name of this PSyDataNode.
Return type:: str

property options

Returns:: the option dictionary of this class.
Return type:: dict[str,Any]

property prefix

Returns the _prefix member of this PSyDataNode.

Returns:: the _prefix member of this PSyDataNode.
Return type:: str

property psy_data_body

Returns:: the Schedule associated with this PSyData region.
Return type:: psyclone.psyir.nodes.Schedule
Raises:: InternalError – if this PSyData node does not have a Schedule as its one and only child.

property region_identifier

Returns:: the unique region identifier, which is a tuple consisting of the module name and region name.
Return type:: tuple[str, str]

property region_name

Returns the _region_name of this PSyDataNode.

Returns:: the _region_name of this PSyDataNode.
Return type:: str

set_region_identifier(module_name, region_name)[source]

Defines a unique region identifier based on module- and region-name. Typically the names will be concatenated to create a file name or a region name. Since the region name is unique in the module, concatenating the strings will then result in a unique region name.

Parameters:

module_name (str) – name of the module.
region_name (str) – name of the region.

property type_name

Returns:: the name of the Fortran derived type associated with this PSyData object.
Return type:: str

property var_name

Returns the _var_name member of this PSyDataNode.

Returns:: the _var_name of this PSyDataNode.
Return type:: str

There are two ways of passing options to the PSyDataNode. The first one is used to pass parameters from the user’s transformation script to the constructor of the node inserted, the second for passing parameters from a derived node to the PSyDataNode base class.

Passing Parameters From the User to the Node Constructor#

Options can be passed from the user via the transformation to the node that will create the code. This is done by using the options dictionary that is a standard parameter for all validate and apply calls of a transformation (see Application). Besides using this dictionary for validation and application parameters, PSyDataTrans passes it to the constructor of the node that is being inserted. An example of a parameter is the region_name, where the user can overwrite the default name given to a region (which can be somewhat cryptic due to the need to be unique). The region name is validated by PSyDataTrans, and then passed to the node constructor. The PSyDataNode stores the name as an instance attribute, so that they can be used at code creation time (PSyIR lowering). Below is the list of all options that the PSyData node supports in the option dictionary:

Parameter Name	Description
region_name	Overwrites the region name used by the `PSyDataNode`. It must be a pair of strings: the first one being the name of the module, the second the name of the region. The names are used e.g. by the `ProfileNode` to define a unique region name for a profiled code region, or by `ExtractNode` to define the file name for the output data- and driver-files.
prefix	A prefix to be used for the module name, the user-defined data type and the variables declared for the API.

Passing Parameter From a Derived Node to the `PSyDataNode`#

The PSyDataNode.lower_to_language_level function also accepts an option dictionary, which is used by derived nodes to control code creation. The lower_to_language_level function is called internally, not directly by the user. If the lower_to_language_level function of a node derived from PSyDataNode is called, it can define this option directory to pass the parameters to the PSyDataNode’s lower_to_language_level function. Here are the options that are currently supported by PSyDataNode:

Parameter Name	Description
pre_var_list	A list of the variable names to be extracted before the instrumented region.
post_var_list	A list of variable names to be extracted after the instrumented region.
pre_var_postfix	An optional postfix that will be appended to each variable name in the `pre_var_list`.
post_var_postfix	An optional postfix that will be appended to each variable name in the `post_var_list`.

If there is no variable to be provided by the PSyData API (i.e both pre_variable_list and post_variable_list are empty), then the PSyDataNode will only create a call to PreStart and PostEnd. This is utilised by the profiling node to make the profiling API libraries (see Profiling) independent of the infrastructure library (since a call to ProvideVariable can contain API-specific variable types). It also reduces the number of calls required before and after the instrumented region which can affect overall performance and precision of any measurements; see Profiling for more details.

The kernel extraction node ExtractNode uses the dependency module to determine which variables are input- and output-parameters, and provides these two lists to the ``lower_to_language_level``() function of its base class, a PSyDataNode node. It also uses the post_var_postfix option as described under lower_to_language_level``() above (see also Extraction Libraries).

PSyData Base Class#

PSyclone provides a base class for all PSyData wrapper libraries. The base class is independent of the API, but it can provide implementations for scalars and arrays for all native Fortran types that are required by the API-specific implementation. The base class does not have to be used, but it provides useful functionality:

Verbosity:: It will check the PSYDATA_VERBOSE environment flag. If it exists, it must have a value of either 0 (no messages), 1 (some messages, typically only PSyDataStart and PSyDataEnd), or 2 (detailed messages, depending on wrapper library). All other values will result in a warning message being printed (and verbosity will be disabled). The verbosity level is available as this%verbosity.
Module- and Region-Name Handling:: The module stores the module name in this%module_name, and the region name as this%region_name.
Variable Index:: It automatically sets this%next_var_index to 1 in PSyDataPreStart PSyDataPreEndDeclaration and PostStart. This variable will also be increased by one for each call to a Declare- or Provide-subroutine. It can be used to provide a reproducible index for declaring and providing a variable (and it also counts the number of declared variables, which can be used in e.g. PSyDataPreEndDeclaration to allocate arrays).
Start/Stop Handling:: The base class maintains a module variable is_enabled. This is set to true at startup, and gets enabled and disabled by calling the module functions PREFIX_PSyDataStart() and PREFIX_PSyDataStop() (see Start and Stop Functions). It is up to the derived classes to actually use this setting. Of course it is also possible to ignore is_enabled and use a different mechanism. For example, the NVIDIA profiling wrapper library calls corresponding start and stop functions in the NVIDIA profiler.
Jinja Support:: The base class is creating using the template language Jinja. It is therefore easy to automatically create the base functions for the argument types actually required by the wrapper library. See Jinja Support in the Base Class for details.

Jinja Support in the Base Class#

Code written for a PSyData library is often very repetitive. For example, an implementation of PreDeclareVariable must be provided for each data type. For LFRic that can easily result in over 10 very similar subroutines (4 basic types integer, logical, 4- and 8-byte reals; and 4- and 8-byte arrays of one to four dimensions). In order to simplify the creation of these subroutines the templating language Jinja is being used. Jinja creates code based on an template, which makes it possible to maintain just one template implementation of a subroutine, from which the various Fortran-type specific implementation will be generated.

Jinja is used in the generic base class PSyDataBase, and the base class for all ReadOnly libraries. It is not required that any library using one of these base classes itself uses Jinja. For example, the ReadOnly library for dl_esm_inf does not use Jinja (except for processing the base class templates of course), while the ReadOnly library for LFRic does. In case of dl_esm_inf, there were only 5 data types that need to be supported, so it was easy to just list these 5 functions in a generic interface. LFRic on the other hand uses many more Fortran basic types, so it uses Jinja to create the code that declares the generic interfaces. The additional advantage is that if new data types are required by LFRic (e.g. if 5-dimensional arrays are used), there will be no code change required (except for declaring the new types in the Makefile).

The PSyData base class PSyDataBaseType is contained in lib/psy_data_base.jinja. It is processed with the script process.py, which will print the processed file to stdout. The Makefile will automatically create the file psy_data_base.f90 from the Jinja template and compile it. If you use the base class in a wrapper library, you have to process the template in your library directory with additional parameters to specify the required types and the prefix. Besides the name of the template file to process the process.py script takes the following parameters:

-types:

A comma-separated list of Fortran basic types (no spaces allowed). The following type names are supported:

real:: 32-bit floating point value
double:: 64-bit floating point value
int:: 32-bit integer value
long:: 64-bit integer value
logical:: 32-bit logical value
char:: A default string value

Default value is real,double,int.

-dims:

A comma-separated list of dimensions (no spaces allowed). Default value is 1,2,3,4.

-prefix:

The prefix to use for the PSyData type and functions. Default is empty (i.e. no prefix). If you specify a prefix, you have to add the _ between the prefix and name explicitly.

-generic-declare:

If this flag is specified, the processed template will also declare a generic subroutine with all declareXXX functions (see Details About Generic Interfaces for details).

-generic-provide:

If this flag is specified, the processed template will also declare a generic subroutine with all provideXXX functions (see Details About Generic Interfaces for details).

Details About Generic Interfaces#

The Fortran standard requires that a generic subroutine is declared in the same program unit in which it is defined. Therefore, if a derived class falls back to the implementation of a function in the base class, the generic subroutine must be declared in the base class, not in the derived class (though some compilers, e.g. gfortran 9 accept it). The two options -generic-declare and -generic-provide are supported so that a derived class can control where the generic subroutines should be declared: if the derived class does not implement say the ‘declaration’ functions itself, the -generic-declare command line option guarantees that the base class defines the declaration functions all as one generic subroutine. The derived class can of course extend this generic subroutine with API-specific implementations. On the other hand, if the derived class implements the ‘declaration’ functions (potentially calling their implementation in the base class), then the derived class must declare the generic subroutine, and the base class must not declare them as well. The same approach is used for the ‘provide’ functions. As an example, the ReadOnly verification library for LFRic in lib/read_only/lfric/ uses -generic-declare when processing the PSyData base class (i.e. all declaration functions are implemented in the PSyData base class), and uses -generic-provide when processing the ReadOnly base class (i.e. all provide functions are implemented in the ReadOnly base class). The LFRic-specific implementation extends the generic subroutine with two new subroutines: DeclareFieldDouble and DeclareFieldVectorDouble (and the same for the corresponding ‘provide’ functions).

Explanation of Jinja Use#

For each specified type name the Jinja template will create methods called DeclareScalar{{type}} and ProvideScalar{{type}} for handling scalar parameters. For array parameters, the functions DeclareArray{{dim}}d{{type}} and ProvideArray{{dim}}d{{type}} will be created for each type and each specified number of dimensions.

Below is an example of using the process.py script in a Makefile for a read-only verification library (taken from lib/read_only/lfric/Makefile):

PROCESS_ARGS = -prefix=read_only_verification -types=real,int,double \
               -dims=1,2,3,4

psy_data_base.f90: ../../psy_data_base.jinja
    ../../process.py $(PROCESS_ARGS) $< > psy_data_base.f90

This will create the processed file psy_data_base.f90 in the directory of the library, where it will be compiled. Having a separate pre-processed source code version of the base class for each library (as opposed to one compiled base class library that is used for all libraries) has the advantage that consistent compiler settings will be used in your library, and consistent parameters have been provided specifying the required types and dimensions.

The process.py script provides the two variables for a template (based on its command line parameters of course): ALL_DIMS stores the list of required dimensions, and ALL_TYPES stores the type information requested when invoking process.py. ALL_TYPES is a three-tuple that lists the name to use when creating subroutine names, the Fortran data type, and number of bits for the data type. While number of bits is not used in the base class, the read-only-verification base class (see PSyData Read-Only-Verification Base Class) uses it. If more types are required, they can be defined in process.py. If the additional types need different numbers of bits and are required in a read-only library, the read-only-verification base class (lib/read_only/read_only_base.jinja) needs to be adjusted as well.

Below is a short excerpt that shows how these variables are defined by default, and how they are used to create subroutines and declare their parameters in lib/read_only/read_only_base.jinja:

{% if ALL_DIMS is not defined %}
   {# Support 1 to 4 dimensional arrays if not specified #}
   {% set ALL_DIMS = [1, 2, 3, 4] %}
{% endif %}

{# The types that are supported. The first entry of each tuple
   is the name used when naming subroutines and in user messages.
   The second entry is the Fortran declaration. The third entry
   is the number of bits. There is slightly different code
   required for 32 and 64 bit values (due to the fact that the
   Fortran ``transfer(value, mould)`` function leaves undefined
   bits when mould is larger than value.) #}

{% if ALL_TYPES is not defined %}
   {% set ALL_TYPES = [ ("Double",  "real(kind=real64)",   64),
                        ("Real",    "real(kind=real32)",   32),
                        ("Logical", "logical(kind=4)",     32),
                        ("Int",     "integer(kind=int32)", 32) ] %}
{% endif %}
...
{% for name, type, bits in ALL_TYPES %}
subroutine DeclareScalar{{name}}(this, name, value)
    {{type}}, intent(in) :: value
...

  {# Now create the declarations of all array implementations #}
  {% for dim in ALL_DIMS %}
    {# Create the ':,:...' string - DIM-times
       We repeat the list [":"] DIM-times, which is then joined #}
    {% set DIMENSION=([":"]*dim)|join(",") %}
! ---------------------------------------------------------------------
subroutine DeclareArray{{dim}}d{{name}}(this, name, value)
    {{type}}, dimension({{DIMENSION}}), intent(in) :: value
  {% endfor %}
{% endfor %}

Since the PSyData API relies on a generic interface to automatically call the right subroutine depending on type, a library must declare these automatically created subroutines (together with additional, library-specific versions) as one generic interface. This can either be done by explicitly listing the subroutines (which is for example done in lib/read_only/dl_esm_inf/read_only.f90, since it uses only 5 different data types), or using Jinja as well. The code below shows how the base class and Jinja are used in lib/read_only/lfric/read_only.f90. The FieldDouble and FieldVectorDouble related functions are explicitly coded in the LFRic-specific library, the rest is taken from the base class. Jinja is then used to create the list of all automatically created subroutines, which allows the declaration of one generic interface for each PreDeclareVariable and ProvideVariable function. While the use of Jinja here is only very minimal (only the list of subroutine names in the generic interfaces is created), the advantage of using Jinja here is that if a new data type is added to LFRic (e.g. a 5-dimensional array), only the parameter to process.py need to be changed, and the correct subroutines will be created in the base class, and the corresponding generic interfaces will be declared without further code changes:

procedure :: DeclareFieldDouble,  ProvideFieldDouble
procedure :: DeclareFieldVectorDouble,  ProvideFieldVectorDouble

{# Collect the various procedures for the same generic interface #}
{# ------------------------------------------------------------- #}
{% set all_declares=[] %}
{% set all_provides=[] %}
{% for name, type, bits in ALL_TYPES %}
  {{ all_declares.append("DeclareScalar"~name) or "" -}}
  {{ all_provides.append("ProvideScalar"~name) or "" -}}
  {% for dim in ALL_DIMS %}
    {{ all_declares.append("DeclareArray"~dim~"d"~name) or "" -}}
    {{ all_provides.append("ProvideArray"~dim~"d"~name) or "" -}}
  {% endfor %}
{% endfor %}

{% set indent="            " %}
generic, public :: PreDeclareVariable => &
    DeclareFieldDouble, &
    DeclareFieldVectorDouble, &
    {{all_declares|join(", &\n"+indent) }}

generic, public :: ProvideVariable => &
    ProvideFieldDouble,       &
    ProvideFieldVectorDouble, &
    {{all_provides|join(", &\n"+indent) }}

Note

Ending the Jinja statements with ‘or ""’ avoids having None added to the files (which would be the output of e.g. the append instruction). The ‘-’ before the closing ‘}}’ also prevents this line from creating any white-spaces. As a result of this the processed file will not have unusual empty lines or indentation.

Static Functions in the Base Class#

The base class provides the four static functions defined in the API (see API). The subroutines PREFIX_PSyDataInit() and PREFIX_PSyDataShutdown() are empty, and PREFIX_PSyDataStart() and PREFIX_PSyDataStop() function just change the module variable is_enabled (which can then be used by a wrapper library to enable or disable its functionality).

If you don’t need specific functionality for these functions, you can just declare them in your wrapper library:

module MyProfileWrapperLibrary
  use psydata_base_mod, only : profile_PSyDataInit, profilePSyDataShutdown, &
                               profile_PSyDataStart, profile_PSyDataStop

If you need to partially change the behaviour of these functions, you have to rename them in your use statement, and then call the renamed base class function like this:

module MyProfileWrapperLibrary
...
contains

subroutine profile_PSyDataStart()
  use psydata_base_mod, only : base_PSyDataStart => profile_PSyDataStart
  ! Do something
  call base_PSyDataStart()
  ! Do something else
end subroutine profile_PSyDataStart

Note

The PSyDataBase template will use the prefix that you have specified as parameter when using the process.py script for naming the static functions. So as an alternative to renaming the symbol when importing them you could also specify a different prefix when processing the base Jinja file, and use this name.

PSyData Read-Only-Verification Base Class#

The ReadOnlyVerification transformation uses the PSyData API to verify that read-only arguments to a subroutine call are not changed. It does this by computing and storing a checksum of each read-only parameter to a subroutine before the call, and verifying that this checksum is not changed after the subroutine call. Since the API-specific instances share a significant part of code (all functions for the non API-specific Fortran types, e.g. scalar values and plain Fortran arrays), a common base class is implemented for these, based on the PSyData Base class (see PSyData Base Class). The ReadOnlyBaseType is provided as a Jinja template as well (see Jinja Support in the Base Class), see lib/read_only/read_only_base.jinja. It takes the same parameters as the PSyDataBaseType, which makes it easy to make sure the PSyDataBaseClass and ReadOnly base classes are created with the same settings (like supported Fortran types and number of dimensions).

This Read-Only-Verification base class uses the PSyData base class. It uses the Declaration functions to count how many variables will be provided. In PreEndDeclaration a checksum array will be allocated.

Note

The PreStart function gets the number of variables as a parameter. The decision not to use this value for allocating the array is that the LFRic read-only implementation stores several checksums for one variable of a certain type (VectorField). The DeclareVariable functions for VectorFields counts the right number of checksums required.

The ReadOnlyBase base class uses the information about the number of bits for the data types to implement the checksum functions. One complication is that the Fortran transfer function results in undefined bits when transferring e.g. a 32-bit value into a 64-bit variable. Therefore any 32-bit value is first transferred to a 32-bit integer value, which is then assigned to the 64-bit integer value, which is added to the overall checksum value.

The two API-specific ReadOnlyVerification libraries are both based on this base class. Therefore they need only implement the checksum functions for the API-specific types - Field and VectorFields in LFRic, and Field in GOcean.

Profiling#

The command line options and transformations available to a user are described in the PSyclone User’s guide (Profiling). This section describes how the PSyData API is used to implement the profiling wrapper API, and which functions must be provided in a wrapper library to allow other existing profiling tools to be used.

Profiling API#

PSyclone uses the PSyData API to allow implementation of profile wrapper libraries that connect to various existing profiling tools. For each existing profiling tool a simple interface library needs to be implemented that maps the PSyclone PSyData calls to the corresponding call for the profiling tool.

Since the profiling API does not need access to any fields or variables, PSyclone will only create calls to PreStart and PostEnd. The profiling wrapper libraries also need the static initialisation and shutdown functions profile_PSyDataInit and profile_PSyDataShutdown. Details can be found in the section API.

The examples in the lib/profiling directory show various ways in which the opaque data type profile_PSyDataType can be used to interface with existing profiling tools - for example by storing an index used by the profiling tool in profile_PSyDataType, or by storing pointers to the profiling data to be able to print all results in a ProfileFinalise() subroutine. Some of the wrapper libraries use the PSyData base class (e.g. dl_timer, simple_timing, template), others do not (e.g. NVIDIA profiling, TAU, Vernier, DrHook wrapper).

Kernel Extraction (PSyKE)#

The PSyclone Kernel Extraction functionality (see PSy Kernel Extractor (PSyKE)) also relies on the PSyData API to write kernel input- and output-arguments to a file.

When an extraction transformation is applied, it will determine the input- and output-variable for the code region using the dependency analysis (see Variable Accesses). The code created by PSyclone’s PSyData transformation will then call the PSyData extraction library as follows:

For each input variable the variable with its original name will be stored in the file before the kernel is called.
For each output variable the variable will be stored with in the file with the postfix _post added after the kernel has been executed.

As example, an input- and output-variable my_field will therefore have two values stored in the file: the input value using the name my_field and the output value using my_field_post. If the variable is a member of a structure, the key-name of the variable to be written to the file will contain the %, e.g. my_field%whole%xstart. It is therefore important that the output format supports special characters (e.g. NetCDF does allow the use of % in names).

Note

If a name clash is detected, e.g. the user has a variable called my_field and another variable called my_field_post, the output postfix _post will be changed to make sure unique names are created by appending a number to the postfix, e.g. _post0, _post1. The same postfix will be applied to all variables, not only to variables that have a name clash.

Note

The stand-alone version of the extraction libraries does not store the actual variable names in the output files, it relies on a strict ordering of the values in the binary output file.

An excerpt of the created code (based on examples/gocean/eg5/extract):

CALL extract_psy_dataPreStart("main", "update", 11, 5)
...
CALL extract_psy_dataPreDeclareVariable("a_fld", a_fld)
CALL extract_psy_dataPreDeclareVariable("a_fld_post", a_fld)
CALL extract_psy_dataPreEndDeclaration
CALL extract_psy_dataProvideVariable("a_fld", a_fld)
...
CALL extract_psy_dataPreEnd
! Kernel execution
...
CALL extract_psy_dataPostStart
CALL extract_psy_dataProvideVariable("a_fld_post", a_fld)
...

The variable a_fld is declared twice, once with its unmodified name representing the input value, and once as a_fld_post, which will be used to store the output value. The values are provided once before the kernel to allow the PSyData library to capture the input values, and once using the name a_fld_post after the kernel execution to store the results.

Driver Creation for GOcean#

The creation of a stand-alone driver can be requested when applying the extraction transformation, e.g.:

extract = GOceanExtractTrans()
extract.apply(schedule.children, {"create_driver": True})

When compiled and executed, this driver will read in the values of all input- and output-variables, execute the instrumented code region, and then compare the results of the output variables (see Extraction Libraries). This program does not depend on any infrastructure library (like ‘dl_esm_inf`’), it only needs the PSyData ReadKernelData library (i.e. lib/extract/netcdf/read_kernel_data_mod), plus any libraries the wrapper depends on (e.g. NetCDF).

Note

The infrastructure is required at compile and link time for now, since the kernel contains metadata. Issue #2049 tracks a solution for this.

The following changes are applied by the GOceanDriverCreator and the base class DriverCreator in order to generate stand-alone code for GOcean:

The dl_esm_inf field type is replaced with 2d Fortran arrays. The structure name used is ‘flattened’, i.e. each % is replaced with a _ to make a standard Fortran name, but the final %data is removed. So my_field%data becomes my_field.
Any other structure access is converted into a variable with the same intrinsic type and flattened name. E.g. my_field%whole%xstart becomes the variable my_field_whole_xstart.
For each input-only variable one variable (with a potentially flattened name) is created. It calls ReadVariable from the PSyData ReadKernelData module, which will allocate the variable if it is an array.
For each input+output variable two variables are created and initialised (especially allocated if the variable is an array) with a call to ReadVariable. The output variable will have a postfix appended (_post by default) to distinguish it from the input value. The value of the input array will be provided to the kernel call. At the end, the newly computed values in the variable will be compared with the corresponding _post variable, which was read from the file.
An output only variable will be declared with the postfix attached, and allocated and initialised in ReadData. Then the variable without postfix will also be declared, and explicitly allocated to have the same shape as the output variable. This variable will be initialised to 0, and then provided to the kernel call. Again, at the end of the call these two variables should have the same value.

Here an example showing some of the driver code created:

integer :: a_fld_whole_ystart

real*8, allocatable, dimension(:,:) :: a_fld
real*8, allocatable, dimension(:,:) :: a_fld_post
real*8, allocatable, dimension(:,:) :: b_fld
real*8, allocatable, dimension(:,:) :: b_fld_post

call extract_psy_data%OpenRead('main', 'update')
call extract_psy_data%ReadVariable('a_fld', a_fld)
call extract_psy_data%ReadVariable('a_fld_post', a_fld_post)
call extract_psy_data%ReadVariable('a_fld%whole%ystart', a_fld_whole_ystart)

call extract_psy_data%ReadVariable('b_fld_post', b_fld_post)
ALLOCATE(b_fld(SIZE(b_fld_post, 1), SIZE(b_fld_post, 2)))
b_fld = 0

do j = a_fld_whole_ystart, a_fld_whole_ystop, 1
  do i = a_fld_whole_xstart, a_fld_whole_xstop, 1
      call update_field_code(i, j, a_fld, b_fld, ...)
  enddo
enddo

if (ALL(a_fld - a_fld_post == 0.0)) then
  PRINT *, "a_fld correct"
else
  PRINT *, "a_fld incorrect. Values are:"
  PRINT *, a_fld
  PRINT *, "a_fld values should be:"
  PRINT *, a_fld_post
end if
if (ALL(b_fld - b_fld_post == 0.0)) then
  PRINT *, "b_fld correct"
else
  PRINT *, "b_fld incorrect. Values are:"
  PRINT *, b_fld
  PRINT *, "b_fld values should be:"
  PRINT *, b_fld_post
end if

The variable a_fld is an input- and output-field, so both values for a_fld and a_fld_post are read in, and after the kernel execution the values are checked for correctness. The variable b_fld on the other hand is an output-variable only, no input values are stored in the file. The code created calls to ReadVariable for b_fld_post, which allocates the variable corresponding to the array size information stored in the data file. Then the driver allocates an array b_fld with the same shape as b_fld_post, which is initialised to 0. This b_fld is provided in the kernel call. After the kernel call, b_fld should be equal to b_fld_post.

Driver Creation for LFRic#

The creation of a stand-alone driver can be requested when applying the extraction transformation, e.g.:

extract = LFRicExtractTrans()
extract.apply(schedule.children, {"create_driver": True})

When compiled and executed, this driver will read in the values of all input- and output-variables, execute the instrumented code region, and then compare the results of the output variables (see Extraction Libraries). ATM this driver will still depend on the LFRic infrastructure library (see issue #1991) and needs the PSyData ReadKernelData library (i.e. lib/extract/netcdf/read_kernel_data_mod), plus any libraries the wrapper depends on (e.g. NetCDF).

The driver creation process is explained in the Python sources:

class psyclone.domain.lfric.LFRicDriverCreator(region_name=None)[source]

This class provides the functionality to create a driver that reads in extracted data produced by using the PSyData kernel-extraction functionality.

Parameters:: region_name (Optional[tuple[str, str]]) – the suggested region_name.

handle_precision_symbols(symbol_table)[source]

This function adds an import of the various precision symbols used by LFRic from the constants_mod module.

Parameters:: symbol_table (SymbolTable) – the symbol table to which the precision symbols must be added.
Return type:: None

verify_and_cleanup_psyir(extract_region)[source]

This implementation removes MPI related calls in LFRic (set_dirty and set_clean. Note that any LFRic-specific StructureReferences should have been replaced as part of the lowering process.

Parameters:: extract_region (Node) – the node with the extracted region.
Raises:: ValueError – if structure references are found (raised in the base class)
Return type:: None

Here is an example showing some of the driver code created:

integer(kind=i_def), allocatable, dimension(:,:) :: map_w0
integer(kind=i_def) :: cell
real(kind=r_def), allocatable, dimension(:) :: field1
real(kind=r_def), allocatable, dimension(:) :: field1_post

call extract_psy_data%OpenRead('main', 'update')
call extract_psy_data%ReadVariable('field1', field1)
call extract_psy_data%ReadVariable('loop0_start', loop0_start)
call extract_psy_data%ReadVariable('loop0_stop', loop0_stop)
call extract_psy_data%ReadVariable('map_w0', map_w0)
call extract_psy_data%ReadVariable('cell_post', cell_post)
cell = 0
call extract_psy_data%ReadVariable('field1_post', field1_post)

do cell = loop0_start, loop0_stop, 1
  call testkern_w0_code(nlayers, field1, ..., map_w0(:,cell))
enddo
if (cell == cell_post) then
  PRINT *, "cell correct"
else
  PRINT *, "cell incorrect. Values are:"
  PRINT *, cell
  PRINT *, "cell values should be:"
  PRINT *, cell_post
end if
if (ALL(field1 - field1_post == 0.0)) then
  PRINT *, "field1 correct"
else
  PRINT *, "field1 incorrect. Values are:"
  PRINT *, field1
  PRINT *, "field1 values should be:"
  PRINT *, field1_post
end if

The variable field1 is an input- and output-variable. The input value is stored in field, the expected output value is field1_post. After calling the kernel, the results of the kernel call in field1 are compared with the expected values in field1_post.

Note

For now the created driver still depends on the infrastructure library and any other modules used. Issue #1991 improves this.

The LFRic driver creation utilises the Module Manager to find and inline all modules required by the driver.

PSyData API#

Introduction to PSyData Classes#

Full Example#

API#

Init and Shutdown Functions#

Start and Stop Functions#

PREFIX_PSyDataType#

PSyDataTrans#

PSyDataNode#

Passing Parameters From the User to the Node Constructor#

Passing Parameter From a Derived Node to the PSyDataNode#

PSyData Base Class#

Jinja Support in the Base Class#

Details About Generic Interfaces#

Explanation of Jinja Use#

Static Functions in the Base Class#

PSyData Read-Only-Verification Base Class#

Profiling#

Profiling API#

Kernel Extraction (PSyKE)#

Driver Creation for GOcean#

Driver Creation for LFRic#

`PREFIX_PSyDataType`#

`PSyDataTrans`#

`PSyDataNode`#

Passing Parameter From a Derived Node to the `PSyDataNode`#