2 Design and Implementation Notes

The NUOPC Layer is implemented in Fortran on top of the public ESMF Fortran API.

The NUOPC utility routines form a very straight forward Fortran API, accessible through the NUOPC Fortran module. The interfaces only use native Fortran types and public ESMF derived types. In order to access the utility API of the NUOPC Layer, user code must include the following two use lines:

  use ESMF
  use NUOPC

2.1 Generic Components

The NUOPC generic components are implemented as a collection of Fortran modules. Each module implements a single, well specified set of standard ESMF_GridComp or ESMF_CplComp methods. The nomenclature of the generic component modules starts with the NUOPC_ prefix and continues with the flavor: Driver, Model, Mediator, or Connector. This is optionally followed by a string of additional descriptive terms. The four flavors of generic components implemented by the NUOPC Layer are:

• NUOPC_Driver - A generic driver component. It implements a child component harness, made of State and Component objects, that follows the NUOPC Common Model Architecture. It is specialized by plugging Model, Mediator, and Connector components into the harness. Driver components can be plugged into the harness to construct component hierarchies. The generic Driver initializes its child components according to a standard Initialization Phase Definition, and drives their Run() methods according a customizable run sequence.

• NUOPC_Model - A generic model component that wraps a model code so it is suitable to be plugged into a generic Driver component.

• NUOPC_Mediator - A generic mediator component that wraps custom coupling code (flux calculations, averaging, etc.) so it is suitable to be plugged into a generic Driver component.

• NUOPC_Connector - A generic component that implements Field matching based on metadata and executes simple transforms (Regrid and Redist). It can be plugged into a generic Driver component.

The user code accesses the desired generic component(s) by including a use line for each one. Each generic component defines a small set of public names that are made available to the user code through the use statement. At a minimum the SetServices method is made public. Some generic components also define a public internal state type by the standard name InternalState. It is recommended that the following syntax is used when accessing a generic component (here with internal state):

  use NUOPC_DriverXYZ, only: &
    DriverXYZ_SS => SetServices, &
    DriverXYZ_IS => InternalState

A generic component is used by user code to implement a specialized version of the component. The user code therefore also must implement a public SetServices routine. The first thing this routine must do is call into the SetServices routine provided by the generic component. It is through this step that the specialized component inherits from the generic component.

There are three mechanisms through which user code specializes generic components.

1. The specializing user code must set entry points for standard component methods not implemented by the generic component. Methods (and phases) that need to be implemented are clearly documented in the generic component description. The user code may further overwrite standard methods already implemented by the generic component code. However, this should rarely be necessary, and may indicate that there is a better fitting generic component available. Finally, some generic components come with generic routines that are suitable candidates for the standard component methods, yet require that the specializing code registers them as appropriate. Setting entry points for standard component methods is done in the SetServices routine right after calling into the generic SetServices method.

2. Some generic components require that specific methods are attached to the component. If a generic component uses specialization through attachable methods, the specific method labels (i.e. the names by which these methods are registered) and the purpose of the method are clearly documented. In some cases attachable methods are optional. This is clearly documented. Further, some generic components attach a default method to a label, which then is used for all phases. This default can be overwritten with a phase specific attachable method. Attaching methods to the component should be done in the SetServices routine right after setting entry points for the standard component methods.

3. Some generic components provide access to an internal state type. The documentation of a generic component indicates which internal state members are used for specialization, and how they are expected to be set. Setting internal state members often requires the availability of other pieces of information. It may happend in the SetServices routine, but more often inside a specialized standard entry point or an attachable method.

Figure 1: The NUOPC Generic Component inheritance structure. The upper tree is rooted in ESMF_GridComp, while the lower tree is rooted in ESMF_CplComp. The ESMF data types are shown in green. The four main NUOPC Generic Component flavors are shown in dark blue boxes. Light blue boxes contain generic components that specialize for common cases, while the yellow box shows a parent class in the inheritance tree.

Components that inherit from a generic component may choose to only specialize certain aspects, leaving other aspects unspecified. This allows a hierarchy of generic components to be implemented with a high degree of code re-use. The variable level of specialization supports the very differing user needs. Figure 1 depicts the inheritance structure of the NUOPC Generic Components. There are two trees, one is rooted in ESMF_GridComp, while the other is rooted in ESMF_CplComp.

2.2 Field Dictionary

The NUOPC Layer uses standard metadata on Fields to guide the decision making that is implemented in generic code. The generic NUOPC_Connector component, for instance, uses the StandardName Attribute to construct a list of matching Fields between the import and export States. The NUOPC Field Dictionary provides a software implementation of a controlled vocabulary for the StandardName Attribute. It also associates each registered StandardName with canonical Units, a default LongName, and a default ShortName.

The NUOPC Layer provides a number of default entries in the Field Dictionary, shown in the table below. The StandardName Attribute of all default entries complies with the Climate and Forecast (CF) conventions as documented at http://cf-pcmdi.llnl.gov/.

Currently it is typically that a user of the NUOPC Layer extends the Field Dictionary by calling the NUOPC_FieldDictionaryAddEntry() interface to add additional entries. It is our intention to grow the number of default entries over time, and to more strongly leverage the NUOPC Field Dictionary to ensure meta data interoperability between codes that use the NUOPC Layer.

Besides the StandardName Attribute, the NUOPC Layer currently only uses the Units entry to verify that Fields are given in their canonical units. The plan is to extend this to support unit conversion in the future. The default LongName and default ShortName associations are provided as a convenience to the implementor of NUOPC compliant components; the NUOPC Layer itself does not base any decisions on these two Attributes.

StandardName	Units	LongName	ShortName
	(canonical)	(default)	(default)
air_pressure_at_sea_level	Pa	Air Pressure at Sea Level	pmsl
magnitude_of_surface_downward_stress	Pa	Magnitude of Surface Downward Stress	taum
precipitation_flux	kg m-2 s-1	Precipitation Flux	prcf
sea_surface_height_above_sea_level	m	Sea Surface Height Above Sea Level	ssh
sea_surface_salinity	1e-3	Sea Surface Salinity	sss
sea_surface_temperature	K	Sea Surface Temperature	sst
surface_eastward_sea_water_velocity	m s-1	Surface Eastward Sea Water Velocity	sscu
surface_downward_eastward_stress	Pa	Surface Downward Eastward Stress	tauu
surface_downward_heat_flux_in_air	W m-2	Surface Downward Heat Flux in Air	hfns
surface_downward_water_flux	kg m-2 s-1	Surface Downward Water Flux	wfns
surface_downward_northward_stress	Pa	Surface Downward Northward Stress	tauv
surface_net_downward_shortwave_flux	W m-2	Surface Net Downward Shortwave Flux	rsns
surface_net_downward_longwave_flux	W m-2	Surface Net Downward Longwave Flux	rlns
surface_northward_sea_water_velocity	m s-1	Surface Northward Sea Water Velocity	sscv

2.3 Metadata

2.3.1 Model and Mediator Component Metadata

The Model and Mediator Component metadata is implemented as an ESMF Attribute Package:

• Convention: NUOPC
• Purpose: General
• Includes:
  • CIM Model Component Simulation Description (see for example the Component Attribute packages section in the ESMF v5.2.0rp2 documentation)
• Description: Model/Mediator component description and nesting metadata.

Name	Definition	Controlled Vocabulary
Verbosity	String value controlling the verbosity of INFO messages.	high, low
InitializePhaseMap	List of string values, mapping the logical NUOPC initialize phases, of a specific Initialize Phase Definition (IPD) version, to the actual ESMF initialize phase number under which the entry point is registered.	IPDvXXpY=Z, where XX = two-digit revision number, e.g. 01, Y = logical NUOPC phase number, Z = actual ESMF phase number, with Y, Z > 0 and Y, Z < 10
NestingGeneration	Integer value enumerating nesting level.	0, 1, 2, ...
Nestling	Integer value enumerating siblings within the same generation.	0, 1, 2, ...
InitializeDataComplete	String value indicating whether all initialize data dependencies have been satisfied.	false, true
InitializeDataProgress	String value indicating whether progress is being made resolving initialize data dependencies.	false, true

2.3.2 Connector Component Metadata

The Connector Component metadata is implemented as an ESMF Attribute Package:

• Convention: NUOPC
• Purpose: General
• Includes:
  • ESG General (see for example the Component Attribute packages section in the ESMF v5.2.0rp2 documentation)
• Description: Basic component description and connection metadata.

Name	Definition	Controlled Vocabulary
Verbosity	String value controlling the verbosity of INFO messages.	high, low
InitializePhaseMap	List of string values, mapping the logical NUOPC initialize phases, of a specific Initialize Phase Definition (IPD) version, to the actual ESMF initialize phase number under which the entry point is registered.	IPDvXXpY=Z, where XX = two-digit revision number, e.g. 01, Y = logical NUOPC phase number, Z = actual ESMF phase number, with Y, Z > 0 and Y, Z < 10
CplList	List of StandardNames of the connected Fields.	N/A

2.3.3 Field Metadata

The Field metadata is implemented as an ESMF Attribute Package:

• Convention: NUOPC
• Purpose: General
• Includes:
  • ESG General
• Description: Basic Field description with connection and time stamp metadata.

Name	Definition	Controlled Vocabulary
Connected	Connected status.	false, true
TimeStamp	Nine integer values representing ESMF Time object.	N/A
ProducerConnection	String value indicating connection details.	open, targeted, connected
ConsumerConnection	String value indicating connection details.	open, targeted, connected
Updated	String value indicating updated status during initialization.	false, true
TransferOfferGeomObject	String value indicating a component's intention to transfer the underlying Grid or Mesh on which an advertised Field object is defined.	will provide, can provide, cannot provide
TransferActiveGeomObject	String value indicating the action a component is supposed to take with respect to transferring the underlying Grid or Mesh on which an advertised Field object is defined.	provide, accept

2.4 Initialization

2.4.1 Initialize Phase Definitions

The interaction between NUOPC compliant components during the initialization process is regulated by the Initialize Phase Definition or IPD. The IPDs are versioned, with a higher version number indicating backward compatibility with all previous versions.

There are two perspectives of looking at the IPD. From the driver perspective the IPD regulates the sequence in which it must call the different phases of the Initialize() routines of its child components. To this end the generic NUOPC_Driver component implements support for IPDs up to a version specified in the API documenation.

The other angle of looking at the IPD is from the driver's child components. From this perspective the IPD assigns specific meaning to each initialize phase. The child components of a driver can be divided into two groups with respect to the meaning the IPD assigns to each initialize phase. In one group are the model, mediator, and driver components, and in the other group are the connector components. The following tables document the meaning of each initialization phase for the two different child component groups for the different IPD versions. The phases are listed in the prescribed sequence used by the driver.

IPDv00 label	Child Group	Meaning
IPDv00p1	model, mediator, driver	Advertise the import and export Fields.
IPDv00p1	connector	Construct the CplList Attribute on the connector.
IPDv00p2	model, mediator, driver	Realize the import and export Fields.
IPDv00p2	connector	Set the Connected Attribute on each import and export Field. Precompute the RouteHandle.
IPDv00p3	model, mediator, driver	Check compatibility of the Fields' Connected status.
IPDv00p4	model, mediator, driver	Handle Field data initialization. Time stamp the export Fields.

IPDv01 label	Child Group	Meaning
IPDv01p1	model, mediator, driver	Advertise the import and export Fields.
IPDv01p1	connector	Construct the CplList Attribute on the connector.
IPDv01p2	model, mediator, driver	unspecified
IPDv01p2	connector	Set the Connected Attribute on each import and export Field.
IPDv01p3	model, mediator, driver	Realize the import and export Fields.
IPDv01p3	connector	Precompute the RouteHandle.
IPDv01p4	model, mediator, driver	Check compatibility of the Fields' Connected status.
IPDv01p5	model, mediator, driver	Handle Field data initialization. Time stamp the export Fields.

IPDv02 label	Child Group	Meaning
IPDv02p1	model, mediator, driver	Advertise the import and export Fields.
IPDv02p1	connector	Construct the CplList Attribute on the connector.
IPDv02p2	model, mediator, driver	unspecified
IPDv02p2	connector	Set the Connected Attribute on each import and export Field.
IPDv02p3	model, mediator, driver	Realize the import and export Fields.
IPDv02p3	connector	Precompute the RouteHandle.
IPDv02p4	model, mediator, driver	Check compatibility of the Fields' Connected status.
IPDv02p5	model, mediator, driver	Handle Field data initialization. Timestamp the export Fields.
A loop is entered over all those model, mediator, driver Components that use IPDv02 and have unsatisfied data dependencies, repeating the following two steps:
Run()	connector	Loop over all Connectors that connect to the Component that is currently indexed by the outer loop.
IPDv02p5	model, mediator, driver	Handle Field data initialization. Time stamp the export Fields.
Repeat these two steps until all data dependencies have been statisfied, or a dead-lock situation is detected.

IPDv03 label	Child Group	Meaning
IPDv03p1	model, mediator, driver	Advertise the import and export Fields, setting TransferOfferGeomObject.
IPDv03p1	connector	Construct the CplList Attribute on the connector.
IPDv03p2	model, mediator, driver	unspecified
IPDv03p2	connector	Set the Connected Attribute on each import and export Field. Set the TransferActionGeomObject attribute.
IPDv03p3	model, mediator, driver	Realize the import and export Fields that have TransferActionGeomObject equal to "provide".
IPDv03p3	connector	Transfer the Grid/Mesh objects (only DistGrid) for Field pairs that have a provider and an acceptor side.
IPDv03p4	model, mediator, driver	Optionally modify the decomposition and distribution information of the accepted Grid/Mesh by replacing the DistGrid.
IPDv03p4	connector	Transfer the full Grid/Mesh objects (with coordinates) for Field pairs that have a provider and an acceptor side.
IPDv03p5	model, mediator, driver	Realize all Fields that have TransferActionGeomObject equal to "accept" on the transferred Grid/Mesh objects.
IPDv03p5	connector	Precompute the RouteHandle.
IPDv03p6	model, mediator, driver	Check compatibility of the Fields' Connected status.
IPDv03p7	model, mediator, driver	Handle Field data initialization. Timestamp the export Fields.
A loop is entered over all those model, mediator, driver Components that use IPDv02 and have unsatisfied data dependencies, repeating the following two steps:
Run()	connector	Loop over all Connectors that connect to the Component that is currently indexed by the outer loop.
IPDv03p7	model, mediator, driver	Handle Field data initialization. Time stamp the export Fields.
Repeat these two steps until all data dependencies have been statisfied, or a dead-lock situation is detected.

2.4.2 Data-Dependencies during Initialize

For multi-model applications it is not uncommon that during start-up one or more components depends on data from one or more other components. These type of data-dependencies during initialize can become very complex very quickly. Finding the "correct" sequence to initialize all components for a complex dependency graph is not trivial. The NUOPC Layer deals with this issue by repeatedly looping over all components that indicate that their initialization has data dependencies on other components. The loop is finally exited when either all components have indicated completion of their initialization, or a dead-lock situation is being detected by the NUOPC Layer.

The data-dependency resolution loop is implemented as part of Initialize Phase Definition version 2 (IPDv02) as defined in section 2.4.1. Participating components communicate their current status to the NUOPC Layer via Field and Component metadata. Participants are those components that contain an IPDv02p5 assignment in their InitializePhaseMap Attribute according to section 2.3.1.

Every time a component's IPDv02p5 initialization phase is called it is responsible for setting the InitializeDataComplete and InitializeDataProgress Attributes according to its current status before returning. For convenience, the NUOPC Layer provides a generic implementation of an IPDv02p5 phase initialize method for Models and Mediators (available as ESMF Initialize phase 5). This generic implementation takes care of setting the InitializeDataProgress Attribute automatically. It does so by inspecting the Updated Field Attribute (see section 2.3.3) on all the Fields in the component's exportState. The generic IPDv02p5 implementation must be specialized by attaching a method for specialization point label_DataInitialize. This specialization method is responsible for checking the Fields in the importState and for initializing any internal data structures and Fields in the exportState. Fields that are fully initialized in the exportState must be indicated by setting their Updated Attribute to "true". Once the component is fully initialized it must further set its InitializeDataComplete Attribute to "true" before returning.

During the execution of the data-dependency resolution loop the NUOPC Layer calls all of the Connectors to a Model/Mediator component before calling the component's IPDv02p5 method. Doing so ensures that all the currently available Fields are passed to the component before it tries to access them during IPDv02p5. Once a component has set its InitializeDataComplete Attribute to "true" it, and the Connectors to it, will no longer be called during the remainder of the resolution loop.

When all of the components with an IPDv02p5 initialization phase have set their InitializeDataComplete Attribute to "true", the NUOPC Layer successfully exits the data-dependency resolution loop. The loop is also interrupted before all InitializeDataComplete Attributes are set to "true" if a full cycle completes without any indicated progress. The NUOPC Layer flags this situation as a potential dead-lock and returns with error.

2.4.3 Transfer of Grid/Mesh Objects between Components

There are modeling scenarios where the need arises to transfer physical grid information from one component to another. One common situation is that of modeling systems that utilize Mediator components to implement the interactions between Model components. Here often the Mediator carries out computations on a Model's native grid. It is both cumbersome and error prone to re-defined the same physical grid in two different components. The Initialize Phase Definition version 3 (IPDv03), defined in section 2.4.1, supports the transfer of ESMF Grid and Mesh objects between Model and/or Mediator components during initialization.

The NUOPC Layer transfer protocol for GeomObjects (i.e. ESMF Grids and Meshes) is based on two Field attributes: TransferOfferGeomObject and TransferActionGeomObject. The TransferOfferGeomObject attribute is used by the Model and/or Mediator components to indicate for each Field their intent for the associated GeomObject. The predefined values of this attribute are: "will provide", "can provide", and "cannot provide". The TransferOfferGeomObject attribute must be set during IPDv03p1.

The generic Connector uses the intents from both sides and constructs a response according to the table below. The response is provided by the Connector during IPDv03p2 by setting the value of the TransferActionGeomObject attribute to either "provide" or "accept" on each Field. Fields indicating TransferActionGeomObject equal to "provide" must be realized on a Grid or Mesh object in the Model/Mediator initialize method for phase IPDv03p3.

Fields that hold "accept" for the value of the TransferActionGeomObject attribute require two additional negotiation steps. By IPDv03p4 the Model/Mediator component can access the transferred Grid/Mesh on the Fields that have the "accept" value. However, only the DistGrid, i.e. the decomposition and distribution information of the Grid/Mesh is available at this stage, not the full physical grid information such as the coordinates. At this stage the Model/Mediator may modify this information by replacing the DistGrid object in the Grid/Mesh. The DistGrid that is set on the Grid/Mesh objects when leaving the Model/Mediator phase IPDv03p4 will consequently be used by the generic Connector to fully transfer the Grid/Mesh object. The fully transferred objects are available on the Fields with "accept" during Model/Mediator phase IPDv03p5, where they can be used to realize the respective Field objects. Realizing typically just requires the ESMF_FieldEmptyComplete() call to be made. At this point all Field objects are fully realized and the initialization process can proceed as usual.

The following table shows how the generic Connector sets the TransferActionGeomObject attribute on the Fields according to the incoming value of TransferOfferGeomObject.

TransferOfferGeomObject Incoming side A	TransferOfferGeomObject Incoming side B	Outgoing setting by generic Connector
"will provide"	"will provide"	A:TransferActionGeomObject="provide" B:TransferActionGeomObject="provide"
"will provide"	"can provide"	A:TransferActionGeomObject="provide" B:TransferActionGeomObject="accept"
"will provide"	"cannot provide"	A:TransferActionGeomObject="provide" B:TransferActionGeomObject="accept"
"can provide"	"will provide"	A:TransferActionGeomObject="accept" B:TransferActionGeomObject="provide"
"can provide"	"can provide"	if (A is import side) then A:TransferActionGeomObject="provide" B:TransferActionGeomObject="accept" if (B is import side) then A:TransferActionGeomObject="accept" B:TransferActionGeomObject="provide"
"can provide"	"cannot provide"	A:TransferActionGeomObject="provide" B:TransferActionGeomObject="accept"
"cannot provide"	"will provide"	A:TransferActionGeomObject="accept" B:TransferActionGeomObject="provide"
"cannot provide"	"can provide"	A:TransferActionGeomObject="accept" B:TransferActionGeomObject="provide"
"cannot provide"	"cannot provide"	Flagged as error!