Subsections

4 Infrastructure: Fields and Grids

24 Overview of Data Classes

The ESMF infrastructure data classes are part of the framework's hierarchy of structures for handling Earth system model data and metadata on parallel platforms. The hierarchy is in complexity; the simplest data class in the infrastructure represents a distributed data array and the most complex data class represents a bundle of physical fields that are discretized on the same grid. Data class methods are called both from user-written code and from other classes internal to the framework.

Data classes are distributed over DEs, or Decomposition Elements. A DE represents a piece of a decomposition. A DELayout is a collection of DEs with some associated connectivity that describes a specific distribution. For example, the distribution of a grid divided into four segments in the x-dimension would be expressed in ESMF as a DELayout with four DEs lying along an x-axis. This abstract concept enables a data decomposition to be defined in terms of threads, MPI processes, virtual decomposition elements, or combinations of these without changes to user code. This is a primary strategy for ensuring optimal performance and portability for codes using ESMF for communications.

ESMF data classes provide a standard, convenient way for developers to collect together information related to model or observational data. The information assembled in a data class includes a data pointer, a set of attributes (e.g. units, although attributes can also be user-defined), and a description of an associated grid. The same set of information within an ESMF data object can be used by the framework to arrange intercomponent data transfers, to perform I/O, for communications such as gathers and scatters, for simplification of interfaces within user code, for debugging, and for other functions. This unifies and organizes codes overall so that the user need not define different representations of metadata for the same field for I/O and for component coupling.

Since it is critical that users be able to introduce ESMF into their codes easily and incrementally, ESMF data classes can be created based on native Fortran pointers. Likewise, there are methods for retrieving native Fortran pointers from within ESMF data objects. This allows the user to perform allocations using ESMF, and to retrieve Fortran arrays later for optimized model calculations. The ESMF data classes do not have associated differential operators or other mathematical methods.

For flexibility, it is not necessary to build an ESMF data object all at once. For example, it's possible to create a field but to defer allocation of the associated field data until a later time.


Key Features
Hierarchy of data structures designed specifically for the Earth system domain and high performance, parallel computing.
Multi-use ESMF structures simplify user code overall.
Data objects support incremental construction and deferred allocation.
Native Fortran arrays can be associated with or retrieved from ESMF data objects, for ease of adoption, convenience, and performance.
A variety of operations are provided for manipulating data in data objects such as regridding, redistribution, halo communication, and sparse matrix multiply.

The main classes that are used for model and observational data manipulation are as follows:

24.1 Bit-for-Bit Considerations

Bit-for-bit reproducibility is at the core of the regression testing schemes of many scientific model codes. The bit-for-bit requirement makes it easy to compare the numerical results of simulation runs using standard binary diff tools.

For the most part, ESMF methods do not modify user data numerically, and thus have no effect on the bit-for-bit characteristics of the model code. The exceptions are the regrid weight generation and the sparse matrix multiplication.

In the case of the regrid weight generation, user data is used to produce interpolation weights following specific numerical schemes. The bit-for-bit reproducibility of the generated weights depends on the implementation details. Section 24.2 provides more details about the bit-for-bit considerations with respect to the regrid weights generated by ESMF.

In the case of the sparse matrix multiplication, which is the typical method that is used to apply the regrid weights, user data is directly manipulated by ESMF. In order to help users with the implementation of their bit-for-bit requirements, while also considering the associated performance impact, the ESMF sparse matrix implementation provides three levels of bit-for-bit support. The strictest level ensures that the numerical results are bit-for-bit identical, even when executing across different numbers of PETs. In the relaxed level, bit-for-bit reproducibility is guaranteed when running across an unchanged number of PETs. The lowest level makes no guarantees about bit-for-bit reproducibility, however, it provides the greatest performance potential for those cases where numerical round-off differences are acceptable. An in-depth discussion of bit-for-bit reproducibility, and the performance aspects of route-based communication methods, such the sparse matrix multiplication, is given in section 37.2.1.


24.2 Regrid

This section describes the regridding methods provided by ESMF. Regridding, also called remapping or interpolation, is the process of changing the grid that underlies data values while preserving qualities of the original data. Different kinds of transformations are appropriate for different problems. Regridding may be needed when communicating data between Earth system model components such as land and atmosphere, or between different data sets to support operations such as visualization.

Regridding can be broken into two stages. The first stage is generation of an interpolation weight matrix that describes how points in the source grid contribute to points in the destination grid. The second stage is the multiplication of values on the source grid by the interpolation weight matrix to produce values on the destination grid. This is implemented as a parallel sparse matrix multiplication.

There are two options for accessing ESMF regridding functionality: offline and integrated. Offline regridding is a process whereby interpolation weights are generated by a separate ESMF command line tool, not within the user code. The ESMF offline regridding tool also only generates the interpolation matrix, the user is responsible for reading in this matrix and doing the actual interpolation (multiplication by the sparse matrix) in their code. Please see Section 12 for a description of the offline regridding command line tool and the options it supports. For user convenience, there is also a method interface to the offline regrid tool functionality which is described in Section 24.3.1. In contrast to offline regridding, integrated regridding is a process whereby interpolation weights are generated via subroutine calls during the execution of the user's code. In addition to generating the weights, integrated regridding can also produce a RouteHandle (described in Section 37.1) which allows the user to perform the parallel sparse matrix multiplication using ESMF methods. In other words, ESMF integrated regridding allows a user to perform the whole process of interpolation within their code.

To see what types of grids and other options are supported in the two types of regridding and their testing status, please see the ESMF Regridding Status webpage for this version of ESMF. Figure 24.2 shows a comparison of different regrid interfaces and where they can be found in the documentation.

The rest of this section further describes the various options available in ESMF regridding.


Table 1: Regrid Interfaces

Name Access via Inputs Outputs Description
      Weights RouteHandle  
ESMF_FieldRegridStore() Subroutine call Field object yes yes Sec. 26.6.69
ESMF_FieldBundleRegridStore() Subroutine call Fieldbundle obj. no yes Sec. 25.5.27
ESMF_RegridWeightGen() Subroutine call Grid files yes no Sec. 24.3.1
ESMF_RegridWeightGen Command Line Tool Grid files yes no Sec. 12



24.2.1 Interpolation methods: bilinear

Bilinear interpolation calculates the value for the destination point as a combination of multiple linear interpolations, one for each dimension of the Grid. Note that for ease of use, the term bilinear interpolation is used for 3D interpolation in ESMF as well, although it should more properly be referred to as trilinear interpolation.

In 2D, ESMF supports bilinear regridding between any combination of the following:

In 3D, ESMF supports bilinear regridding between any combination of the following:

Restrictions:

To use the bilinear method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates. This method will also work with a destination Field built on a LocStream that contains coordinates, or with a source or destination Field built on an XGrid.


24.2.2 Interpolation methods: higher-order patch

Patch (or higher-order) interpolation is the ESMF version of a technique called “patch recovery” commonly used in finite element modeling [25] [22]. It typically results in better approximations to values and derivatives when compared to bilinear interpolation. Patch interpolation works by constructing multiple polynomial patches to represent the data in a source cell. For 2D grids, these polynomials are currently 2nd degree 2D polynomials. One patch is constructed for each corner of the source cell, and the patch is constructed by doing a least squares fit through the data in the cells surrounding the corner. The interpolated value at the destination point is then a weighted average of the values of the patches at that point.

The patch method has a larger stencil than the bilinear, for this reason the patch weight matrix can be correspondingly larger than the bilinear matrix (e.g. for a quadrilateral grid the patch matrix is around 4x the size of the bilinear matrix). This can be an issue when performing a regrid operation close to the memory limit on a machine.

The patch method does not guarantee that after regridding the range of values in the destination field is within the range of values in the source field. For example, if the mininum value in the source field is 0.0, then it's possible that after regridding with the patch method, the destination field will contain values less than 0.0.

In 2D, ESMF supports patch regridding between any combination of the following:

In 3D, ESMF supports patch regridding between any combination of the following:

Restrictions:

To use the patch method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates. This method will also work with a destination Field built on a LocStream that contains coordinates, or with a source or destination Field built on an XGrid.


24.2.3 Interpolation methods: nearest source to destination

In nearest source to destination interpolation (ESMF_REGRIDMETHOD_NEAREST_STOD) each destination point is mapped to the closest source point. A given source point may map to multiple destination points, but no destination point will receive input from more than one source point. If two points are equally close, then the point with the smallest sequence index is arbitrarily used (i.e. the point which would have the smallest index in the weight matrix).

In 2D, ESMF supports nearest source to destination regridding between any combination of the following:

In 3D, ESMF supports nearest source to destination regridding between any combination of the following:

Restrictions:
NONE

To use the nearest source to destination method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates. This method will also work with a source or destination Field built on a LocStream that contains coordinates, or when the source or destination Field is built on an XGrid.


24.2.4 Interpolation methods: nearest destination to source

In nearest destination to source interpolation (ESMF_REGRIDMETHOD_NEAREST_DTOS) each source point is mapped to the closest destination point. A given destination point may receive input from multiple source points, but no source point will map to more than one destination point. If two points are equally close, then the point with the smallest sequence index is arbitrarily used (i.e. the point which would have the smallest index in the weight matrix). Note that with this method the unmapped destination point detection currently doesn't work, so no error will be returned even if there are destination points that don't map to any source point.

In 2D, ESMF supports nearest destination to source regridding between any combination of the following:

In 3D, ESMF supports nearest destination to source regridding between any combination of the following:

Restrictions:

To use the nearest destination to source method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates. This method will also work with a source or destination Field built on a LocStream that contains coordinates, or when the source or destination Field is built on an XGrid.


24.2.5 Interpolation methods: first-order conservative

The goal of this method is to preserve the integral of the field across the interpolation from source to destination. (For a more in-depth description of what this preservation of the integral (i.e. conservation) means please see section 24.2.7.) In this method the value across each source cell is treated as a constant, so it will typically have a larger interpolation error than the bilinear or patch methods. The first-order method used here is similar to that described in the following paper [28].

In the first-order method, the values for a particular destination cell are a calculated as a combination of the values of the intersecting source cells. The weight of a given source cell's contribution to the total being the amount that that source cell overlaps with the destination cell. In particular, the weight is the ratio of the area of intersection of the source and destination cells to the area of the whole destination cell.

To see a description of how the different normalization options affect the values and integrals produced by the conservative methods see section 24.2.8. For Grids, Meshes, or XGrids on a sphere this method uses great circle cells, for a description of potential problems with these see 24.2.9.

In 2D, ESMF supports conservative regridding between any combination of the following:

In 3D, ESMF supports conservative regridding between any combination of the following:

Restrictions:

To use the conservative method the user should create their Fields on the center stagger location (ESMF_STAGGERLOC_CENTER in 2D or ESMF_STAGGERLOC_CENTER_VCENTER in 3D) for Grids or the element location (ESMF_MESHLOC_ELEMENT) for Meshes. For Grids, the corner stagger location (ESMF_STAGGERLOC_CORNER in 2D or ESMF_STAGGERLOC_CORNER_VFACE in 3D) must contain coordinates describing the outer perimeter of the Grid cells. This method will also work when the source or destination Field is built on an XGrid.


24.2.6 Interpolation methods: second-order conservative

Like the first-order conservative method, this method's goal is to preserve the integral of the field across the interpolation from source to destination. (For a more in-depth description of what this preservation of the integral (i.e. conservation) means please see section 24.2.7.) The difference between the first and second-order conservative methods is that the second-order takes the source gradient into account, so it yields a smoother destination field that typically better matches the source field. This difference between the first and second-order methods is particularly apparent when going from a coarse source grid to a finer destination grid. Another difference is that the second-order method does not guarantee that after regridding the range of values in the destination field is within the range of values in the source field. For example, if the mininum value in the source field is 0.0, then it's possible that after regridding with the second-order method, the destination field will contain values less than 0.0. The implementation of this method is based on the one described in this paper [19].

Like the first-order method, the values for a particular destination cell with the second-order method are a combination of the values of the intersecting source cells with the weight of a given source cell's contribution to the total being the amount that that source cell overlaps with the destination cell. However, with the second-order conservative interpolation there are additional terms that take into account the gradient of the field across the source cell. In particular, the value $d$ for a given destination cell is calculated as:

$d=\sum^{intersecting-source-cells}_{i}(s_{i}+\nabla s_{i} \cdot (c_{si}-c_{d}))$

Where:

$s_{i}$
is the intersecting source cell value.
$\nabla s_{i}$
is the intersecting source cell gradient.
$c_{si}$
is the intersecting source cell centroid.
$c_{d}$
is the destination cell centroid.

To see a description of how the different normalization options affect the values and integrals produced by the conservative methods see section 24.2.8. For Grids, Meshes, or XGrids on a sphere this method uses great circle cells, for a description of potential problems with these see 24.2.9.

In 2D, ESMF supports second-order conservative regridding between any combination of the following:

In 3D, ESMF supports second-order conservative regridding between any combination of the following:

Restrictions:

To use the second-order conservative method the user should create their Fields on the center stagger location (ESMF_STAGGERLOC_CENTER for Grids or the element location (ESMF_MESHLOC_ELEMENT) for Meshes. For Grids, the corner stagger location (ESMF_STAGGERLOC_CORNER in 2D must contain coordinates describing the outer perimeter of the Grid cells. This method will also work when the source or destination Field is built on an XGrid.


24.2.7 Conservation

Conservation means that the following equation will hold: $\sum^{all-source-cells}(V_{si}*A_{si}) = \sum^{all-destination-cells}(V_{dj}*A_{dj})$, where V is the variable being regridded and A is the area of a cell. The subscripts s and d refer to source and destination values, and the i and j are the source and destination grid cell indices (flattening the arrays to 1 dimension).

If the user doesn't specify a cell areas in the involved Grids or Meshes, then the areas (A) in the above equation are calculated by ESMF. For Cartesian grids, the area of a grid cell calculated by ESMF is the typical Cartesian area. For grids on a sphere, cell areas are calculated by connecting the corner coordinates of each grid cell with great circles. If the user does specify the areas in the Grid or Mesh, then the conservation will be adjusted to work for the areas provided by the user. This means that the above equation will hold, but with the areas (A) being the ones specified by the user.

The user should be aware that because of the conservation relationship between the source and destination fields, the more the total source area differs from the total destination area the more the values of the source field will differ from the corresponding values of the destination field, likely giving a higher interpolation error. It is best to have the total source and destination areas the same (this will automatically be true if no user areas are specified). For source and destination grids that only partially overlap, the overlapping regions of the source and destination should be the same.


24.2.8 The effect of normalization options on integrals and values produced by conservative methods

It is important to note that by default (i.e. using destination area normalization) conservative regridding doesn't normalize the interpolation weights by the destination fraction. This means that for a destination grid which only partially overlaps the source grid the destination field that is output from the regrid operation should be divided by the corresponding destination fraction to yield the true interpolated values for cells which are only partially covered by the source grid. The fraction also needs to be included when computing the total source and destination integrals. (To include the fraction in the conservative weights, the user can specify the fraction area normalization type. This can be done by specifying normType=ESMF_NORMTYPE_FRACAREA when invoking ESMF_FieldRegridStore().)

For weights generated using destination area normalization (either by not specifying any normalization type or by specifying normType=ESMF_NORMTYPE_DSTAREA), if a destination field extends outside the unmasked source field, then the values of the cells which extend partway outside the unmasked source field are decreased by the fraction they extend outside. To correct these values, the destination field (dst_field) resulting from the ESMF_FieldRegrid() call can be divided by the destination fraction dst_frac from the ESMF_FieldRegridStore() call. The following pseudocode demonstrates how to do this:


 for each destination element i
    if (dst_frac(i) not equal to 0.0) then
       dst_field(i)=dst_field(i)/dst_frac(i)
    end if
 end for

For weights generated using destination area normalization (either by not specifying any normalization type or by specifying normType=ESMF_NORMTYPE_DSTAREA), the following pseudo-code shows how to compute the total destination integral (dst_total) given the destination field values (dst_field) resulting from the ESMF_FieldRegrid() call, the destination area (dst_area) from the ESMF_FieldRegridGetArea() call, and the destination fraction (dst_frac) from the ESMF_FieldRegridStore() call. As shown in the previous paragraph, it also shows how to adjust the destination field (dst_field) resulting from the ESMF_FieldRegrid() call by the fraction (dst_frac) from the ESMF_FieldRegridStore() call:


 dst_total=0.0
 for each destination element i
    if (dst_frac(i) not equal to 0.0) then
       dst_total=dst_total+dst_field(i)*dst_area(i) 
       dst_field(i)=dst_field(i)/dst_frac(i)
       ! If mass computed here after dst_field adjust, would need to be:
       ! dst_total=dst_total+dst_field(i)*dst_area(i)*dst_frac(i) 
    end if
 end for

For weights generated using fraction area normalization (by specifying normType=ESMF_NORMTYPE_FRACAREA), no adjustment of the destination field is necessary. The following pseudo-code shows how to compute the total destination integral (dst_total) given the destination field values (dst_field) resulting from the ESMF_FieldRegrid() call, the destination area (dst_area) from the ESMF_FieldRegridGetArea() call, and the destination fraction (dst_frac) from the ESMF_FieldRegridStore() call:

 dst_total=0.0
 for each destination element i
      dst_total=dst_total+dst_field(i)*dst_area(i)*dst_frac(i) 
 end for

For both normalization types, the following pseudo-code shows how to compute the total source integral (src_total) given the source field values (src_field), the source area (src_area) from the ESMF_FieldRegridGetArea() call, and the source fraction (src_frac) from the ESMF_FieldRegridStore() call:

 src_total=0.0
 for each source element i
    src_total=src_total+src_field(i)*src_area(i)*src_frac(i)
 end for


24.2.9 Great circle cells

For Grids, Meshes, or XGrids on a sphere some combinations of interpolation options (e.g. first and second-order conservative methods) use cells whose edges are great circles. This section describes some behavior that the user may not expect from these cells and some potential solutions.

A great circle edge isn't necessarily the same as a straight line in latitude longitude space. For small edges, this difference will be small, but for long edges it could be significant. This means if the user expects cell edges as straight lines in latitude longitude space, they should avoid using one large cell with long edges to compute an average over a region (e.g. over an ocean basin).

Also, the user should also avoid using cells that contain one edge that runs half way or more around the earth, because the regrid weight calculation assumes the edge follows the shorter great circle path. There isn't a unique great circle edge defined between points on the exact opposite side of the earth from one another (antipodal points). However, the user can work around both of these problem by breaking the long edge into two smaller edges by inserting an extra node, or by breaking the large target grid cells into two or more smaller grid cells. This allows the application to resolve the ambiguity in edge direction.


24.2.10 Masking

Masking is the process whereby parts of a Grid, Mesh, or LocStream can be marked to be ignored during an operation, such as when they are used in regridding. Masking can be used on a Field created from a regridding source to indicate that certain portions should not be used to generate regridded data. This is useful, for example, if a portion of the source contains unusable values. Masking can also be used on a Field created from a regridding destination to indicate that a certain portion should not receive regridded data. This is useful, for example, when part of the destination isn't being used (e.g. the land portion of an ocean grid).

The user may mask out points in the source Field or destination Field or both. To do masking the user sets mask information in the Grid (see 31.3.17), Mesh (see 33.3.11), or LocStream (see 32.2.2) upon which the Fields passed into the ESMF_FieldRegridStore() call are built. The srcMaskValues and dstMaskValues arguments to that call can then be used to specify which values in that mask information indicate that a location should be masked out. For example, if dstMaskValues is set to (/1,2/), then any location that has a value of 1 or 2 in the mask information of the Grid, Mesh or LocStream upon which the destination Field is built will be masked out.

Masking behavior differs slightly between regridding methods. For non-conservative regridding methods (e.g. bilinear or high-order patch), masking is done on points. For these methods, masking a destination point means that that point won't participate in regridding (e.g. won't be interpolated to). For these methods, masking a source point means that the entire source cell using that point is masked out. In other words, if any corner point making up a source cell is masked then the cell is masked. For conservative regridding methods (e.g. first-order conservative) masking is done on cells. Masking a destination cell means that the cell won't participate in regridding (e.g. won't be interpolated to). Similarly, masking a source cell means that the cell won't participate in regridding (e.g. won't be interpolated from). For any type of interpolation method (conservative or non-conservative) the masking is set on the location upon which the Fields passed into the regridding call are built. For example, if Fields built on ESMF_STAGGERLOC_CENTER are passed into the ESMF_FieldRegridStore() call then the masking should also be set on ESMF_STAGGERLOC_CENTER.


24.2.11 Extrapolation methods: overview

Extrapolation in the ESMF regridding system is a way to automatically fill some or all of the destination points left unmapped by a regridding method. Weights generated by the extrapolation method are merged into the regridding weights to yield one set of weights or routehandle. Currently extrapolation is not supported with conservative regridding methods, because doing so would result in non-conservative weights.


24.2.12 Extrapolation methods: nearest source to destination

In nearest source to destination extrapolation (ESMF_EXTRAPMETHOD_NEAREST_STOD) each unmapped destination point is mapped to the closest source point. A given source point may map to multiple destination points, but no destination point will receive input from more than one source point. If two points are equally close, then the point with the smallest sequence index is arbitrarily used (i.e. the point which would have the smallest index in the weight matrix).

If there is at least one unmasked source point, then this method is expected to fill all unmapped points.


24.2.13 Extrapolation methods: inverse distance weighted average

In inverse distance weighted average extrapolation (ESMF_EXTRAPMETHOD_NEAREST_IDAVG) each unmapped destination point is the weighted average of the closest N source points. The weight is the reciprocal of the distance of the source point from the destination point raised to a power P. All the weights contributing to one destination point are normalized so that they sum to 1.0. The user can choose N and P when using this method, but defaults are also provided. For example, when calling ESMF_FieldRegridStore() N is specified via the argument extrapNumSrcPnts and P is specified via the argument extrapDistExponent.

If there is at least one unmasked source point, then this method is expected to fill all unmapped points.


24.2.14 Extrapolation methods: creep fill

In creep fill extrapolation (ESMF_EXTRAPMETHOD_CREEP) unmapped destination points are filled by repeatedly moving data from mapped locations to neighboring unmapped locations for a user specified number of levels. More precisely, for each creeped point, its value is the average of the values of the point's immediate neighbors in the previous level. For the first level, the values are the average of the point's immediate neighbors in the destination points mapped by the regridding method. The number of creep levels is specified by the user. For example, in ESMF_FieldRegridStore() this number of levels is specified via the extrapNumLevels argument.

Unlike some extrapolation methods, creep fill does not necessarily fill all unmapped destination points. Unfilled destination points are still unmapped with the usual consequences (e.g. they won't be in the resulting regridding matrix, and won't be set by the application of the regridding weights).

Because it depends on the connections in the destination grid, creep fill extrapolation is not supported when the destination Field is built on a Location Stream (ESMF_LocStream). Also, creep fill is currently only supported for 2D Grids, Meshes, or XGrids

24.2.15 Unmapped destination points

If a destination point can't be mapped to a location in the source grid by the combination of regrid method and optional follow on extrapolation method, then the user has two choices. The user may ignore those destination points that can't be mapped by setting the unmappedaction argument to ESMF_UNMAPPEDACTION_IGNORE (Ignored points won't be included in the sparse matrix or routeHandle). If the user needs the unmapped points, the ESMF_FieldRegridStore() method has the capability to return a list of them using the unmappedDstList argument. In addition to ignoring them, the user also has the option to return an error if unmapped destination points exist. This is the default behavior, so the user can either not set the unmappedaction argument or the user can set it to ESMF_UNMAPPEDACTION_ERROR. Currently, the unmapped destination error detection doesn't work with the nearest destination to source regrid method (ESMF_REGRIDMETHOD_NEAREST_DTOS), so with this method the regridding behaves as if ESMF_UNMAPPEDACTION_IGNORE is always on.

24.2.16 Spherical grids and poles

In the case that the Grid is on a sphere (coordSys=ESMF_COORDSYS_SPH_DEG or ESMF_COORDSYS_SPH_RAD) then the coordinates given in the Grid are interpreted as latitude and longitude values. The coordinates can either be in degrees or radians as indicated by the coordSys flag set during Grid creation. As is true with many global models, this application currently assumes the latitude and longitude refer to positions on a perfect sphere, as opposed to a more complex and accurate representation of the Earth's true shape such as would be used in a GIS system. (ESMF's current user base doesn't require this level of detail in representing the Earth's shape, but it could be added in the future if necessary.)

For Grids on a sphere, the regridding occurs in 3D Cartesian to avoid problems with periodicity and with the pole singularity. This library supports four options for handling the pole region (i.e. the empty area above the top row of the source grid or below the bottom row of the source grid). Note that all of these pole options currently only work for the Fields build on the Grid class.

The first option is to leave the pole region empty (polemethod=ESMF_POLEMETHOD_NONE), in this case if a destination point lies above or below the top row of the source grid, it will fail to map, yielding an error (unless unmappedaction=ESMF_UNMAPPEDACTION_IGNORE is specified).

With the next two options (ESMF_POLEMETHOD_ALLAVG and ESMF_POLEMETHOD_NPNTAVG), the pole region is handled by constructing an artificial pole in the center of the top and bottom row of grid points and then filling in the region from this pole to the edges of the source grid with triangles. The pole is located at the average of the position of the points surrounding it, but moved outward to be at the same radius as the rest of the points in the grid. The difference between the two artificial pole options is what value is used at the pole. The option (polemethod=ESMF_POLEMETHOD_ALLAVG) sets the value at the pole to be the average of the values of all of the grid points surrounding the pole. The option (polemethod=ESMF_POLEMETHOD_NPNTAVG) allows the user to choose a number N from 1 to the number of source grid points around the pole. The value N is set via the argument regridPoleNPnts. For each destination point, the value at the pole is then the average of the N source points surrounding that destination point.

The last option (polemethod=ESMF_POLEMETHOD_TEETH) does not construct an artificial pole, instead the pole region is covered by connecting points across the top and bottom row of the source Grid into triangles. As this makes the top and bottom of the source sphere flat, for a big enough difference between the size of the source and destination pole regions, this can still result in unmapped destination points. Only pole option ESMF_POLEMETHOD_NONE is currently supported with the conservative interpolation methods (e.g. regridmethod=ESMF_REGRIDMETHOD_CONSERVE) and with the nearest neighbor interpolation options (e.g. regridmethod=ESMF_REGRIDMETHOD_NEAREST_STOD).


Table 2: Line Type Support by Regrid Method (* indicates the default)

Regrid Method Line Type
  ESMF_LINETYPE_CART ESMF_LINETYPE_GREAT_CIRCLE
ESMF_REGRIDMETHOD_BILINEAR Y* Y
ESMF_REGRIDMETHOD_PATCH Y* Y
ESMF_REGRIDMETHOD_NEAREST_STOD Y* N
ESMF_REGRIDMETHOD_NEAREST_DTOS Y* N
ESMF_REGRIDMETHOD_CONSERVE N/A Y*
ESMF_REGRIDMETHOD_CONSERVE_2ND N/A Y*


Another variation in the regridding supported with spherical grids is line type. This is controlled in the ESMF_FieldRegridStore() method by the lineType argument. This argument allows the user to select the path of the line which connects two points on a sphere surface. This in turn controls the path along which distances are calculated and the shape of the edges that make up a cell. Both of these quantities can influence how interpolation weights are calculated, for example in bilinear interpolation the distances are used to calculate the weights and the cell edges are used to determine to which source cell a destination point should be mapped.

ESMF currently supports two line types: ESMF_LINETYPE_CART and ESMF_LINETYPE_GREAT_CIRCLE. The ESMF_LINETYPE_CART option specifies that the line between two points follows a straight path through the 3D Cartesian space in which the sphere is embedded. Distances are measured along this 3D Cartesian line. Under this option cells are approximated by planes in 3D space, and their boundaries are 3D Cartesian lines between their corner points. The ESMF_LINETYPE_GREAT_CIRCLE option specifies that the line between two points follows a great circle path along the sphere surface. (A great circle is the shortest path between two points on a sphere.) Distances are measured along the great circle path. Under this option cells are on the sphere surface, and their boundaries are great circle paths between their corner points.

Figure 24.2.16 shows which line types are supported for each regrid method as well as the defaults (indicated by *).


24.2.17 Vector regridding

ESMF's initial vector regridding capability is intended to give cleaner results for 2D spherical vectors expressed in terms of local directions (e.g. east and north) than regridding each vector component separately. To do this, it converts the vectors to 3D Cartesian space and then does the regridding there. This allows all the vectors participating in the regridding to have a consistent representation. After regridding, the resulting 3D vectors are then converted back to the local direction form. This entire process is expressed in the usual weight matrix and/or routeHandle form and so the typical ESMF_FieldRegridStore()/ESMF_FieldRegrid()/ESMF_FieldRegridRelease() regridding paradigm can be used. However, the weight matrix will be in the format that allows it to contain tensor dimension indices (i.e. the leading dimension of the factorIndexList will be of size 4).

In this initial version, the meaning of the different entries in the vector dimension are fixed. They will be interpreted as:

1st entry
the east component of the vector
2nd entry
the north component of the vector

Note that because the different components are mixed, using vector regridding with a conservative regrid method will not necessarily produce vectors whose components are conservative.

24.2.18 Troubleshooting guide

The below is a list of problems users commonly encounter with regridding and potential solutions. This is by no means an exhaustive list, so if none of these problems fit your case, or if the solutions don't fix your problem, please feel free to email esmf support (esmf_support@ucar.edu).


Problem: Regridding is too slow.


Possible Cause: The ESMF_FieldRegridStore() method is called more than is necessary.
The ESMF_FieldRegridStore() operation is a complex one and can be relatively slow for some cases (large Grids, 3D grids, etc.)

Solution: Reduce the number of ESMF_FieldRegridStore() calls to the minimum necessary. The routeHandle generated by the ESMF_FieldRegridStore() call depends on only four factors: the stagger locations that the input Fields are created on, the coordinates in the Grids the input Fields are built on at those stagger locations, the padding of the input Fields (specified by the totalWidth arguments in FieldCreate) and the size of the tensor dimensions in the input Fields (specified by the ungridded arguments in FieldCreate). For any pair of Fields which share these attributes with the Fields used in the ESMF_FieldRegridStore call the same routeHandle can be used. Note that the data in the Fields does NOT matter, the same routeHandle can be used no matter how the data in the Fields changes.

In particular:


Problem: Distortions in destination Field at periodic boundary.


Possible Cause: The Grid overlaps itself. With a periodic Grid, the regrid system expects the first point to not be a repeat of the last point. In other words, regrid constructs its own connection and overlap between the first and last points of the periodic dimension and so the Grid doesn't need to contain these. If the Grid does, then this can cause problems.

Solution: Define the Grid so that it doesn't contain the overlap point. This typically means simply making the Grid one point smaller in the periodic dimension. If a Field constructed on the Grid needs to contain these overlap points then the user can use the totalWidth arguments to include this extra padding in the Field. Note, however, that the regrid won't update these extra points, so the user will have to do a copy to fill the points in the overlap region in the Field.

24.2.19 Restrictions and Future Work

This section contains restrictions that apply to the entire regridding system. For restrictions that apply to just one interpolation method, see the section corresponding to that method above.

24.2.20 Design and implementation notes

The ESMF regrid weight calculation functionality has been designed to enable it to support a wide range of grid and interpolation types without needing to support each individual combination of source grid type, destination grid type, and interpolation method. To avoid the quadratic growth of the number of pairs of grid types, all grids are converted to a common internal format and the regrid weight calculation is performed on that format. This vastly reduces the variety of grids that need to be supported in the weight calculations for each interpolation method. It also has the added benefit of making it straightforward to add new grid types and to allow them to work with all the existing grid types. To hook into the existing weight calculation code, the new type just needs to be converted to the internal format.

The internal grid format used by the ESMF regrid weight calculation is a finite element unstructured mesh. This was chosen because it was the most general format and all the others could be converted to it. The ESMF finite element unstructured mesh (ESMF FEM) is similar in some respects to the SIERRA [20] package developed at Sandia National Laboratory. The ESMF code relies on some of the same underlying toolkits (e.g. Zoltan [18] library for calculating mesh partitions) and adds a layer on top that allows the calculation of regrid weights and some mesh operations (e.g. mesh redistribution) that ESMF needs. The ESMF FEM has similar notions to SIERRA about the basic structure of the mesh entities, fields, iteration and a similar notion of parallel distribution.

Currently we use the ESMF FEM internal mesh to hold the structure of our Mesh class and in our regrid weight calculation. The parts of the internal FEM code that are used/tested by ESMF are the following:

24.3 File-based Regrid API

 

24.3.1 ESMF_RegridWeightGen - Generate regrid weight file from grid files


INTERFACE:

   ! Private name; call using ESMF_RegridWeightGen()
   subroutine ESMF_RegridWeightGenFile(srcFile, dstFile, &
     weightFile, rhFile, regridmethod, polemethod, regridPoleNPnts, lineType, normType, &
     extrapMethod, extrapNumSrcPnts, extrapDistExponent, extrapNumLevels, &
     unmappedaction, ignoreDegenerate, srcFileType, dstFileType, &
     srcRegionalFlag, dstRegionalFlag, srcMeshname, dstMeshname,  &
     srcMissingvalueFlag, srcMissingvalueVar, &
     dstMissingvalueFlag, dstMissingvalueVar, &
     useSrcCoordFlag, srcCoordinateVars, &
     useDstCoordFlag, dstCoordinateVars, &
     useSrcCornerFlag, useDstCornerFlag, &
     useUserAreaFlag, largefileFlag, &
     netcdf4fileFlag, weightOnlyFlag, &
     tileFilePath, &
     verboseFlag, checkFlag, rc)
ARGUMENTS:
 
   character(len=*),             intent(in)            :: srcFile
   character(len=*),             intent(in)            :: dstFile
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   character(len=*),             intent(in),  optional :: weightFile
   character(len=*),             intent(in),  optional :: rhFile
   type(ESMF_RegridMethod_Flag), intent(in),  optional :: regridmethod
   type(ESMF_PoleMethod_Flag),   intent(in),  optional :: polemethod
   integer,                      intent(in),  optional :: regridPoleNPnts
   type(ESMF_LineType_Flag),     intent(in),  optional :: lineType
   type(ESMF_NormType_Flag),     intent(in),  optional :: normType
   type(ESMF_ExtrapMethod_Flag),   intent(in),    optional :: extrapMethod
   integer,                        intent(in),    optional :: extrapNumSrcPnts
   real,                           intent(in),    optional :: extrapDistExponent
   integer,                      intent(in), optional :: extrapNumLevels
   type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction
   logical,                      intent(in),  optional :: ignoreDegenerate
   type(ESMF_FileFormat_Flag),   intent(in),  optional :: srcFileType
   type(ESMF_FileFormat_Flag),   intent(in),  optional :: dstFileType
   logical,                      intent(in),  optional :: srcRegionalFlag
   logical,                      intent(in),  optional :: dstRegionalFlag
   character(len=*),             intent(in),  optional :: srcMeshname
   character(len=*),             intent(in),  optional :: dstMeshname
   logical,                      intent(in),  optional :: srcMissingValueFlag
   character(len=*),             intent(in),  optional :: srcMissingValueVar
   logical,                      intent(in),  optional :: dstMissingValueFlag
   character(len=*),             intent(in),  optional :: dstMissingValueVar
   logical,                      intent(in),  optional :: useSrcCoordFlag
   character(len=*),             intent(in),  optional :: srcCoordinateVars(:)
   logical,                      intent(in),  optional :: useDstCoordFlag
   character(len=*),             intent(in),  optional :: dstCoordinateVars(:)
   logical,                      intent(in),  optional :: useSrcCornerFlag
   logical,                      intent(in),  optional :: useDstCornerFlag
   logical,                      intent(in),  optional :: useUserAreaFlag
   logical,                      intent(in),  optional :: largefileFlag
   logical,                      intent(in),  optional :: netcdf4fileFlag
   logical,                      intent(in),  optional :: weightOnlyFlag
   character(len=*),             intent(in),  optional :: tileFilePath
   logical,                      intent(in),  optional :: verboseFlag
   logical,                      intent(in),  optional :: checkFlag
   integer,                      intent(out), optional :: rc
DESCRIPTION:

This subroutine provides the same function as the ESMF_RegridWeightGen application described in Section 12. It takes two grid files in NetCDF format and writes out an interpolation weight file also in NetCDF format. The interpolation weights can be generated with the bilinear (24.2.1), higher-order patch (24.2.2), or first order conservative (24.2.5) methods. The grid files can be in one of the following four formats:

The weight file is created in SCRIP format (12.9). The optional arguments allow users to specify various options to control the regrid operation, such as which pole option to use, whether to use user-specified area in the conservative regridding, or whether ESMF should generate masks using a given variable's missing value. There are also optional arguments specific to a certain type of the grid file. All the optional arguments are similar to the command line arguments for the ESMF_RegridWeightGen application (12.6). The acceptable values and the default value for the optional arguments are listed below.

The arguments are:

srcFile
The source grid file name.
dstFile
The destination grid file name.
weightFile
The interpolation weight file name.
[rhFile]
The RouteHandle file name.
[regridmethod]
The type of interpolation. Please see Section 54.51 for a list of valid options. If not specified, defaults to ESMF_REGRIDMETHOD_BILINEAR.
[polemethod]
A flag to indicate which type of artificial pole to construct on the source Grid for regridding. Please see Section 54.48 for a list of valid options. The default value varies depending on the regridding method and the grid type and format.
[regridPoleNPnts]
If polemethod is set to ESMF_POLEMETHOD_NPNTAVG, this argument is required to specify how many points should be averaged over at the pole.
[lineType]
This argument controls the path of the line which connects two points on a sphere surface. This in turn controls the path along which distances are calculated and the shape of the edges that make up a cell. Both of these quantities can influence how interpolation weights are calculated. As would be expected, this argument is only applicable when srcField and dstField are built on grids which lie on the surface of a sphere. Section 54.36 shows a list of valid options for this argument. If not specified, the default depends on the regrid method. Section 54.36 has the defaults by line type. Figure 24.2.16 shows which line types are supported for each regrid method as well as showing the default line type by regrid method.
[normType]
This argument controls the type of normalization used when generating conservative weights. This option only applies to weights generated with regridmethod=ESMF_REGRIDMETHOD_CONSERVE. Please see Section 54.45 for a list of valid options. If not specified normType defaults to ESMF_NORMTYPE_DSTAREA.
[extrapMethod]
The type of extrapolation. Please see Section 54.18 for a list of valid options. If not specified, defaults to ESMF_EXTRAPMETHOD_NONE.
[extrapNumSrcPnts]
The number of source points to use for the extrapolation methods that use more than one source point (e.g. ESMF_EXTRAPMETHOD_NEAREST_IDAVG). If not specified, defaults to 8.
[extrapDistExponent]
The exponent to raise the distance to when calculating weights for the ESMF_EXTRAPMETHOD_NEAREST_IDAVG extrapolation method. A higher value reduces the influence of more distant points. If not specified, defaults to 2.0.
[unmappedaction]
Specifies what should happen if there are destination points that can't be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR.
[ignoreDegenerate]
Ignore degenerate cells when checking the input Grids or Meshes for errors. If this is set to true, then the regridding proceeds, but degenerate cells will be skipped. If set to false, a degenerate cell produces an error. If not specified, ignoreDegenerate defaults to false.
[srcFileType]
The file format of the source grid. Please see Section 54.20 for a list of valid options. If not specifed, the program will determine the file format automatically.
[dstFileType]
The file format of the destination grid. Please see Section 54.20 for a list of valid options. If not specifed, the program will determine the file format automatically.
[srcRegionalFlag]
If .TRUE., the source grid is a regional grid, otherwise, it is a global grid. The default value is .FALSE.
[dstRegionalFlag]
If .TRUE., the destination grid is a regional grid, otherwise, it is a global grid. The default value is .FALSE.
[srcMeshname]
If the source file is in UGRID format, this argument is required to define the dummy variable name in the grid file that contains the mesh topology info.
[dstMeshname]
If the destination file is in UGRID format, this argument is required to define the dummy variable name in the grid file that contains the mesh topology info.
[srcMissingValueFlag]
If .TRUE., the source grid mask will be constructed using the missing values of the variable defined in srcMissingValueVar. This flag is only used for the grid defined in the GRIDSPEC or the UGRID file formats. The default value is .FALSE..
[srcMissingValueVar]
If srcMissingValueFlag is .TRUE., the argument is required to define the variable name whose missing values will be used to construct the grid mask. It is only used for the grid defined in the GRIDSPEC or the UGRID file formats.
[dstMissingValueFlag]
If .TRUE., the destination grid mask will be constructed using the missing values of the variable defined in dstMissingValueVar. This flag is only used for the grid defined in the GRIDSPEC or the UGRID file formats. The default value is .FALSE..
[dstMissingValueVar]
If dstMissingValueFlag is .TRUE., the argument is required to define the variable name whose missing values will be used to construct the grid mask. It is only used for the grid defined in the GRIDSPEC or the UGRID file formats.
[useSrcCoordFlag]
If .TRUE., the coordinate variables defined in srcCoordinateVars will be used as the longitude and latitude variables for the source grid. This flag is only used for the GRIDSPEC file format. The default is .FALSE.
[srcCoordinateVars]
If useSrcCoordFlag is .TRUE., this argument defines the longitude and ! latitude variables in the source grid file to be used for the regrid. This argument is only used when the grid file is in GRIDSPEC format. srcCoordinateVars should be a array of 2 elements.
[useDstCoordFlag]
If .TRUE., the coordinate variables defined in dstCoordinateVars will be used as the longitude and latitude variables for the destination grid. This flag is only used for the GRIDSPEC file format. The default is .FALSE.
[dstCoordinateVars]
If useDstCoordFlag is .TRUE., this argument defines the longitude and latitude variables in the destination grid file to be used for the regrid. This argument is only used when the grid file is in GRIDSPEC format. dstCoordinateVars should be a array of 2 elements.
[useSrcCornerFlag]
If useSrcCornerFlag is .TRUE., the corner coordinates of the source file will be used for regridding. Otherwise, the center coordinates will be us ed. The default is .FALSE. The corner stagger is not supported for the SCRIP formatted input grid or multi-tile GRIDSPEC MOSAIC input grid.
[useDstCornerFlag]
If useDstCornerFlag is .TRUE., the corner coordinates of the destination file will be used for regridding. Otherwise, the center coordinates will be used. The default is .FALSE. The corner stagger is not supported for the SCRIP formatted input grid or multi-tile GRIDSPEC MOSAIC input grid.
[useUserAreaFlag]
If .TRUE., the element area values defined in the grid files are used. Only the SCRIP and ESMF format grid files have user specified areas. This flag is only used for conservative regridding. The default is .FALSE..
[largefileFlag]
If .TRUE., the output weight file is in NetCDF 64bit offset format. The default is .FALSE..
[netcdf4fileFlag]
If .TRUE., the output weight file is in NetCDF4 file format. The default is .FALSE..
[weightOnlyFlag]
If .TRUE., the output weight file only contains factorList and factorIndexList. The default is .FALSE..
[tileFilePath]
Optional argument to define the path where the tile files reside. If it is given, it overwrites the path defined in gridlocation variable in the mosaic file.
[verboseFlag]
If .TRUE., it will print summary information about the regrid parameters, default to .FALSE..
[checkFlag]
checkFlag value to pass into ESMF_FieldRegridStore(), if not provided has same default as ESMF_FieldRegridStore() which is false.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

24.3.2 ESMF_RegridWeightGen - Generate regrid routeHandle and an optional weight file from grid files with user-specified distribution


INTERFACE:

   ! Private name; call using ESMF_RegridWeightGen()
   subroutine ESMF_RegridWeightGenDG(srcFile, dstFile, regridRouteHandle, &
     srcElementDistgrid, dstElementDistgrid, &
     srcNodalDistgrid, dstNodalDistgrid, &
     weightFile, regridmethod, lineType, normType, &
     extrapMethod, extrapNumSrcPnts, extrapDistExponent, extrapNumLevels,&
     unmappedaction, ignoreDegenerate, useUserAreaFlag, &
     largefileFlag, netcdf4fileFlag, &
     weightOnlyFlag, verboseFlag, rc)
ARGUMENTS:
 
   character(len=*),             intent(in)            :: srcFile
   character(len=*),             intent(in)            :: dstFile
   type(ESMF_RouteHandle),       intent(out)           :: regridRouteHandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DistGrid),          intent(in),  optional :: srcElementDistgrid
   type(ESMF_DistGrid),          intent(in),  optional :: dstElementDistgrid
   character(len=*),             intent(in),  optional :: weightFile
   type(ESMF_DistGrid),          intent(in),  optional :: srcNodalDistgrid
   type(ESMF_DistGrid),          intent(in),  optional :: dstNodalDistgrid
   type(ESMF_RegridMethod_Flag), intent(in),  optional :: regridmethod
   type(ESMF_LineType_Flag),     intent(in),  optional :: lineType
   type(ESMF_NormType_Flag),     intent(in),  optional :: normType
   type(ESMF_ExtrapMethod_Flag),   intent(in),    optional :: extrapMethod
   integer,                        intent(in),    optional :: extrapNumSrcPnts
   real,                           intent(in),    optional :: extrapDistExponent
   integer,                      intent(in),  optional :: extrapNumLevels
   type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction
   logical,                      intent(in),  optional :: ignoreDegenerate
   logical,                      intent(in),  optional :: useUserAreaFlag
   logical,                      intent(in),  optional :: largefileFlag
   logical,                      intent(in),  optional :: netcdf4fileFlag
   logical,                      intent(in),  optional :: weightOnlyFlag
   logical,                      intent(in),  optional :: verboseFlag
   integer,                      intent(out), optional :: rc
DESCRIPTION:

This subroutine does online regridding weight generation from files with user specified distribution. The main differences between this API and the one in 24.3.1 are listed below:

The arguments are:

srcFile
The source grid file name in SCRIP grid file format
dstFile
The destination grid file name in SCRIP grid file format
regridRouteHandle
The regrid RouteHandle returned by ESMF_FieldRegridStore()
srcElementDistgrid
An optional distGrid that specifies the distribution of the source grid's elements. If not specified, a system-defined block decomposition is used.
dstElementDistgrid
An optional distGrid that specifies the distribution of the destination grid's elements. If not specified, a system-defined block decomposition is used.
weightFile
The interpolation weight file name. If present, an output weight file will be generated.
srcNodalDistgrid
An optional distGrid that specifies the distribution of the source grid's nodes
dstNodalDistgrid
An optional distGrid that specifies the distribution of the destination grid's nodes
[regridmethod]
The type of interpolation. Please see Section 54.51 for a list of valid options. If not specified, defaults to ESMF_REGRIDMETHOD_BILINEAR.
[lineType]
This argument controls the path of the line which connects two points on a sphere surface. This in turn controls the path along which distances are calculated and the shape of the edges that make up a cell. Both of these quantities can influence how interpolation weights are calculated. As would be expected, this argument is only applicable when srcField and dstField are built on grids which lie on the surface of a sphere. Section 54.36 shows a list of valid options for this argument. If not specified, the default depends on the regrid method. Section 54.36 has the defaults by line type. Figure 24.2.16 shows which line types are supported for each regrid method as well as showing the default line type by regrid method.
[normType]
This argument controls the type of normalization used when generating conservative weights. This option only applies to weights generated with regridmethod=ESMF_REGRIDMETHOD_CONSERVE. Please see Section 54.45 for a list of valid options. If not specified normType defaults to ESMF_NORMTYPE_DSTAREA.
[extrapMethod]
The type of extrapolation. Please see Section 54.18 for a list of valid options. If not specified, defaults to ESMF_EXTRAPMETHOD_NONE.
[extrapNumSrcPnts]
The number of source points to use for the extrapolation methods that use more than one source point (e.g. ESMF_EXTRAPMETHOD_NEAREST_IDAVG). If not specified, defaults to 8..
[extrapDistExponent]
The exponent to raise the distance to when calculating weights for the ESMF_EXTRAPMETHOD_NEAREST_IDAVG extrapolation method. A higher value reduces the influence of more distant points. If not specified, defaults to 2.0.
[unmappedaction]
Specifies what should happen if there are destination points that can't be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR.
[ignoreDegenerate]
Ignore degenerate cells when checking the input Grids or Meshes for errors. If this is set to true, then the regridding proceeds, but degenerate cells will be skipped. If set to false, a degenerate cell produces an error. If not specified, ignoreDegenerate defaults to false.
[useUserAreaFlag]
If .TRUE., the element area values defined in the grid files are used. Only the SCRIP and ESMF format grid files have user specified areas. This flag is only used for conservative regridding. The default is .FALSE.
[largefileFlag]
If .TRUE., the output weight file is in NetCDF 64bit offset format. The default is .FALSE.
[netcdf4fileFlag]
If .TRUE., the output weight file is in NetCDF4 file format. The default is .FALSE.
[weightOnlyFlag]
If .TRUE., the output weight file only contains factorList and factorIndexList. The default is .FALSE.
[verboseFlag]
If .TRUE., it will print summary information about the regrid parameters, default to .FALSE.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

24.3.3 ESMF_FileRegrid - Regrid variables defined in the grid files


INTERFACE:

   subroutine ESMF_FileRegrid(srcFile, dstFile, srcVarName, dstVarName, &
     dstLoc, srcDataFile, dstDataFile, tileFilePath, &
     dstCoordVars, regridmethod, polemethod, regridPoleNPnts, &
     unmappedaction, ignoreDegenerate, srcRegionalFlag, dstRegionalFlag, &
     verboseFlag, rc)
ARGUMENTS:
 
   character(len=*),             intent(in)            :: srcFile
   character(len=*),             intent(in)            :: dstFile
   character(len=*),             intent(in)            :: srcVarName
   character(len=*),             intent(in)            :: dstVarName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   character(len=*),             intent(in),  optional :: dstLoc
   character(len=*),             intent(in),  optional :: srcDataFile     
   character(len=*),             intent(in),  optional :: dstDataFile     
   character(len=*),             intent(in),  optional :: tileFilePath
   character(len=*),             intent(in),  optional :: dstCoordVars
   type(ESMF_RegridMethod_Flag), intent(in),  optional :: regridmethod
   type(ESMF_PoleMethod_Flag),   intent(in),  optional :: polemethod
   integer,                      intent(in),  optional :: regridPoleNPnts
   type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction
   logical,                      intent(in),  optional :: ignoreDegenerate
   logical,                      intent(in),  optional :: srcRegionalFlag
   logical,                      intent(in),  optional :: dstRegionalFlag
   logical,                      intent(in),  optional :: verboseFlag
   integer,                      intent(out), optional :: rc
DESCRIPTION:

This subroutine provides the same function as the ESMF_Regrid application described in Section 13. It takes two grid files in NetCDF format and interpolate the variable defined in the source grid file to the destination variable using one of the ESMF supported regrid methods - bilinear (24.2.1), higher-order patch (24.2.2), first order conservative (24.2.5) or nearest neighbor methods. The grid files can be in one of the following two formats:

The optional arguments allow users to specify various options to control the regrid operation, such as which pole option to use, or whether to use user-specified area in the conservative regridding. The acceptable values and the default value for the optional arguments are listed below.

The arguments are:

srcFile
The source grid file name.
dstFile
The destination grid file name.
srcVarName
The source variable names to be regridded. If more than one, separate them by comma.
dstVarName
The destination variable names to be regridded to. If more than one, separate them by comma.
[dstLoc]
The destination variable's location, either 'node' or 'face'. This argument is only used when the destination grid file is UGRID, the regridding method is non-conservative and the destination variable does not exist in the destination grid file. If not specified, default is 'face'.
[srcDataFile]
The input data file prefix if the srcFile is in GRIDSPEC MOSAIC fileformat. The tilename and the file extension (.nc) will be added to the prefix. The tilename is defined in the MOSAIC file using variable "gridtiles".
[dstDataFile]
The output data file prefix if the dstFile is in GRIDSPEC MOSAIC fileformat. The tilename and the file extension (.nc) will be added to the prefix. The tilename is defined in the MOSAIC file using variable "gridtiles".
[tileFilePath]
The alternative file path for the tile files and mosaic data files when either srcFile or dstFile is a GRIDSPEC MOSAIC grid. The path can be either relative or absolute. If it is relative, it is relative to the working directory. When specified, the gridlocation variable defined in the Mosaic file will be ignored.
[dstCoordVars]
The destination coordinate variable names if the dstVarName does not exist in the dstFile
[regridmethod]
The type of interpolation. Please see Section 54.51 for a list of valid options. If not specified, defaults to ESMF_REGRIDMETHOD_BILINEAR.
[polemethod]
A flag to indicate which type of artificial pole to construct on the source Grid for regridding. Please see Section 54.48 for a list of valid options. The default value varies depending on the regridding method and the grid type and format.
[regridPoleNPnts]
If polemethod is set to ESMF_POLEMETHOD_NPNTAVG, this argument is required to specify how many points should be averaged over at the pole.
[unmappedaction]
Specifies what should happen if there are destination points that can't be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR.
[ignoreDegenerate]
Ignore degenerate cells when checking the input Grids or Meshes for errors. If this is set to true, then the regridding proceeds, but degenerate cells will be skipped. If set to false, a degenerate cell produces an error. If not specified, ignoreDegenerate defaults to false.
[srcRegionalFlag]
If .TRUE., the source grid is a regional grid, otherwise, it is a global grid. The default value is .FALSE.
[dstRegionalFlag]
If .TRUE., the destination grid is a regional grid, otherwise, it is a global grid. The default value is .FALSE.
[verboseFlag]
If .TRUE., it will print summary information about the regrid parameters, default to .FALSE.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

24.4 Restrictions and Future Work

  1. 32-bit index limitation: Currently all index space dimensions in an ESMF object are represented by signed 32-bit integers. This limits the number of elements in one-dimensional objects to the 32-bit limit. This limit can be crossed by higher dimensional objects, where the product space is only limited by the 64-bit sequence index representation.

25 FieldBundle Class

25.1 Description

A FieldBundle functions mainly as a convenient container for storing similar Fields. It represents “bundles” of Fields that are discretized on the same Grid, Mesh, LocStream, or XGrid and distributed in the same manner. The FieldBundle is an important data structure because it can be added to a State, which is used for sending and receiving data between Components.

In the common case where FieldBundle is built on top of a Grid, Fields within a FieldBundle may be located at different locations relative to the vertices of their common Grid. The Fields in a FieldBundle may be of different dimensions, as long as the Grid dimensions that are distributed are the same. For example, a surface Field on a distributed lat/lon Grid and a 3D Field with an added vertical dimension on the same distributed lat/lon Grid can be included in the same FieldBundle.

FieldBundles can be created and destroyed, can have Attributes added or retrieved, and can have Fields added, removed, replaced, or retrieved. Methods include queries that return information about the FieldBundle itself and about the Fields that it contains. The Fortran data pointer of a Field within a FieldBundle can be obtained by first retrieving the Field with a call to ESMF_FieldBundleGet(), and then using ESMF_FieldGet() to get the data.

In the future FieldBundles will serve as a mechanism for performance optimization. ESMF will take advantage of the similarities of the Fields within a FieldBundle to optimize collective communication, I/O, and regridding. See Section 25.3 for a description of features that are scheduled for future work.

25.2 Use and Examples

Examples of creating, accessing and destroying FieldBundles and their constituent Fields are provided in this section, along with some notes on FieldBundle methods.


25.2.1 Creating a FieldBundle from a list of Fields

A user can create a FieldBundle from a predefined list of Fields. In the following example, we first create an ESMF_Grid, then build 3 different ESMF_Fields with different names. The ESMF_FieldBundle is created from the list of 3 Fields.

!-------------------------------------------------------------------------
!   !  Create several Fields and add them to a new FieldBundle.
 
    grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/100,200/), &
                                  regDecomp=(/2,2/), name="atmgrid", rc=rc)

    call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R8, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    field(1) = ESMF_FieldCreate(grid, arrayspec, &
                                staggerloc=ESMF_STAGGERLOC_CENTER, &
                                name="temperature", rc=rc)

    field(2) = ESMF_FieldCreate(grid, arrayspec, &
                                staggerloc=ESMF_STAGGERLOC_CENTER, &
                                name="pressure", rc=rc)

    field(3) = ESMF_FieldCreate(grid, arrayspec, &
                                staggerloc=ESMF_STAGGERLOC_CENTER, &
                                name="heat flux", rc=rc)

    bundle1 = ESMF_FieldBundleCreate(fieldList=field(1:3), &
                                name="atmosphere data", rc=rc)

    print *, "FieldBundle example 1 returned"


25.2.2 Creating an empty FieldBundle then add one Field to it

A user can create an empty FieldBundle then add Fields to the empty FieldBundle. In the following example, we use the previously defined ESMF_Grid to build an ESMF_Field. An empty ESMF_FieldBundle is created, then the Field is added to the FieldBundle.

!-------------------------------------------------------------------------
!   !  Create an empty FieldBundle and then add a single field to it.


    simplefield = ESMF_FieldCreate(grid, arrayspec, &
                  staggerloc=ESMF_STAGGERLOC_CENTER, name="rh", rc=rc)

    bundle2 = ESMF_FieldBundleCreate(name="time step 1", rc=rc)

    call ESMF_FieldBundleAdd(bundle2, (/simplefield/), rc=rc)

    call ESMF_FieldBundleGet(bundle2, fieldCount=fieldcount, rc=rc)

    print *, "FieldBundle example 2 returned, fieldcount =", fieldcount


25.2.3 Creating an empty FieldBundle then add a list of Fields to it

A user can create an empty FieldBundle then add multiple Fields to the empty FieldBundle. In the following example, we use the previously defined ESMF_Grid and ESMF_Fields. An empty ESMF_FieldBundle is created, then three Fields are added to the FieldBundle.

!-------------------------------------------------------------------------
!   !  Create an empty FieldBundle and then add multiple fields to it.

    bundle3 = ESMF_FieldBundleCreate(name="southern hemisphere", rc=rc)

    call ESMF_FieldBundleAdd(bundle3, field(1:3), rc=rc)

    call ESMF_FieldBundleGet(bundle3, fieldCount=fieldcount, rc=rc)

    print *, "FieldBundle example 3 returned, fieldcount =", fieldcount


25.2.4 Query a Field stored in the FieldBundle by name or index

Users can query a Field stored in a FieldBundle by the Field's name or index. In the following example, the pressure Field stored in FieldBundle is queried by its name then by its index through ESMF_FieldBundleGet() method.

!-------------------------------------------------------------------------
!   !  Get a Field back from a FieldBundle, first by name and then by index.
!   !  Also get the FieldBundle name.

    call ESMF_FieldBundleGet(bundle1, "pressure", field=returnedfield1, rc=rc)

    call ESMF_FieldGet(returnedfield1, name=fname1, rc=rc)

    call ESMF_FieldBundleGet(bundle1, 2, returnedfield2, rc=rc)

    call ESMF_FieldGet(returnedfield2, name=fname2, rc=rc)

    call ESMF_FieldBundleGet(bundle1, name=bname1, rc=rc)

    print *, "FieldBundle example 4 returned, field names = ", &
                   trim(fname1), ", ", trim(fname2)
    print *, "FieldBundle name = ", trim(bname1)


25.2.5 Query FieldBundle for Fields list either alphabetical or in order of addition

Users can query the list of Fields stored in a FieldBundle. By default the returned list of Fields are ordered alphabetically by the Field names. Users can also retrieve the list of Fields in the order by which the Fields were added to the FieldBundle.

    call ESMF_FieldBundleGet(bundle1, fieldList=r_fields, rc=rc)

    do i = 1, 3
      call ESMF_FieldGet(r_fields(i), name=fname1, rc=rc)

      print *, fname1
    enddo

    call ESMF_FieldBundleGet(bundle1, fieldList=r_fields, &
      itemorderflag=ESMF_ITEMORDER_ADDORDER, rc=rc)

    do i = 1, 3
      call ESMF_FieldGet(r_fields(i), name=fname1, rc=rc)

      print *, fname1
    enddo


25.2.6 Create a packed FieldBundle on a Grid

Create a packed fieldbundle from user supplied field names and a packed Fortran array pointer that contains the data of the packed fields on a Grid.

Create a 2D grid of 4x1 regular decomposition on 4 PETs, each PET has 10x50 elements. The index space of the entire Grid is 40x50.

  gridxy = ESMF_GridCreateNoPeriDim(maxIndex=(/40,50/), regDecomp=(/4,1/), rc=rc)

Allocate a packed Fortran array pointer containing 10 packed fields, each field has 3 time slices and uses the 2D grid created. Note that gridToFieldMap uses the position of the grid dimension as elements, 3rd element of the packedPtr is 10, 4th element of the packedPtr is 50.

  allocate(packedPtr(10, 3, 10, 50)) ! fieldDim, time, y, x
  fieldDim = 1
  packedFB = ESMF_FieldBundleCreate(fieldNameList, packedPtr, gridxy, fieldDim, &
  gridToFieldMap=(/3,4/), staggerloc=ESMF_Staggerloc_Center, rc=rc)


25.2.7 Create a packed FieldBundle on a Mesh

Similarly we could create a packed fieldbundle from user supplied field names and a packed Fortran array pointer that contains the data of the packed fields on a Mesh.

Due to the verbosity of the MeshCreate process, the code for MeshCreate is not shown below, user can either refer to the MeshCreate section 33.3.1 or examine the FieldBundleCreate example source code contained in the ESMF source distribution directly. A ESMF Mesh on 4 PETs with one mesh element on each PET is created.

Allocate the packed Fortran array pointer, the first dimension is fieldDim; second dimension is the data associated with mesh element, since there is only one mesh element on each processor in this example, the allocation is 1; last dimension is the time dimension which contains 3 time slices.

      allocate(packedPtr3D(10, 1, 3))
      fieldDim = 1
      packedFB = ESMF_FieldBundleCreate(fieldNameList, packedPtr3D, meshEx, fieldDim, &
        gridToFieldMap=(/2/), meshloc=ESMF_MESHLOC_ELEMENT, rc=rc)

25.2.8 Destroy a FieldBundle

The user must call ESMF_FieldBundleDestroy() before deleting any of the Fields it contains. Because Fields can be shared by multiple FieldBundles and States, they are not deleted by this call.

!-------------------------------------------------------------------------

     call ESMF_FieldBundleDestroy(bundle1, rc=rc)


25.2.9 Redistribute data from a source FieldBundle to a destination FieldBundle

The ESMF_FieldBundleRedist interface can be used to redistribute data from source FieldBundle to destination FieldBundle. This interface is overloaded by type and kind; In the version of ESMF_FieldBundleRedist without factor argument, a default value of factor 1 is used.

In this example, we first create two FieldBundles, a source FieldBundle and a destination FieldBundle. Then we use ESMF_FieldBundleRedist to redistribute data from source FieldBundle to destination FieldBundle.

    ! perform redist
    call ESMF_FieldBundleRedistStore(srcFieldBundle, dstFieldBundle, &
         routehandle, rc=rc)

    call ESMF_FieldBundleRedist(srcFieldBundle, dstFieldBundle, &
         routehandle, rc=rc)


25.2.10 Redistribute data from a packed source FieldBundle to a packed destination FieldBundle

The ESMF_FieldBundleRedist interface can be used to redistribute data from source FieldBundle to destination FieldBundle when both Bundles are packed with same number of fields.

In this example, we first create two packed FieldBundles, a source FieldBundle and a destination FieldBundle. Then we use ESMF_FieldBundleRedist to redistribute data from source FieldBundle to destination FieldBundle.

The same Grid is used where the source and destination packed FieldBundle are built upon. Source and destination Bundle have different memory layout.

    allocate(srcfptr(3,5,10), dstfptr(10,5,3))
    srcfptr = lpe
    srcFieldBundle = ESMF_FieldBundleCreate((/'field01', 'field02', 'field03'/), &
      srcfptr, grid, 1, gridToFieldMap=(/2,3/), rc=rc)

    dstFieldBundle = ESMF_FieldBundleCreate((/'field01', 'field02', 'field03'/), &
      dstfptr, grid, 3, gridToFieldMap=(/2,1/), rc=rc)

    ! perform redist
    call ESMF_FieldBundleRedistStore(srcFieldBundle, dstFieldBundle, &
         routehandle, rc=rc)

    call ESMF_FieldBundleRedist(srcFieldBundle, dstFieldBundle, &
         routehandle, rc=rc)


25.2.11 Perform sparse matrix multiplication from a source FieldBundle to a destination FieldBundle

The ESMF_FieldBundleSMM interface can be used to perform SMM from source FieldBundle to destination FieldBundle. This interface is overloaded by type and kind;

In this example, we first create two FieldBundles, a source FieldBundle and a destination FieldBundle. Then we use ESMF_FieldBundleSMM to perform sparse matrix multiplication from source FieldBundle to destination FieldBundle.

The operation performed in this example is better illustrated in section 26.3.33.

Section 28.2.18 provides a detailed discussion of the sparse matrix multiplication operation implemented in ESMF.

    call ESMF_VMGetCurrent(vm, rc=rc)

    call ESMF_VMGet(vm, localPet=lpe, rc=rc)

    ! create distgrid and grid
    distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
        regDecomp=(/4/), &
        rc=rc)

    grid = ESMF_GridCreate(distgrid=distgrid, &
        gridEdgeLWidth=(/0/), gridEdgeUWidth=(/0/), &
        name="grid", rc=rc)

    call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_I4, rc=rc)

    ! create field bundles and fields
    srcFieldBundle = ESMF_FieldBundleCreate(rc=rc)

    dstFieldBundle = ESMF_FieldBundleCreate(rc=rc)

    do i = 1, 3
        srcField(i) = ESMF_FieldCreate(grid, arrayspec, &
            totalLWidth=(/1/), totalUWidth=(/2/), &
            rc=rc)

        call ESMF_FieldGet(srcField(i), localDe=0, farrayPtr=srcfptr, rc=rc)

        srcfptr = 1

        call ESMF_FieldBundleAdd(srcFieldBundle, (/srcField(i)/), rc=rc)

        dstField(i) = ESMF_FieldCreate(grid, arrayspec, &
            totalLWidth=(/1/), totalUWidth=(/2/), &
            rc=rc)

        call ESMF_FieldGet(dstField(i), localDe=0, farrayPtr=dstfptr, rc=rc)

        dstfptr = 0

        call ESMF_FieldBundleAdd(dstFieldBundle, (/dstField(i)/), rc=rc)

    enddo

    ! initialize factorList and factorIndexList
    allocate(factorList(4))
    allocate(factorIndexList(2,4))
    factorList = (/1,2,3,4/)
    factorIndexList(1,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
    factorIndexList(2,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
    call ESMF_FieldBundleSMMStore(srcFieldBundle, dstFieldBundle, &
        routehandle, factorList, factorIndexList, rc=rc)

    ! perform smm
    call ESMF_FieldBundleSMM(srcFieldBundle, dstFieldBundle, routehandle, &
          rc=rc)

    ! release SMM route handle
    call ESMF_FieldBundleSMMRelease(routehandle, rc=rc)


25.2.12 Perform FieldBundle halo update

ESMF_FieldBundleHalo interface can be used to perform halo updates for all the Fields contained in the ESMF_FieldBundle.

In this example, we will set up a FieldBundle for a 2D inviscid and compressible flow problem. We will illustrate the FieldBundle halo update operation but we will not solve the non-linear PDEs. The emphasis here is to demonstrate how to set up halo regions, how a numerical scheme updates the exclusive regions, and how a halo update communicates data in the halo regions. Here are the governing equations:

$u_t + u u_x + v u_y + \frac{1}{\rho} p_x = 0$ (conservation of momentum in x-direction)

$v_t + u v_x + v v_y + \frac{1}{\rho} p_y = 0$ (conservation of momentum in y-direction)

${\rho}_t + {\rho u}_x + {\rho v}_y = 0$ (conservation of mass)

$\frac{\rho}{\rho^\gamma} + u {(\frac{p}{\rho^\gamma})}_x + v {(\frac{p}{\rho^\gamma})}_y = 0$ (conservation of energy)

The four unknowns are pressure $p$, density $\rho$, velocity ($u$, $v$). The grids are set up using Arakawa D stagger ($p$ on corner, $\rho$ at center, $u$ and $v$ on edges). $p$, $\rho$, $u$, and $v$ are bounded by necessary boundary conditions and initial conditions.

Section 28.2.15 provides a detailed discussion of the halo operation implemented in ESMF.

    ! create distgrid and grid according to the following decomposition 
    ! and stagger pattern, r is density.
    !
    ! p--------u-------+p+-------u--------p
    ! !                 |                 |
    ! !                 |                 |
    ! !                 |                 |
    ! v        r        v        r        v
    ! !      PET 0      |      PET 1      |
    ! !                 |                 |
    ! !                 |                 |
    ! p--------u-------+p+-------u--------p
    ! !                 |                 |
    ! !                 |                 |
    ! !                 |                 |
    ! v        r        v        r        v
    ! !      PET 2      |      PET 3      |
    ! !                 |                 |
    ! !                 |                 |
    ! p--------u-------+p+-------u--------p
    !
    distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/256,256/), &
        regDecomp=(/2,2/), &
        rc=rc)

    grid = ESMF_GridCreate(distgrid=distgrid, name="grid", rc=rc)

    call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc=rc)

    ! create field bundles and fields
    fieldBundle = ESMF_FieldBundleCreate(rc=rc)

    ! set up exclusive/total region for the fields
    !
    ! halo: L/U, nDim, nField, nPet
    ! halo configuration for pressure, and similarly for density, u, and v
    halo(1,1,1,1) = 0
    halo(2,1,1,1) = 0
    halo(1,2,1,1) = 0
    halo(2,2,1,1) = 0
    halo(1,1,1,2) = 1   ! halo in x direction on left hand side of pet 1
    halo(2,1,1,2) = 0
    halo(1,2,1,2) = 0
    halo(2,2,1,2) = 0
    halo(1,1,1,3) = 0
    halo(2,1,1,3) = 1   ! halo in y direction on upper side of pet 2
    halo(1,2,1,3) = 0
    halo(2,2,1,3) = 0
    halo(1,1,1,4) = 1   ! halo in x direction on left hand side of pet 3
    halo(2,1,1,4) = 1   ! halo in y direction on upper side of pet 3
    halo(1,2,1,4) = 0
    halo(2,2,1,4) = 0

    ! names and staggers of the 4 unknown fields
    names(1) = "pressure"
    names(2) = "density"
    names(3) = "u"
    names(4) = "v"
    staggers(1) = ESMF_STAGGERLOC_CORNER
    staggers(2) = ESMF_STAGGERLOC_CENTER
    staggers(3) = ESMF_STAGGERLOC_EDGE2
    staggers(4) = ESMF_STAGGERLOC_EDGE1
    
    ! create a FieldBundle
    lpe = lpe + 1
    do i = 1, 4
        field(i) = ESMF_FieldCreate(grid, arrayspec, &
                totalLWidth=(/halo(1,1,i,lpe), halo(1,2,i,lpe)/), &
                totalUWidth=(/halo(2,1,i,lpe), halo(2,2,i,lpe)/), &
                staggerloc=staggers(i), name=names(i), &
                rc=rc)

        call ESMF_FieldBundleAdd(fieldBundle, (/field(i)/), rc=rc)

    enddo

    ! compute the routehandle
    call ESMF_FieldBundleHaloStore(fieldBundle, routehandle=routehandle, &
                                   rc=rc)

    do iter = 1, 10
        do i = 1, 4
            call ESMF_FieldGet(field(i), farrayPtr=fptr, &
                exclusiveLBound=excllb, exclusiveUBound=exclub, rc=rc)

            sizes = exclub - excllb
            ! fill the total region with 0.
            fptr = 0.
            ! only update the exclusive region on local PET
            do j = excllb(1), exclub(1)
              do k = excllb(2), exclub(2)
                fptr(j,k) = iter * cos(2.*PI*j/sizes(1))*sin(2.*PI*k/sizes(2))
              enddo 
            enddo 
        enddo
        ! call halo execution to update the data in the halo region,
        ! it can be verified that the halo regions change from 0. 
        ! to non zero values.
        call ESMF_FieldBundleHalo(fieldbundle, routehandle=routehandle, rc=rc)

    enddo
    ! release halo route handle
    call ESMF_FieldBundleHaloRelease(routehandle, rc=rc)

25.3 Restrictions and Future Work

  1. No enforcement of the same Grid, Mesh, LocStream, or XGrid restriction. While the documentation indicates in several places (including the Design and Implementation Notes) that a FieldBundle can only contain Fields that are built on the same Grid, Mesh, LocStream, or XGrid, and all Fields must have the same distribution, the actual FieldBundle implementation is more general and supports bundling of any Fields. The documentation, however, is in line with the long term plan of making the restrictive FieldBundle definition the default behavior. The more general bundling option would then be retained as a special case that requires explicit specification by the user. There is currently no functional difference in the FieldBundle implementation that profits from the documented restrictive approach. In addition, the general bundling option has been supported for a long time. Note however that the documented restrictive behavior is the anticipated long term default for FieldBundles.

  2. No mathematical operators. The FieldBundle class does not support differential or other mathematical operators. We do not anticipate providing this functionality in the near future.

  3. Limited validation and print options. We are planning to increase the number of validity checks available for FieldBundles as soon as possible. We also will be working on print options.

  4. Packed data has limited supported. One of the options that we are currently working on for FieldBundles is packing. Packing means that the data from all the Fields that comprise the FieldBundle are manipulated collectively. This operation can be done without destroying the original Field data. Packing is being designed to facilitate optimized regridding, data communication, and I/O operations. This will reduce the latency overhead of the communication.

    CAUTION: For communication methods, the undistributed dimension representing the number of fields must have identical size between source and destination packed data. Communication methods do not permute the order of fields in the source and destination packed FieldBundle.

  5. Interleaving Fields within a FieldBundle. Data locality is important for performance on some computing platforms. An interleave option will be added to allow the user to create a packed FieldBundle in which Fields are either concatenated in memory or in which Field elements are interleaved.

25.4 Design and Implementation Notes

  1. Fields in a FieldBundle reference the same Grid, Mesh, LocStream, or XGrid. In order to reduce memory requirements and ensure consistency, the Fields within a FieldBundle all reference the same Grid, Mesh, LocStream, or XGrid object. This restriction may be relaxed in the future.

25.5 Class API: Basic FieldBundle Methods

25.5.1 ESMF_FieldBundleAssignment(=) - FieldBundle assignment


INTERFACE:

   interface assignment(=)
   fieldbundle1 = fieldbundle2
ARGUMENTS:
   type(ESMF_FieldBundle) :: fieldbundle1
   type(ESMF_FieldBundle) :: fieldbundle2
STATUS:

DESCRIPTION:

Assign fieldbundle1 as an alias to the same ESMF fieldbundle object in memory as fieldbundle2. If fieldbundle2 is invalid, then fieldbundle1 will be equally invalid after the assignment.

The arguments are:

fieldbundle1
The ESMF_FieldBundle object on the left hand side of the assignment.
fieldbundle2
The ESMF_FieldBundle object on the right hand side of the assignment.

25.5.2 ESMF_FieldBundleOperator(==) - FieldBundle equality operator


INTERFACE:

   interface operator(==)
   if (fieldbundle1 == fieldbundle2) then ... endif
   OR
   result = (fieldbundle1 == fieldbundle2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_FieldBundle), intent(in) :: fieldbundle1
   type(ESMF_FieldBundle), intent(in) :: fieldbundle2
STATUS:

DESCRIPTION:

Test whether fieldbundle1 and fieldbundle2 are valid aliases to the same ESMF fieldbundle object in memory. For a more general comparison of two ESMF FieldBundles, going beyond the simple alias test, the ESMF_FieldBundleMatch() function (not yet implemented) must be used.

The arguments are:

fieldbundle1
The ESMF_FieldBundle object on the left hand side of the equality operation.
fieldbundle2
The ESMF_FieldBundle object on the right hand side of the equality operation.

25.5.3 ESMF_FieldBundleOperator(/=) - FieldBundle not equal operator


INTERFACE:

   interface operator(/=)
   if (fieldbundle1 /= fieldbundle2) then ... endif
   OR
   result = (fieldbundle1 /= fieldbundle2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_FieldBundle), intent(in) :: fieldbundle1
   type(ESMF_FieldBundle), intent(in) :: fieldbundle2
STATUS:

DESCRIPTION:

Test whether fieldbundle1 and fieldbundle2 are not valid aliases to the same ESMF fieldbundle object in memory. For a more general comparison of two ESMF FieldBundles, going beyond the simple alias test, the ESMF_FieldBundleMatch() function (not yet implemented) must be used.

The arguments are:

fieldbundle1
The ESMF_FieldBundle object on the left hand side of the non-equality operation.
fieldbundle2
The ESMF_FieldBundle object on the right hand side of the non-equality operation.

25.5.4 ESMF_FieldBundleAdd - Add Fields to a FieldBundle


INTERFACE:

     ! Private name; call using ESMF_FieldBundleAdd()
     subroutine ESMF_FieldBundleAddList(fieldbundle, fieldList, &
       multiflag, relaxedflag, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
     type(ESMF_Field), intent(in) :: fieldList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: multiflag
     logical, intent(in), optional :: relaxedflag
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Add Field(s) to a FieldBundle. It is an error if fieldList contains Fields that match by name Fields already contained in fieldbundle when multiflag is set to .false. and relaxedflag is set to .false..

fieldbundle
ESMF_FieldBundle to be added to.
fieldList
List of ESMF_Field objects to be added.
[multiflag]
A setting of .true. allows multiple items with the same name to be added to ESMF_FieldBundle. For .false. added items must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "add" under multiflag=.false. mode, where it is not an error if fieldList contains items with names that are also found in ESMF_FieldBundle. The ESMF_FieldBundle is left unchanged for these items. For .false. this is treated as an error condition. The default setting is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.5 ESMF_FieldBundleAddReplace - Conditionally add or replace Fields in a FieldBundle


INTERFACE:

     subroutine ESMF_FieldBundleAddReplace(fieldbundle, fieldList, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
     type(ESMF_Field), intent(in) :: fieldList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Fields in fieldList that do not match any Fields by name in fieldbundle are added to the FieldBundle. Fields in fieldList that match any Fields by name in fieldbundle replace those Fields.

fieldbundle
ESMF_FieldBundle to be manipulated.
fieldList
List of ESMF_Field objects to be added or used as replacement.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.6 ESMF_FieldBundleCreate - Create a non packed FieldBundle from a list of Fields


INTERFACE:

   ! Private name; call using ESMF_FieldBundleCreate()
   function ESMF_FieldBundleCreateDefault(fieldList, &
       multiflag, relaxedflag, name, rc)
RETURN VALUE:
     type(ESMF_FieldBundle) :: ESMF_FieldBundleCreateDefault
ARGUMENTS:
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Field), intent(in), optional :: fieldList(:)
     logical, intent(in), optional :: multiflag
     logical, intent(in), optional :: relaxedflag
     character (len=*),intent(in), optional :: name
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_FieldBundle object from a list of existing Fields.

The creation of a FieldBundle leaves the bundled Fields unchanged, they remain valid individual objects. a FieldBundle is a light weight container of Field references. The actual data remains in place, there are no data movements or duplications associated with the creation of an FieldBundle.

[fieldList]
List of ESMF_Field objects to be bundled.
[multiflag]
A setting of .true. allows multiple items with the same name to be added to fieldbundle. For .false. added items must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "add" under multiflag=.false. mode, where it is not an error if fieldList contains items with names that are also found in fieldbundle. The fieldbundle is left unchanged for these items. For .false. this is treated as an error condition. The default setting is .false..
[name]
Name of the created ESMF_FieldBundle. A default name is generated if not specified.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.7 ESMF_FieldBundleCreate - Create a packed FieldBundle from Fortran array pointer and Grid


INTERFACE:

   ! Private name; call using ESMF_FieldBundleCreate() 
   function ESMF_FieldBundleCreateGrid<rank><type><kind>(fieldNameList, & 
   farrayPtr, grid, fieldDim, & 
   indexflag, staggerLoc, & 
   gridToFieldMap, & 
   totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
   type(ESMF_FieldBundle) :: ESMF_FieldBundleCreateGridDataPtr<rank><type><kind>
ARGUMENTS:
   character(len=*), intent(in) :: fieldNameList(:) 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: farrayPtr 
   type(ESMF_Grid), intent(in) :: grid 
   integer, intent(in) :: fieldDim 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_Index_Flag), intent(in), optional :: indexflag 
   type(ESMF_StaggerLoc), intent(in), optional :: staggerloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create a packed FieldBundle from user supplied list of field names, pre-allocated Fortran array pointer, and ESMF_Grid object.

The arguments are:

fieldNameList
A list of field names for the Fields held by the packed FieldBundle.
farrayPtr
Pre-allocated Fortran array pointer holding the memory of the list of Fields.
grid
The ESMF_Grid object on which the Fields in the packed FieldBundle are built.
fieldDim
The dimension in the farrayPtr that contains the indices of Fields to be packed.
[indexflag]
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options. All Fields in packed FieldBundle use identical indexflag setting.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER. All Fields in packed FieldBundle use identical staggerloc setting.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. All Fields in packed FieldBundle use identical gridToFieldMap setting.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farrayPtr. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the farrayPtr. That is, for each gridded dimension the farrayPtr size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ). All Fields in packed FieldBundle use identical totalLWidth setting.
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farrayPtr. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the farrayPtr. That is, for each gridded dimension the farrayPtr size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ). All Fields in packed FieldBundle use identical totalUWidth setting.
[name]
FieldBundle name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.8 ESMF_FieldBundleCreate - Create a packed FieldBundle from Fortran array pointer and Mesh


INTERFACE:

   ! Private name; call using ESMF_FieldBundleCreate() 
   function ESMF_FieldBundleCreateMesh<rank><type><kind>(fieldNameList, & 
   farrayPtr, Mesh, fieldDim, & 
   meshLoc, gridToFieldMap, name, rc)
RETURN VALUE:
   type(ESMF_FieldBundle) :: ESMF_FieldBundleCreateMeshDataPtr<rank><type><kind>
ARGUMENTS:
   character(len=*), intent(in) :: fieldNameList(:) 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: farrayPtr 
   type(ESMF_Mesh), intent(in) :: mesh 
   integer, intent(in) :: fieldDim 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_MeshLoc), intent(in), optional:: meshloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create a packed FieldBundle from user supplied list of field names, pre-allocated Fortran array pointer, and ESMF_Mesh object.

The arguments are:

fieldNameList
A list of field names for the Fields held by the packed FieldBundle.
farrayPtr
Pre-allocated Fortran array pointer holding the memory of the list of Fields.
mesh
The ESMF_Mesh object on which the Fields in the packed FieldBundle are built.
fieldDim
The dimension in the farrayPtr that contains the indices of Fields to be packed.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the mesh's dimCount. The list elements map each dimension of the mesh to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the mesh's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the mesh. Ungridded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. All Fields in packed FieldBundle use identical gridToFieldMap setting.
[name]
FieldBundle name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.9 ESMF_FieldBundleDestroy - Release resources associated with a FieldBundle


INTERFACE:

   subroutine ESMF_FieldBundleDestroy(fieldbundle, noGarbage, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: noGarbage
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Destroy an ESMF_FieldBundle object. The member Fields are not touched by this operation and remain valid objects that need to be destroyed individually if necessary.

The arguments are:

fieldbundle
ESMF_FieldBundle object to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.10 ESMF_FieldBundleGet - Get object-wide information from a FieldBundle


INTERFACE:

     ! Private name; call using ESMF_FieldBundleGet()
     subroutine ESMF_FieldBundleGetListAll(fieldbundle, &
       itemorderflag, geomtype, grid, locstream, mesh, xgrid, &
       fieldCount, fieldList, fieldNameList, isPacked, name, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_ItemOrder_Flag), intent(in), optional :: itemorderflag
     type(ESMF_GeomType_Flag), intent(out), optional :: geomtype
     type(ESMF_Grid), intent(out), optional :: grid
     type(ESMF_LocStream), intent(out), optional :: locstream
     type(ESMF_Mesh), intent(out), optional :: mesh
     type(ESMF_XGrid), intent(out), optional :: xgrid
     integer, intent(out), optional :: fieldCount
     type(ESMF_Field), intent(out), optional :: fieldList(:)
     character(len=*), intent(out), optional :: fieldNameList(:)
     logical, intent(out), optional :: isPacked
     character(len=*), intent(out), optional :: name
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get the list of all Fields and field names bundled in a FieldBundle.

fieldbundle
ESMF_FieldBundle to be queried.
[itemorderflag]
Specifies the order of the returned items in the fieldList or the fieldNameList. The default is ESMF_ITEMORDER_ABC. See 54.34 for a full list of options.
[geomtype]
Flag that indicates what type of geometry this FieldBundle object holds. Can be ESMF_GEOMTYPE_GRID, ESMF_GEOMTYPE_MESH, ESMF_GEOMTYPE_LOCSTREAM, ESMF_GEOMTYPE_XGRID
[grid]
The Grid object that this FieldBundle object holds.
[locstream]
The LocStream object that this FieldBundle object holds.
[mesh]
The Mesh object that this FieldBundle object holds.
[xgrid]
The XGrid object that this FieldBundle object holds.
[fieldCount]
Upon return holds the number of Fields bundled in the fieldbundle.
[fieldList]
Upon return holds a list of Fields bundled in ESMF_FieldBundle. The argument must be allocated to be at least of size fieldCount.
[fieldNameList]
Upon return holds a list of the names of the fields bundled in ESMF_FieldBundle. The argument must be allocated to be at least of size fieldCount.
[isPacked]
Upon return holds the information if this FieldBundle is packed.
[name]
Name of the fieldbundle object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.11 ESMF_FieldBundleGet - Get information about a Field by name and optionally return a Field


INTERFACE:

     ! Private name; call using ESMF_FieldBundleGet()
     subroutine ESMF_FieldBundleGetItem(fieldbundle, fieldName, &
       field, fieldCount, isPresent, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
     character(len=*), intent(in) :: fieldName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Field), intent(out), optional :: field
     integer, intent(out), optional :: fieldCount
     logical, intent(out), optional :: isPresent
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get information about items that match fieldName in FieldBundle.

fieldbundle
ESMF_FieldBundle to be queried.
fieldName
Specified name.
[field]
Upon return holds the requested field item. It is an error if this argument was specified and there is not exactly one field item in ESMF_FieldBundle that matches fieldName.
[fieldCount]
Number of Fields with fieldName in ESMF_FieldBundle.
[isPresent]
Upon return indicates whether field(s) with fieldName exist in ESMF_FieldBundle.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.12 ESMF_FieldBundleGet - Get a list of Fields by name


INTERFACE:

     ! Private name; call using ESMF_FieldBundleGet()
     subroutine ESMF_FieldBundleGetList(fieldbundle, fieldName, fieldList, &
       itemorderflag, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
     character(len=*), intent(in) :: fieldName
     type(ESMF_Field), intent(out) :: fieldList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_ItemOrder_Flag), intent(in), optional :: itemorderflag
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get the list of Fields from fieldbundle that match fieldName.

fieldbundle
ESMF_FieldBundle to be queried.
fieldName
Specified name.
fieldList
List of Fields in ESMF_FieldBundle that match fieldName. The argument must be allocated to be at least of size fieldCount returned for this fieldName.
[itemorderflag]
Specifies the order of the returned items in the fieldList. The default is ESMF_ITEMORDER_ABC. See 54.34 for a full list of options.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.13 ESMF_FieldBundleGet - Access the Field at a specific index in a FieldBundle


INTERFACE:

     ! Private name; call using ESMF_FieldBundleGet()
     subroutine ESMF_FieldBundleGetIndex(fieldbundle, fieldIndex, field, &
       rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
     integer, intent(in) :: fieldIndex
     type(ESMF_Field), intent(inout) :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
DESCRIPTION:

Get the fieldIndex-th Field from a FieldBundle. The order of the Fields in FieldBundle is not guranteed. If this call is used iteratively, then any Add, Replace, or Remove call on the FieldBundle can change the order and/or number of Fields in that FieldBundle.

fieldbundle
ESMF_FieldBundle to be queried.
fieldIndex
The index of the Field to be returned.
field
The fieldIndex-th Field to be returned.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.14 ESMF_FieldBundleGet - Get Fortran array pointer from a packed FieldBundle


INTERFACE:

   ! Private name; call using ESMF_FieldBundleGet() 
   function ESMF_FieldBundleGetDataPtr<rank><type><kind>(fieldBundle, & 
   localDe, farrayPtr, & 
   rc)
RETURN VALUE:
   type(ESMF_FieldBundle) :: ESMF_FieldBundleGetDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_FieldBundle), intent(in) :: fieldBundle 
   integer, intent(in), optional :: localDe 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: farrayPtr 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(out), optional :: rc
DESCRIPTION:

Get a Fortran pointer to DE-local memory allocation within packed FieldBundle. It's erroneous to perform this call on a FieldBundle that's not packed.

The arguments are:

fieldBundle
ESMF_FieldBundle object.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0. In the case where packed FieldBundle is created on a Grid, the number of localDes can be queried from the Grid attached to the FieldBundle. In the case where packed FieldBundle is created on a Mesh, the number of localDes is 1.
farrayPtr
Fortran array pointer which will be pointed at DE-local memory allocation in packed FieldBundle.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.15 ESMF_FieldBundleHalo - Execute a FieldBundle halo operation


INTERFACE:

   subroutine ESMF_FieldBundleHalo(fieldbundle, routehandle, &
     checkflag, rc)
ARGUMENTS:
         type(ESMF_FieldBundle), intent(inout) :: fieldbundle
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: checkflag
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed halo operation for the Fields in fieldbundle. The FieldBundle must match the respective FieldBundle used during ESMF_FieldBundleHaloStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

See ESMF_FieldBundleHaloStore() on how to precompute routehandle.

fieldbundle
ESMF_FieldBundle with source data. The data in this FieldBundle may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[checkflag]
If set to .TRUE. the input FieldBundle pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.16 ESMF_FieldBundleHaloRelease - Release resources associated with a FieldBundle halo operation


INTERFACE:

   subroutine ESMF_FieldBundleHaloRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: noGarbage
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with a FieldBundle halo operation. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.17 ESMF_FieldBundleHaloStore - Precompute a FieldBundle halo operation


INTERFACE:

     subroutine ESMF_FieldBundleHaloStore(fieldbundle, routehandle, &
       rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
     type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store a FieldBundle halo operation over the data in fieldbundle. By definition, all elements in the total Field regions that lie outside the exclusive regions will be considered potential destination elements for the halo operation. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleHalo() on any pair of FieldBundles that matches srcFieldBundle and dstFieldBundle in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

fieldbundle
ESMF_FieldBundle containing data to be haloed. The data in this FieldBundle may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.18 ESMF_FieldBundleIsCreated - Check whether a FieldBundle object has been created


INTERFACE:

   function ESMF_FieldBundleIsCreated(fieldbundle, rc)
RETURN VALUE:
     logical :: ESMF_FieldBundleIsCreated
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
DESCRIPTION:

Return .true. if the fieldbundle has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

fieldbundle
ESMF_FieldBundle queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.19 ESMF_FieldBundlePrint - Print FieldBundle information


INTERFACE:

   subroutine ESMF_FieldBundlePrint(fieldbundle, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Print internal information of the specified fieldbundle object.

The arguments are:

fieldbundle
ESMF_FieldBundle object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.20 ESMF_FieldBundleRead - Read Fields to a FieldBundle from file(s)


INTERFACE:

   subroutine ESMF_FieldBundleRead(fieldbundle, fileName, &
     singleFile, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
     character(*), intent(in) :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: singleFile
     integer, intent(in), optional :: timeslice
     type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
     integer, intent(out), optional :: rc
DESCRIPTION:

Read field data to a FieldBundle object from file(s). For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

Limitations:

The arguments are:

fieldbundle
An ESMF_FieldBundle object.
fileName
The name of the file from which FieldBundle data is read. If the FieldBundle contains multi-tile Arrays, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being read from a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be read from "myfile1.nc", tile 2 from "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[singleFile]
A logical flag, the default is .true., i.e., all Fields in the bundle are stored in one single file. If .false., each field is stored in separate files; these files are numbered with the name based on the argument "file". That is, a set of files are named: [file_name]001, [file_name]002, [file_name]003,...
[timeslice]
The time-slice number of the variable read from file.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.21 ESMF_FieldBundleRedist - Execute a FieldBundle redistribution


INTERFACE:

   subroutine ESMF_FieldBundleRedist(srcFieldBundle, dstFieldBundle, &
     routehandle, checkflag, rc)
ARGUMENTS:
         type(ESMF_FieldBundle), intent(in), optional :: srcFieldBundle
         type(ESMF_FieldBundle), intent(inout), optional :: dstFieldBundle
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: checkflag
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed redistribution from srcFieldBundle to dstFieldBundle. Both srcFieldBundle and dstFieldBundle must match the respective FieldBundles used during ESMF_FieldBundleRedistStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcFieldBundle and dstFieldBundle arguments are optional in support of the situation where srcFieldBundle and/or dstFieldBundle are not defined on all PETs. The srcFieldBundle and dstFieldBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

See ESMF_FieldBundleRedistStore() on how to precompute routehandle.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 25.2.9.

[srcFieldBundle]
ESMF_FieldBundle with source data.
[dstFieldBundle]
ESMF_FieldBundle with destination data.
routehandle
Handle to the precomputed Route.
[checkflag]
If set to .TRUE. the input FieldBundle pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.22 ESMF_FieldBundleRedistRelease - Release resources associated with a FieldBundle redistribution


INTERFACE:

   subroutine ESMF_FieldBundleRedistRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: noGarbage
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with a FieldBundle redistribution. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.23 ESMF_FieldBundleRedistStore - Precompute a FieldBundle redistribution with local factor argument


INTERFACE:

   ! Private name; call using ESMF_FieldBundleRedistStore()
   subroutine ESMF_FieldBundleRedistStore<type><kind>(srcFieldBundle, &
   dstFieldBundle, routehandle, factor, &
   ignoreUnmatchedIndicesFlag, srcToDstTransposeMap, rc)
ARGUMENTS:
   type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
   type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
   type(ESMF_RouteHandle), intent(inout) :: routehandle
   <type>(ESMF_KIND_<kind>), intent(in) :: factor
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   logical, intent(in), optional :: ignoreUnmatchedIndicesFlag(:)
   integer, intent(in), optional :: srcToDstTransposeMap(:)
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store a FieldBundle redistribution operation from srcFieldBundle to dstFieldBundle. PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.

Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. Redistribution corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.

Source and destination FieldBundles may be of different <type><kind>. Further source and destination FieldBundles may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleRedist() on any pair of FieldBundles that matches srcFieldBundle and dstFieldBundle in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 25.2.9.

The arguments are:

srcFieldBundle
ESMF_FieldBundle with source data.
dstFieldBundle
ESMF_FieldBundle with destination data. The data in this FieldBundle may be destroyed by this call.
routehandle
Handle to the precomputed Route.
factor
Factor by which to multiply source data.
[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover the identical index space, using precisely matching sequence indices. If set to .true., mismatching sequence indices between source and destination side are silently ignored. The size of this array argument must either be 1 or equal the number of Fields in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the handling of unmatched indices is specified for each Field pair separately. If only one element is specified, it is used for all Field pairs.
[srcToDstTransposeMap]
List with as many entries as there are dimensions in srcFieldBundle. Each entry maps the corresponding srcFieldBundle dimension against the specified dstFieldBundle dimension. Mixing of distributed and undistributed dimensions is supported.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.24 ESMF_FieldBundleRedistStore - Precompute a FieldBundle redistribution without local factor argument


INTERFACE:

   ! Private name; call using ESMF_FieldBundleRedistStore()
   subroutine ESMF_FieldBundleRedistStoreNF(srcFieldBundle, dstFieldBundle, &
     routehandle, ignoreUnmatchedIndicesFlag, &
     srcToDstTransposeMap, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
     type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
     type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: ignoreUnmatchedIndicesFlag(:)
     integer, intent(in), optional :: srcToDstTransposeMap(:)
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store a FieldBundle redistribution operation from srcFieldBundle to dstFieldBundle. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. Redistribution corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleRedist() on any pair of FieldBundles that matches srcFieldBundle and dstFieldBundle in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 25.2.9.

The arguments are:

srcFieldBundle
ESMF_FieldBundle with source data.
dstFieldBundle
ESMF_FieldBundle with destination data. The data in this FieldBundle may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover the identical index space, using precisely matching sequence indices. If set to .true., mismatching sequence indices between source and destination side are silently ignored. The size of this array argument must either be 1 or equal the number of Fields in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the handling of unmatched indices is specified for each Field pair separately. If only one element is specified, it is used for all Field pairs.
[srcToDstTransposeMap]
List with as many entries as there are dimensions in srcFieldBundle. Each entry maps the corresponding srcFieldBundle dimension against the specified dstFieldBundle dimension. Mixing of distributed and undistributed dimensions is supported.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.25 ESMF_FieldBundleRegrid - Execute a FieldBundle regrid operation


INTERFACE:

   subroutine ESMF_FieldBundleRegrid(srcFieldBundle, dstFieldBundle, &
          routehandle, zeroregion, termorderflag, checkflag, rc)
ARGUMENTS:
         type(ESMF_FieldBundle), intent(in), optional :: srcFieldBundle
         type(ESMF_FieldBundle), intent(inout), optional :: dstFieldBundle
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         type(ESMF_Region_Flag), intent(in), optional :: zeroregion
         type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag(:)
         logical, intent(in), optional :: checkflag
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed regrid from srcFieldBundle to dstFieldBundle. Both srcFieldBundle and dstFieldBundle must match the respective FieldBundles used during ESMF_FieldBundleRedistStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcFieldBundle and dstFieldBundle arguments are optional in support of the situation where srcFieldBundle and/or dstFieldBundle are not defined on all PETs. The srcFieldBundle and dstFieldBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

See ESMF_FieldBundleRegridStore() on how to precompute routehandle.

This call is collective across the current VM.

[srcFieldBundle]
ESMF_FieldBundle with source data.
[dstFieldBundle]
ESMF_FieldBundle with destination data.
routehandle
Handle to the precomputed Route.
[zeroregion]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in dstFieldBundle will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in dstFieldBundle will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination FieldBundle that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings.
[termorderflag]
Specifies the order of the source side terms in all of the destination sums. The termorderflag only affects the order of terms during the execution of the RouteHandle. See the 37.2.1 section for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods. See 54.60 for a full list of options. The size of this array argument must either be 1 or equal the number of Fields in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the term order for each Field Regrid operation is indicated separately. If only one term order element is specified, it is used for all Field pairs. The default is (/ESMF_TERMORDER_FREE/), allowing maximum flexibility in the order of terms for optimum performance.
[checkflag]
If set to .TRUE. the input FieldBundle pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.26 ESMF_FieldBundleRegridRelease - Release resources associated with a FieldBundle regrid operation


INTERFACE:

   subroutine ESMF_FieldBundleRegridRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: noGarbage
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with a FieldBundle regrid operation. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.27 ESMF_FieldBundleRegridStore - Precompute a FieldBundle regrid operation


INTERFACE:

   subroutine ESMF_FieldBundleRegridStore(srcFieldBundle, dstFieldBundle, &
        srcMaskValues, dstMaskValues, regridmethod, polemethod, regridPoleNPnts, &
        lineType, normType, extrapMethod, extrapNumSrcPnts, extrapDistExponent, &
        extrapNumLevels, unmappedaction, ignoreDegenerate, srcTermProcessing, &
        pipelineDepth, routehandle, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
     type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer(ESMF_KIND_I4), target, intent(in), optional :: srcMaskValues(:)
     integer(ESMF_KIND_I4), target, intent(in), optional :: dstMaskValues(:)
     type(ESMF_RegridMethod_Flag), intent(in), optional :: regridmethod
     type(ESMF_PoleMethod_Flag), intent(in), optional :: polemethod
     integer, intent(in), optional :: regridPoleNPnts
     type(ESMF_LineType_Flag), intent(in), optional :: lineType
     type(ESMF_NormType_Flag), intent(in), optional :: normType
     type(ESMF_ExtrapMethod_Flag), intent(in), optional :: extrapMethod
     integer, intent(in), optional :: extrapNumSrcPnts
     real, intent(in), optional :: extrapDistExponent
     integer, intent(in), optional :: extrapNumLevels
     type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction
     logical, intent(in), optional :: ignoreDegenerate
     integer, intent(inout), optional :: srcTermProcessing
     integer, intent(inout), optional :: pipelineDepth
     type(ESMF_RouteHandle), intent(inout), optional :: routehandle
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store a FieldBundle regrid operation over the data in srcFieldBundle and dstFieldBundle pair.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleRegrid() on any pair of FieldBundles that matches srcFieldBundle and dstFieldBundle in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcFieldbundle
Source ESMF_FieldBundle containing data to be regridded.
dstFieldbundle
Destination ESMF_FieldBundle. The data in this FieldBundle may be overwritten by this call.
[srcMaskValues]
Mask information can be set in the Grids (see 31.3.17) or Meshes (see 33.3.11) upon which the Fields in the srcFieldbundle are built. The srcMaskValues argument specifies the values in that mask information which indicate a source point should be masked out. In other words, a location is masked if and only if the value for that location in the mask information matches one of the values listed in srcMaskValues. If srcMaskValues is not specified, no masking will occur.
[dstMaskValues]
Mask information can be set in the Grids (see 31.3.17) or Meshes (see 33.3.11) upon which the Fields in the dstFieldbundle are built. The dstMaskValues argument specifies the values in that mask information which indicate a destination point should be masked out. In other words, a location is masked if and only if the value for that location in the mask information matches one of the values listed in dstMaskValues. If dstMaskValues is not specified, no masking will occur.
[regridmethod]
The type of interpolation. Please see Section 54.51 for a list of valid options. If not specified, defaults to ESMF_REGRIDMETHOD_BILINEAR.
[polemethod]
Which type of artificial pole to construct on the source Grid for regridding. Please see section 54.48 for a list of valid options. If not specified, defaults to ESMF_POLEMETHOD_ALLAVG.
[regridPoleNPnts]
If polemethod is ESMF_POLEMETHOD_NPNTAVG. This parameter indicates how many points should be averaged over. Must be specified if polemethod is ESMF_POLEMETHOD_NPNTAVG.
[lineType]
This argument controls the path of the line which connects two points on a sphere surface. This in turn controls the path along which distances are calculated and the shape of the edges that make up a cell. Both of these quantities can influence how interpolation weights are calculated. As would be expected, this argument is only applicable when srcField and dstField are built on grids which lie on the surface of a sphere. Section 54.36 shows a list of valid options for this argument. If not specified, the default depends on the regrid method. Section 54.36 has the defaults by line type. Figure 24.2.16 shows which line types are supported for each regrid method as well as showing the default line type by regrid method.
[normType]
This argument controls the type of normalization used when generating conservative weights. This option only applies to weights generated with regridmethod=ESMF_REGRIDMETHOD_CONSERVE. Please see Section 54.45 for a list of valid options. If not specified normType defaults to ESMF_NORMTYPE_DSTAREA.
[extrapMethod]
The type of extrapolation. Please see Section 54.18 for a list of valid options. If not specified, defaults to ESMF_EXTRAPMETHOD_NONE.
[extrapNumSrcPnts]
The number of source points to use for the extrapolation methods that use more than one source point (e.g. ESMF_EXTRAPMETHOD_NEAREST_IDAVG). If not specified, defaults to 8.
[extrapDistExponent]
The exponent to raise the distance to when calculating weights for the ESMF_EXTRAPMETHOD_NEAREST_IDAVG extrapolation method. A higher value reduces the influence of more distant points. If not specified, defaults to 2.0.
[extrapNumLevels]
The number of levels to output for the extrapolation methods that fill levels (e.g. ESMF_EXTRAPMETHOD_CREEP). When a method is used that requires this, then an error will be returned, if it is not specified.
[unmappedaction]
Specifies what should happen if there are destination points that can not be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR.
[ignoreDegenerate]
Ignore degenerate cells when checking the input Grids or Meshes for errors. If this is set to true, then the regridding proceeds, but degenerate cells will be skipped. If set to false, a degenerate cell produces an error. If not specified, ignoreDegenerate defaults to false.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldRegridStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldRegridStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[routehandle]
Handle to the precomputed Route.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.28 ESMF_FieldBundleRemove - Remove Fields from FieldBundle


INTERFACE:

   subroutine ESMF_FieldBundleRemove(fieldbundle, fieldNameList, &
     multiflag, relaxedflag, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
     character(len=*), intent(in) :: fieldNameList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: multiflag
     logical, intent(in), optional :: relaxedflag
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Remove field(s) by name from FieldBundle. In the relaxed setting it is not an error if fieldNameList contains names that are not found in fieldbundle.

fieldbundle
ESMF_FieldBundle from which to remove items.
fieldNameList
List of items to remove.
[multiflag]
A setting of .true. allows multiple Fields with the same name to be removed from fieldbundle. For .false., items to be removed must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "remove" where it is not an error if fieldNameList contains item names that are not found in fieldbundle. For .false. this is treated as an error condition. Further, in multiflag=.false. mode, the relaxed definition of "remove" also covers the case where there are multiple items in fieldbundle that match a single entry in fieldNameList. For relaxedflag=.false. this is treated as an error condition. The default setting is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.29 ESMF_FieldBundleReplace - Replace Fields in FieldBundle


INTERFACE:

   subroutine ESMF_FieldBundleReplace(fieldbundle, fieldList, &
     multiflag, relaxedflag, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(inout) :: fieldbundle
     type(ESMF_Field), intent(in) :: fieldList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: multiflag
     logical, intent(in), optional :: relaxedflag
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Replace field(s) by name in FieldBundle. In the relaxed setting it is not an error if fieldList contains Fields that do not match by name any item in fieldbundle. These Fields are simply ignored in this case.

fieldbundle
ESMF_FieldBundle in which to replace items.
fieldList
List of items to replace.
[multiflag]
A setting of .true. allows multiple items with the same name to be replaced in fieldbundle. For .false., items to be replaced must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "replace" where it is not an error if fieldList contains items with names that are not found in fieldbundle. These items in fieldList are ignored in the relaxed mode. For .false. this is treated as an error condition. Further, in multiflag=.false. mode, the relaxed definition of "replace" also covers the case where there are multiple items in fieldbundle that match a single entry by name in fieldList. For relaxedflag=.false. this is treated as an error condition. The default setting is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.30 ESMF_FieldBundleSet - Associate a Grid with an empty FieldBundle


INTERFACE:

       ! Private name; call using ESMF_FieldBundleSet()
       subroutine ESMF_FieldBundleSetGrid(fieldbundle, grid, rc)
ARGUMENTS:
       type(ESMF_FieldBundle), intent(inout) :: fieldbundle
       type(ESMF_Grid), intent(in) :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, intent(out), optional :: rc
DESCRIPTION:

Sets the grid for a fieldbundle.

The arguments are:

fieldbundle
An ESMF_FieldBundle object.
grid
The ESMF_Grid which all ESMF_Fields added to this ESMF_FieldBundle must have.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.31 ESMF_FieldBundleSet - Associate a Mesh with an empty FieldBundle


INTERFACE:

       ! Private name; call using ESMF_FieldBundleSet()
       subroutine ESMF_FieldBundleSetMesh(fieldbundle, mesh, rc)
ARGUMENTS:
       type(ESMF_FieldBundle), intent(inout) :: fieldbundle
       type(ESMF_Mesh), intent(in) :: mesh
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, intent(out), optional :: rc
DESCRIPTION:

Sets the mesh for a fieldbundle.

The arguments are:

fieldbundle
An ESMF_FieldBundle object.
mesh
The ESMF_Mesh which all ESMF_Fields added to this ESMF_FieldBundle must have.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.32 ESMF_FieldBundleSet - Associate a LocStream with an empty FieldBundle


INTERFACE:

       ! Private name; call using ESMF_FieldBundleSet()
       subroutine ESMF_FieldBundleSetLS(fieldbundle, locstream, &
         rc)
ARGUMENTS:
       type(ESMF_FieldBundle), intent(inout) :: fieldbundle
       type(ESMF_LocStream), intent(in) :: locstream
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, intent(out), optional :: rc
DESCRIPTION:

Sets the locstream for a fieldbundle.

The arguments are:

fieldbundle
An ESMF_FieldBundle object.
locstream
The ESMF_LocStream which all ESMF_Fields added to this ESMF_FieldBundle must have.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.33 ESMF_FieldBundleSet - Associate a XGrid with an empty FieldBundle


INTERFACE:

       ! Private name; call using ESMF_FieldBundleSet()
       subroutine ESMF_FieldBundleSetXGrid(fieldbundle, xgrid, &
         rc)
ARGUMENTS:
       type(ESMF_FieldBundle), intent(inout) :: fieldbundle
       type(ESMF_XGrid), intent(in) :: xgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, intent(out), optional :: rc
DESCRIPTION:

Sets the xgrid for a fieldbundle

The arguments are:

fieldbundle
An ESMF_FieldBundle object.
xgrid
The ESMF_XGrid which all ESMF_Fields added to this ESMF_FieldBundle must have.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.34 ESMF_FieldBundleSMM - Execute a FieldBundle sparse matrix multiplication


INTERFACE:

   subroutine ESMF_FieldBundleSMM(srcFieldBundle, dstFieldBundle, &
     routehandle, &
     zeroregion, & ! DEPRECATED ARGUMENT
     zeroregionflag, termorderflag, checkflag, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in), optional :: srcFieldBundle
     type(ESMF_FieldBundle), intent(inout), optional :: dstFieldBundle
     type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Region_Flag), intent(in), optional :: zeroregion ! DEPRECATED ARGUMENT
     type(ESMF_Region_Flag), intent(in), target, optional :: zeroregionflag(:)
     type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag(:)
     logical, intent(in), optional :: checkflag
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed sparse matrix multiplication from srcFieldBundle to dstFieldBundle. Both srcFieldBundle and dstFieldBundle must match the respective FieldBundles used during ESMF_FieldBundleRedistStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcFieldBundle and dstFieldBundle arguments are optional in support of the situation where srcFieldBundle and/or dstFieldBundle are not defined on all PETs. The srcFieldBundle and dstFieldBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

See ESMF_FieldBundleSMMStore() on how to precompute routehandle.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 25.2.11.

[srcFieldBundle]
ESMF_FieldBundle with source data.
[dstFieldBundle]
ESMF_FieldBundle with destination data.
routehandle
Handle to the precomputed Route.
[zeroregion]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in all Fields in dstFieldBundle will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in the Fields in dstFieldBundle will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination Fields that will be updated by the sparse matrix multiplication. See section 54.50
[zeroregionflag]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in the destination Field will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in the destination Field will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. A setting of ESMF_REGION_SELECT will only zero out those elements in the destination Field that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings. The size of this array argument must either be 1 or equal the number of Fields in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the zero region for each Field SMM operation is indicated separately. If only one zero region element is specified, it is used for all Field pairs.
[termorderflag]
Specifies the order of the source side terms in all of the destination sums. The termorderflag only affects the order of terms during the execution of the RouteHandle. See the 37.2.1 section for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods. See 54.60 for a full list of options. The size of this array argument must either be 1 or equal the number of Fields in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the term order for each Field SMM operation is indicated separately. If only one term order element is specified, it is used for all Field pairs. The default is (/ESMF_TERMORDER_FREE/), allowing maximum flexibility in the order of terms for optimum performance.
[checkflag]
If set to .TRUE. the input FieldBundle pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.35 ESMF_FieldBundleSMMRelease - Release resources associated with a FieldBundle sparse matrix multiplication


INTERFACE:

   subroutine ESMF_FieldBundleSMMRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: noGarbage
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with a FieldBundle sparse matrix multiplication. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.36 ESMF_FieldBundleSMMStore - Precompute a FieldBundle sparse matrix multiplication with local factors


INTERFACE:

   ! Private name; call using ESMF_FieldBundleSMMStore()
   subroutine ESMF_FieldBundleSMMStore<type><kind>(srcFieldBundle, &
   dstFieldBundle, routehandle, factorList, factorIndexList, &
   ignoreUnmatchedIndicesFlag, srcTermProcessing, rc)
ARGUMENTS:
   type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
   type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
   type(ESMF_RouteHandle), intent(inout) :: routehandle
   <type>(ESMF_KIND_<kind>), intent(in) :: factorList(:)
   integer, intent(in), :: factorIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   logical, intent(in), optional :: ignoreUnmatchedIndicesFlag(:)
   integer, intent(inout), optional :: srcTermProcessing(:)
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store a FieldBundle sparse matrix multiplication operation from srcFieldBundle to dstFieldBundle. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleSMM() on any pair of FieldBundles that matches srcFieldBundle and dstFieldBundle in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 25.2.11.

The arguments are:

srcFieldBundle
ESMF_FieldBundle with source data.
dstFieldBundle
ESMF_FieldBundle with destination data. The data in this FieldBundle may be destroyed by this call.
routehandle
Handle to the precomputed Route.
factorList
List of non-zero coefficients.
factorIndexList
Pairs of sequence indices for the factors stored in factorList.

The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.

In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcFieldBundle while factorIndexList(2,:) specifies the sequence index of the destination element in dstFieldBundle. For this format to be a valid option source and destination FieldBundles must have matching number of tensor elements (the product of the sizes of all Field tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.

The size 4 format is more general and does not require a matching tensor element count. Here the

factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcFieldBundle. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstFieldBundle.

See section 28.2.18 for details on the definition of sequence indices and tensor sequence indices.

[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover all of the squence indices defined in the sparse matrix. An error will be returned if a sequence index in the sparse matrix does not match on either the source or destination side. If set to .true., mismatching sequence indices are silently ignored. The size of this array argument must either be 1 or equal the number of Fieldss in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the handling of unmatched indices is specified for each Field pair separately. If only one element is specified, it is used for all Field pairs.

[srcTermProcessing]
Source term summing options for route handle creation. See ESMF_FieldRegridStore documentation for a full parameter description. Two forms may be provided. If a single element list is provided, this integer value is applied across all bundle members. Otherwise, the list must contain as many elements as there are bundle members. For the special case of accessing the auto-tuned parameter (providing a negative integer value), the list length must equal the bundle member count.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.37 ESMF_FieldBundleSMMStore - Precompute a FieldBundle sparse matrix multiplication


INTERFACE:

   ! Private name; call using ESMF_FieldBundleSMMStore()
     subroutine ESMF_FieldBundleSMMStoreNF(srcFieldBundle, dstFieldBundle, &
         routehandle, ignoreUnmatchedIndicesFlag, &
         srcTermProcessing, rc)
ARGUMENTS:
         type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
         type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
         type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical, intent(in), optional :: ignoreUnmatchedIndicesFlag(:)
         integer, intent(inout), optional :: srcTermProcessing(:)
         integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store a FieldBundle sparse matrix multiplication operation from srcFieldBundle to dstFieldBundle. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleSMM() on any pair of FieldBundles that matches srcFieldBundle and dstFieldBundle in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8, ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 25.2.11.

The arguments are:

srcFieldBundle
ESMF_FieldBundle with source data.
dstFieldBundle
ESMF_FieldBundle with destination data. The data in this FieldBundle may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover all of the squence indices defined in the sparse matrix. An error will be returned if a sequence index in the sparse matrix does not match on either the source or destination side. If set to .true., mismatching sequence indices are silently ignored. The size of this array argument must either be 1 or equal the number of Fieldss in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the handling of unmatched indices is specified for each Field pair separately. If only one element is specified, it is used for all Field pairs.
[srcTermProcessing]
Source term summing options for route handle creation. See ESMF_FieldRegridStore documentation for a full parameter description. Two forms may be provided. If a single element list is provided, this integer value is applied across all bundle members. Otherwise, the list must contain as many elements as there are bundle members. For the special case of accessing the auto-tuned parameter (providing a negative integer value), the list length must equal the bundle member count.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.38 ESMF_FieldBundleSMMStore - Precompute field bundle sparse matrix multiplication using factors read from file


INTERFACE:

   ! Private name; call using ESMF_FieldBundleSMMStore()
     subroutine ESMF_FieldBundleSMMStoreFromFile(srcFieldBundle, dstFieldBundle, &
       filename, routehandle, ignoreUnmatchedIndicesFlag, &
       srcTermProcessing, rc)
   ! ARGUMENTS:
       type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
       type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
       character(len=*), intent(in) :: filename
       type(ESMF_RouteHandle), intent(inout) :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical, intent(in), optional :: ignoreUnmatchedIndicesFlag(:)
       integer, intent(inout), optional :: srcTermProcessing(:)
       integer, intent(out), optional :: rc
DESCRIPTION:

Compute an ESMF_RouteHandle using factors read from file.

The arguments are:

srcFieldBundle
ESMF_FieldBundle with source data.

dstFieldBundle
ESMF_FieldBundle with destination data. The data in this field bundle may be destroyed by this call.

filename
Path to the file containing weights for creating an ESMF_RouteHandle. See (12.9) for a description of the SCRIP weight file format. Only "row", "col", and "S" variables are required. They must be one-dimensionsal with dimension "n_s".

routehandle
Handle to the ESMF_RouteHandle.

[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover all of the squence indices defined in the sparse matrix. An error will be returned if a sequence index in the sparse matrix does not match on either the source or destination side. If set to .true., mismatching sequence indices are silently ignored. The size of this array argument must either be 1 or equal the number of Fieldss in the srcFieldBundle and dstFieldBundle arguments. In the latter case, the handling of unmatched indices is specified for each Field pair separately. If only one element is specified, it is used for all Field pairs.

[srcTermProcessing]
Source term summing options for route handle creation. See ESMF_FieldRegridStore documentation for a full parameter description. Two forms may be provided. If a single element list is provided, this integer value is applied across all bundle members. Otherwise, the list must contain as many elements as there are bundle members. For the special case of accessing the auto-tuned parameter (providing a negative integer value), the list length must equal the bundle member count.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.39 ESMF_FieldBundleValidate - Validate fieldbundle internals


INTERFACE:

   subroutine ESMF_FieldBundleValidate(fieldbundle, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
     integer, intent(out), optional :: rc
DESCRIPTION:

Validates that the fieldbundle is internally consistent. The method returns an error code if problems are found.

The arguments are:

fieldbundle
Specified ESMF_FieldBundle object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

25.5.40 ESMF_FieldBundleWrite - Write the Fields into a file


INTERFACE:

   subroutine ESMF_FieldBundleWrite(fieldbundle, fileName, &
       convention, purpose, singleFile, overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_FieldBundle), intent(in) :: fieldbundle
     character(*), intent(in) :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(*), intent(in), optional :: convention
     character(*), intent(in), optional :: purpose
     logical, intent(in), optional :: singleFile
     logical , intent(in), optional :: overwrite
     type(ESMF_FileStatus_Flag), intent(in), optional :: status
     integer, intent(in), optional :: timeslice
     type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
     integer, intent(out), optional :: rc
DESCRIPTION:

Write the Fields into a file. For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

When convention and purpose arguments are specified, NetCDF dimension labels and variable attributes are written from each Field in the FieldBundle from the corresponding Attribute package. Additionally, Attributes may be set on the FieldBundle level under the same Attribute package. This allows the specification of global attributes within the file. As with individual Fields, the value associated with each name may be either a scalar character string, or a scalar or array of type integer, real, or double precision.

Limitations:

The arguments are:

fieldbundle
An ESMF_FieldBundle object.
fileName
The name of the output file to which field bundle data is written. If the FieldBundle contains multi-tile Arrays, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being written to a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be written to "myfile1.nc", tile 2 to "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[convention]
Specifies an Attribute package associated with the FieldBundle, and the contained Fields, used to create NetCDF dimension labels and attributes in the file. When this argument is present, the purpose argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[purpose]
Specifies an Attribute package associated with the FieldBundle, and the contained Fields, used to create NetCDF dimension labels and attributes in the file. When this argument is present, the convention argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[singleFile]
A logical flag, the default is .true., i.e., all fields in the bundle are written in one single file. If .false., each field will be written in separate files; these files are numbered with the name based on the argument "file". That is, a set of files are named: [file_name]001, [file_name]002, [file_name]003,...
[overwrite]
A logical flag, the default is .false., i.e., existing field data may not be overwritten. If .true., only the data corresponding to the fields name will be be overwritten. If the timeslice option is given, only data for the given timeslice will be overwritten. Note that it is always an error to attempt to overwrite a NetCDF variable with data which has a different shape.
[status]
The file status. Please see Section 54.22 for the list of options. If not present, defaults to ESMF_FILESTATUS_UNKNOWN.
[timeslice]
Some I/O formats (e.g. NetCDF) support the output of data in form of time slices. The timeslice argument provides access to this capability. timeslice must be positive. The behavior of this option may depend on the setting of the overwrite flag:
overwrite = .false.:
If the timeslice value is less than the maximum time already in the file, the write will fail.
overwrite = .true.:
Any positive timeslice value is valid.
By default, i.e. by omitting the timeslice argument, no provisions for time slicing are made in the output file, however, if the file already contains a time axis for the variable, a timeslice one greater than the maximum will be written.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26 Field Class

26.1 Description

An ESMF Field represents a physical field, such as temperature. The motivation for including Fields in ESMF is that bundles of Fields are the entities that are normally exchanged when coupling Components.

The ESMF Field class contains distributed and discretized field data, a reference to its associated grid, and metadata. The Field class stores the grid staggering for that physical field. This is the relationship of how the data array of a field maps onto a grid (e.g. one item per cell located at the cell center, one item per cell located at the NW corner, one item per cell vertex, etc.). This means that different Fields which are on the same underlying ESMF Grid but have different staggerings can share the same Grid object without needing to replicate it multiple times.

Fields can be added to States for use in inter-Component data communications. Fields can also be added to FieldBundles, which are groups of Fields on the same underlying Grid. One motivation for packing Fields into FieldBundles is convenience; another is the ability to perform optimized collective data transfers.

Field communication capabilities include: data redistribution, regridding, scatter, gather, sparse-matrix multiplication, and halo update. These are discussed in more detail in the documentation for the specific method calls. ESMF does not currently support vector fields, so the components of a vector field must be stored as separate Field objects.

26.1.1 Operations

The Field class allows the user to easily perform a number of operations on the data stored in a Field. This section gives a brief summary of the different types of operations and the range of their capabilities. The operations covered here are: redistribution (ESMF_FieldRedistStore()), sparse matrix multiply (ESMF_FieldSMMStore()), and regridding (ESMF_FieldRegridStore()).

The redistribution operation (ESMF_FieldRedistStore()) allows the user to move data between two Fields with the same size, but different distribution. This operation is useful, for example, to move data between two components with different distributions. Please see Section 26.3.30 for an example of the redistribution capability.

The sparse matrix multiplication operation (ESMF_FieldSMMStore()) allows the user to multiply the data in a Field by a sparse matrix. This operation is useful, for example, if the user has an interpolation matrix and wants to apply it to the data in a Field. Please see Section 26.3.33 for an example of the sparse matrix multiply capability.

The regridding operation (ESMF_FieldRegridStore()) allows the user to move data from one grid to another while maintaining certain properties of the data. Regridding is also called interpolation or remapping. In the Field regridding operation the grids the data is being moved between are the grids associated with the Fields storing the data. The regridding operation works on Fields built on Meshes, Grids, or Location Streams. There are six regridding methods available: bilinear, higher-order patch, two types of nearest neighbor, first-order conservative, and second-order conservative. Please see section 24.2 for a more indepth description of regridding including in which situations each method is supported. Please see section 26.3.25 for a description of the regridding capability as it applies to Fields. Several sections following section 26.3.25 contain examples of using regridding.

26.2 Constants


26.2.1 ESMF_FIELDSTATUS

DESCRIPTION:
An ESMF_Field can be in different status after initialization. Field status can be queried using ESMF_FieldGet() method.

The type of this flag is:

type(ESMF_FieldStatus_Flag)

The valid values are:

ESMF_FIELDSTATUS_EMPTY
Field is empty without geombase or data storage. Such a Field can be added to a ESMF_State and participate ESMF_StateReconcile().
ESMF_FIELDSTATUS_GRIDSET
Field is partially created. It has a geombase object internally created and the geombase object associates with either a ESMF_Grid, or a ESMF_Mesh, or an ESMF_XGrid, or a ESMF_LocStream. It's an error to set another geombase object in such a Field. It can also be added to a ESMF_State and participate ESMF_StateReconcile().
ESMF_FIELDSTATUS_COMPLETE
Field is completely created with geombase and data storage internally allocated.

26.3 Use and Examples

A Field serves as an annotator of data, since it carries a description of the grid it is associated with and metadata such as name and units. Fields can be used in this capacity alone, as convenient, descriptive containers into which arrays can be placed and retrieved. However, for most codes the primary use of Fields is in the context of import and export States, which are the objects that carry coupling information between Components. Fields enable data to be self-describing, and a State holding ESMF Fields contains data in a standard format that can be queried and manipulated.

The sections below go into more detail about Field usage.

26.3.1 Field create and destroy

Fields can be created and destroyed at any time during application execution. However, these Field methods require some time to complete. We do not recommend that the user create or destroy Fields inside performance-critical computational loops.

All versions of the ESMF_FieldCreate() routines require a Grid object as input, or require a Grid be added before most operations involving Fields can be performed. The Grid contains the information needed to know which Decomposition Elements (DEs) are participating in the processing of this Field, and which subsets of the data are local to a particular DE.

The details of how the create process happens depend on which of the variants of the ESMF_FieldCreate() call is used. Some of the variants are discussed below.

There are versions of the ESMF_FieldCreate() interface which create the Field based on the input Grid. The ESMF can allocate the proper amount of space but not assign initial values. The user code can then get the pointer to the uninitialized buffer and set the initial data values.

Other versions of the ESMF_FieldCreate() interface allow user code to attach arrays that have already been allocated by the user. Empty Fields can also be created in which case the data can be added at some later time.

For versions of Create which do not specify data values, user code can create an ArraySpec object, which contains information about the typekind and rank of the data values in the array. Then at Field create time, the appropriate amount of memory is allocated to contain the data which is local to each DE.

When finished with a ESMF_Field, the ESMF_FieldDestroy method removes it. However, the objects inside the ESMF_Field created externally should be destroyed separately, since objects can be added to more than one ESMF_Field. For example, the same ESMF_Grid can be referenced by multiple ESMF_Fields. In this case the internal Grid is not deleted by the ESMF_FieldDestroy call.


26.3.2 Get Fortran data pointer, bounds, and counts information from a Field

A user can get bounds and counts information from an ESMF_Field through the ESMF_FieldGet() interface. Also available through this interface is the intrinsic Fortran data pointer contained in the internal ESMF_Array object of an ESMF_Field. The bounds and counts information are DE specific for the associated Fortran data pointer.

For a better discussion of the terminologies, bounds and widths in ESMF e.g. exclusive, computational, total bounds for the lower and upper corner of data region, etc.., user can refer to the explanation of these concepts for Grid and Array in their respective sections in the Reference Manual, e.g. Section 28.2.6 on Array and Section 31.3.19 on Grid.

In this example, we first create a 3D Field based on a 3D Grid and Array. Then we use the ESMF_FieldGet() interface to retrieve the data pointer, potentially updating or verifying its values. We also retrieve the bounds and counts information of the 3D Field to assist in data element iteration.

    xdim = 180
    ydim = 90
    zdim = 50

    ! create a 3D data Field from a Grid and Array.
    ! first create a Grid
    grid3d = ESMF_GridCreateNoPeriDim(minIndex=(/1,1,1/), &
            maxIndex=(/xdim,ydim,zdim/), &
            regDecomp=(/2,2,1/), name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_GridGet(grid=grid3d, staggerloc=ESMF_STAGGERLOC_CENTER, &
           distgrid=distgrid3d, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_GridGetFieldBounds(grid=grid3d, localDe=0, &
        staggerloc=ESMF_STAGGERLOC_CENTER, totalCount=fa_shape, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    allocate(farray(fa_shape(1), fa_shape(2), fa_shape(3)) )

    ! create an Array
    array3d = ESMF_ArrayCreate(distgrid3d, farray, &
        indexflag=ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create a Field
    field = ESMF_FieldCreate(grid=grid3d, array=array3d, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! retrieve the Fortran data pointer from the Field
    call ESMF_FieldGet(field=field, localDe=0, farrayPtr=farray1, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! retrieve the Fortran data pointer from the Field and bounds
    call ESMF_FieldGet(field=field, localDe=0, farrayPtr=farray1, &
        computationalLBound=compLBnd, computationalUBound=compUBnd, &
        exclusiveLBound=exclLBnd, exclusiveUBound=exclUBnd, &
        totalLBound=totalLBnd, totalUBound=totalUBnd, &
        computationalCount=comp_count, &
        exclusiveCount=excl_count, &
        totalCount=total_count, &
        rc=rc)


    ! iterate through the total bounds of the field data pointer
    do k = totalLBnd(3), totalUBnd(3)
        do j = totalLBnd(2), totalUBnd(2)
            do i = totalLBnd(1), totalUBnd(1)
                farray1(i, j, k) = sin(2*i/total_count(1)*PI) + &
                    sin(4*j/total_count(2)*PI) + &
                    sin(8*k/total_count(2)*PI)
            enddo
        enddo
    enddo


26.3.3 Get Grid, Array, and other information from a Field

A user can get the internal ESMF_Grid and ESMF_Array from a ESMF_Field. Note that the user should not issue any destroy command on the retrieved grid or array object since they are referenced from within the ESMF_Field. The retrieved objects should be used in a read-only fashion to query additional information not directly available through the ESMF_FieldGet() interface.

    call ESMF_FieldGet(field, grid=grid, array=array, &
        typekind=typekind, dimCount=dimCount, staggerloc=staggerloc, &
        gridToFieldMap=gridToFieldMap, &
        ungriddedLBound=ungriddedLBound, ungriddedUBound=ungriddedUBound, &
        totalLWidth=totalLWidth, totalUWidth=totalUWidth, &
        name=name, &
        rc=rc)


26.3.4 Create a Field with a Grid, typekind, and rank

A user can create an ESMF_Field from an ESMF_Grid and typekind/rank. This create method associates the two objects.

We first create a Grid with a regular distribution that is 10x20 index in 2x2 DEs. This version of Field create simply associates the data with the Grid. The data is referenced explicitly on a regular 2x2 uniform grid. Finally we create a Field from the Grid, typekind, rank, and a user specified StaggerLoc.

This example also illustrates a typical use of this Field creation method. By creating a Field from a Grid and typekind/rank, the user allows the ESMF library to create a internal Array in the Field. Then the user can use ESMF_FieldGet() to retrieve the Fortran data array and necessary bounds information to assign initial values to it.

    ! create a grid
    grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
          regDecomp=(/2,2/), name="atmgrid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create a Field from the Grid and arrayspec
    field1 = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_R4, &
        indexflag=ESMF_INDEX_DELOCAL, &
        staggerloc=ESMF_STAGGERLOC_CENTER, name="pressure", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_FieldGet(field1, localDe=0, farrayPtr=farray2dd, &
        totalLBound=ftlb, totalUBound=ftub, totalCount=ftc, rc=rc)

    do i = ftlb(1), ftub(1)
        do j = ftlb(2), ftub(2)
            farray2dd(i, j) = sin(i/ftc(1)*PI) * cos(j/ftc(2)*PI)
        enddo
    enddo

    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.5 Create a Field with a Grid and Arrayspec

A user can create an ESMF_Field from an ESMF_Grid and a ESMF_Arrayspec with corresponding rank and type. This create method associates the two objects.

We first create a Grid with a regular distribution that is 10x20 index in 2x2 DEs. This version of Field create simply associates the data with the Grid. The data is referenced explicitly on a regular 2x2 uniform grid. Then we create an ArraySpec. Finally we create a Field from the Grid, ArraySpec, and a user specified StaggerLoc.

This example also illustrates a typical use of this Field creation method. By creating a Field from a Grid and an ArraySpec, the user allows the ESMF library to create a internal Array in the Field. Then the user can use ESMF_FieldGet() to retrieve the Fortran data array and necessary bounds information to assign initial values to it.

    ! create a grid
    grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
          regDecomp=(/2,2/), name="atmgrid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! setup arrayspec
    call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create a Field from the Grid and arrayspec
    field1 = ESMF_FieldCreate(grid, arrayspec, &
         indexflag=ESMF_INDEX_DELOCAL, &
         staggerloc=ESMF_STAGGERLOC_CENTER, name="pressure", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_FieldGet(field1, localDe=0, farrayPtr=farray2dd, &
        totalLBound=ftlb, totalUBound=ftub, totalCount=ftc, rc=rc)

    do i = ftlb(1), ftub(1)
        do j = ftlb(2), ftub(2)
            farray2dd(i, j) = sin(i/ftc(1)*PI) * cos(j/ftc(2)*PI)
        enddo
    enddo

    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

A user can also create an ArraySpec that has a different rank from the Grid, For example, the following code shows creation of of 3D Field from a 2D Grid using a 3D ArraySpec.

This example also demonstrates the technique to create a typical 3D data Field that has 2 gridded dimensions and 1 ungridded dimension.

First we create a 2D grid with an index space of 180x360 equivalent to 180x360 Grid cells (note that for a distributed memory computer, this means each grid cell will be on a separate PE!). In the FieldCreate call, we use gridToFieldMap to indicate the mapping between Grid dimension and Field dimension. For the ungridded dimension (typically the altitude), we use ungriddedLBound and ungriddedUBound to describe its bounds. Internally the ungridded dimension has a stride of 1, so the number of elements of the ungridded dimension is ungriddedUBound - ungriddedLBound + 1.

Note that gridToFieldMap in this specific example is (/1,2/) which is the default value so the user can neglect this argument for the FieldCreate call.

    grid2d = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), &
          maxIndex=(/180,360/), regDecomp=(/2,2/), name="atmgrid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_ArraySpecSet(arrayspec, 3, ESMF_TYPEKIND_R4, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    field1 = ESMF_FieldCreate(grid2d, arrayspec, &
         indexflag=ESMF_INDEX_DELOCAL, &
         staggerloc=ESMF_STAGGERLOC_CENTER, &
         gridToFieldMap=(/1,2/), &
         ungriddedLBound=(/1/), ungriddedUBound=(/50/), &
         name="pressure", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.6 Create a Field with a Grid and Array

A user can create an ESMF_Field from an ESMF_Grid and a ESMF_Array. The Grid was created in the previous example.

This example creates a 2D ESMF_Field from a 2D ESMF_Grid and a 2D ESMF_Array.

    ! Get necessary information from the Grid
    call ESMF_GridGet(grid, staggerloc=ESMF_STAGGERLOC_CENTER, &
        distgrid=distgrid, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Create a 2D ESMF_TYPEKIND_R4 arrayspec
    call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Create a ESMF_Array from the arrayspec and distgrid
    array2d = ESMF_ArrayCreate(arrayspec=arrayspec, &
            distgrid=distgrid, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Create a ESMF_Field from the grid and array
    field4 = ESMF_FieldCreate(grid, array2d, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.7 Create an empty Field and complete it with FieldEmptySet and FieldEmptyComplete

A user can create an ESMF_Field in three steps: first create an empty ESMF_Field; then set a ESMF_Grid on the empty ESMF_Field; and finally complete the ESMF_Field by calling ESMF_FieldEmptyComplete.

    ! create an empty Field
    field3 = ESMF_FieldEmptyCreate(name="precip", rc=rc)

    ! use FieldGet to retrieve the Field Status
    call ESMF_FieldGet(field3, status=fstatus, rc=rc)

Once the Field is created, we can verify that the status of the Field is ESMF_FIELDSTATUS_EMPTY.

    ! Test the status of the Field
    if (fstatus /= ESMF_FIELDSTATUS_EMPTY) then
         call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif

Next we set a Grid on the empty Field. We use the 2D grid created in a previous example simply to demonstrate the method. The Field data points will be on east edge of the Grid cells with the specified ESMF_STAGGERLOC_EDGE1.

    ! Set a grid on the Field
    call ESMF_FieldEmptySet(field3, grid2d, &
             staggerloc=ESMF_STAGGERLOC_EDGE1, rc=rc)

    ! use FieldGet to retrieve the Field Status again
    call ESMF_FieldGet(field3, status=fstatus, rc=rc)

    ! Test the status of the Field
    if (fstatus /= ESMF_FIELDSTATUS_GRIDSET) then
         call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif

The partially created Field is completed by specifying the typekind of its data storage. This method is overloaded with one of the following parameters, arrayspec, typekind, Fortran array, or Fortran array pointer. Additional optional arguments can be used to specify ungridded dimensions and halo regions similar to the other Field creation methods.

    ! Complete the Field by specifying the data typekind
    ! to be allocated internally.
    call ESMF_FieldEmptyComplete(field3, typekind=ESMF_TYPEKIND_R8, &
      ungriddedLBound=(/1/), ungriddedUBound=(/5/), rc=rc)

    ! use FieldGet to retrieve the Field Status again
    call ESMF_FieldGet(field3, status=fstatus, rc=rc)

    ! Test the status of the Field
    if (fstatus /= ESMF_FIELDSTATUS_COMPLETE) then
         call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif


26.3.8 Create an empty Field and complete it with FieldEmptyComplete

A user can create an empty ESMF_Field. Then the user can finalize the empty ESMF_Field from a ESMF_Grid and an intrinsic Fortran data array. This interface is overloaded for typekind and rank of the Fortran data array.

In this example, both the grid and the Fortran array pointer are 2 dimensional and each dimension of the grid is mapped to the corresponding dimension of the Fortran array pointer, i.e. 1st dimension of grid maps to 1st dimension of Fortran array pointer, 2nd dimension of grid maps to 2nd dimension of Fortran array pointer, so on and so forth.

In order to create or complete a Field from a Grid and a Fortran array pointer, certain rules of the Fortran array bounds must be obeyed. We will discuss these rules as we progress in Field creation examples. We will make frequent reference to the terminologies for bounds and widths in ESMF. For a better discussion of these terminologies and concepts behind them, e.g. exclusive, computational, total bounds for the lower and upper corner of data region, etc.., users can refer to the explanation of these concepts for Grid and Array in their respective sections in the Reference Manual, e.g. Section 28.2.6 on Array and Section 31.3.19 on Grid. The examples here are designed to help a user to get up to speed with creating Fields for typical use.

This example introduces a helper method, the ESMF_GridGetFieldBounds interface that facilitates the computation of Fortran data array bounds and shape to assist ESMF_FieldEmptyComplete finalizing a Field from an intrinsic Fortran data array and a Grid.

    ! create an empty Field
    field3 = ESMF_FieldEmptyCreate(name="precip", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! use FieldGet to retrieve total counts
    call ESMF_GridGetFieldBounds(grid2d, localDe=0, &
        staggerloc=ESMF_STAGGERLOC_CENTER, totalCount=ftc, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! allocate the 2d Fortran array based on retrieved total counts
    allocate(farray2d(ftc(1), ftc(2)))

    ! finalize the Field
    call ESMF_FieldEmptyComplete(field3, grid2d, farray2d, rc=rc)


26.3.9 Create a 7D Field with a 5D Grid and 2D ungridded bounds from a Fortran data array

In this example, we will show how to create a 7D Field from a 5D ESMF_Grid and 2D ungridded bounds with arbitrary halo widths and gridToFieldMap.

We first create a 5D DistGrid and a 5D Grid based on the DistGrid; then ESMF_GridGetFieldBounds computes the shape of a 7D array in fsize. We can then create a 7D Field from the 5D Grid and the 7D Fortran data array with other assimilating parameters.

    ! create a 5d distgrid
    distgrid5d = ESMF_DistGridCreate(minIndex=(/1,1,1,1,1/), &
        maxIndex=(/10,4,10,4,6/), regDecomp=(/2,1,2,1,1/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Create a 5d Grid
    grid5d = ESMF_GridCreate(distgrid=distgrid5d, name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! use FieldGet to retrieve total counts
    call ESMF_GridGetFieldBounds(grid5d, localDe=0, ungriddedLBound=(/1,2/), &
        ungriddedUBound=(/4,5/), &
        totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
        gridToFieldMap=(/3,2,5,4,1/), &
        totalCount=fsize, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! allocate the 7d Fortran array based on retrieved total counts
    allocate(farray7d(fsize(1), fsize(2), fsize(3), fsize(4), fsize(5), &
                        fsize(6), fsize(7)))

    ! create the Field
    field7d = ESMF_FieldCreate(grid5d, farray7d, ESMF_INDEX_DELOCAL, &
        ungriddedLBound=(/1,2/), ungriddedUBound=(/4,5/), &
        totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
        gridToFieldMap=(/3,2,5,4,1/), &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

A user can allocate the Fortran array in a different manner using the lower and upper bounds returned from FieldGet through the optional totalLBound and totalUBound arguments. In the following example, we create another 7D Field by retrieving the bounds and allocate the Fortran array with this approach. In this scheme, indexing the Fortran array is sometimes more convenient than using the shape directly.

    call ESMF_GridGetFieldBounds(grid5d, localDe=0, ungriddedLBound=(/1,2/), &
        ungriddedUBound=(/4,5/), &
        totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
        gridToFieldMap=(/3,2,5,4,1/), &
        totalLBound=flbound, totalUBound=fubound, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    allocate(farray7d2(flbound(1):fubound(1), flbound(2):fubound(2), &
                       flbound(3):fubound(3), flbound(4):fubound(4), &
                       flbound(5):fubound(5), flbound(6):fubound(6), &
                       flbound(7):fubound(7)) )

    field7d2 = ESMF_FieldCreate(grid5d, farray7d2, ESMF_INDEX_DELOCAL, &
        ungriddedLBound=(/1,2/), ungriddedUBound=(/4,5/), &
        totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
        gridToFieldMap=(/3,2,5,4,1/), &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.10 Shared memory features: DE pinning, sharing, and migration

See 28.2.13 for a introduction of the DE pinning feature. Here we focus on demonstrating the use of the DE pinning feature in the context of ESMF Field.

When an ESMF Field object is created, the specified underlying DistGrid indicates how many Decomposition Elements (DEs) are created. Each DE has its own memory allocation to hold user data. The DELayout, referenced by the DistGrid, determines which PET is considered the owner of each of the DEs. Queried for the local DEs, the Field object returns the list of DEs that are owned by the local PET making the query.

By default DEs are pinned to the PETs under which they were created. The memory allocation associated with a specific DE is only defined in the VAS of the PET to which the DE is pinned. As a consequence, only the PET owning a DE has access to its memory allocation.

On shared memory systems, however, ESMF allows DEs to be pinned to SSIs instead of PETs. In this case the PET under which a DE was created is still consider the owner, but now all PETs under the same SSI have access to the DE. For this the memory allocation associated with the DE is mapped into the VAS of all the PETs under the SSI.

To create an Field with each DE pinned to SSI instead of PET, first query the VM for the available level of support.

  call ESMF_VMGet(vm, ssiSharedMemoryEnabledFlag=ssiSharedMemoryEnabled, rc=rc)

  if (ssiSharedMemoryEnabled) then

Knowing that the SSI shared memory feature is available, it is now possible to create an Field object with DE to SSI pinning.

    grid = ESMF_GridCreateNoPeriDim(maxIndex=(/40,10/), regDecomp=(/4,1/), &
      coordSys = ESMF_COORDSYS_CART, &
      rc=rc)

    field = ESMF_FieldCreate(typekind=ESMF_TYPEKIND_R8, grid=grid, &
      pinflag=ESMF_PIN_DE_TO_SSI, rc=rc)

Just as in the cases discussed before, where the same Grid was used, a default DELayout with as many DEs as PETs in the VM is constructed. Setting the pinflag to ESMF_PIN_DE_TO_SSI does not change the fact that each PET owns exactly one of the DEs. However, assuming that this code is run on a set of PETs that are all located under the same SSI, every PET now has access to all of the DEs. The situation can be observed by querying for both the localDeCount, and the ssiLocalDeCount.

    call ESMF_FieldGet(field, localDeCount=localDeCount, &
      ssiLocalDeCount=ssiLocalDeCount, rc=rc)

Assuming execution on 4 PETs, all located on the same SSI, the values of the returned variable are localDeCount==1 and ssiLocalDeCount==4 on all of the PETs. The mapping between each PET's local DE, and the global DE index is provided through the localDeToDeMap array argument. The amount of mapping information returned is dependent on how large localDeToDeMap has been sized by the user. For size(localDeToDeMap)==localDeCount, only mapping information for those DEs owned by the local PET is filled in. However for size(localDeToDeMap)==ssiLocalDeCount, mapping information for all locally accessible DEs is returned, including those owned by other PETs on the same SSI.

    allocate(localDeToDeMap(0:ssiLocalDeCount-1))
    call ESMF_FieldGet(field, localDeToDeMap=localDeToDeMap, rc=rc)

The first localDeCount entries of localDeToDeMap are always the global DE indices of the DEs owned by the local PET. The remaining ssiLocalDeCount-localDeCount entries are the global DE indices of DEs shared by other PETs. The ordering of the shared DEs is from smallest to greatest, excluding the locally owned DEs, which were already listed at the beginning of localDeToDeMap. For the current case, again assuming execution on 4 PETs all located on the same SSI, we expect the following situation:

PET 0: localDeToDeMap==(/0,1,2,3/)
PET 1: localDeToDeMap==(/1,0,2,3/)
PET 2: localDeToDeMap==(/2,0,1,3/)
PET 3: localDeToDeMap==(/3,0,1,2/)

Each PET can access the memory allocations associated with all of the DEs listed in the localDeToDeMap returned by the Field object. Direct access to the Fortran array pointer of a specific memory allocation is available through ESMF_FieldGet(). Here each PET queries for the farrayPtr of localDe==2, i.e. the 2nd shared DE.

    call ESMF_FieldGet(field, farrayPtr=myFarray, localDe=2, rc=rc)

Now variable myFarray on PETs 0 and 1 both point to the same memory allocation for global DE 2. Both PETs have access to the same piece of shared memory! The same is true for PETs 2 and 3, pointing to the shared memory allocation of global DE 1.

It is important to note that all of the typical considerations surrounding shared memory programming apply when accessing shared DEs! Proper synchronization between PETs accessing shared DEs is critical to avoid race conditions. Also performance issues like false sharing need to be considered for optimal use.

For a simple demonstration, PETs 0 and 2 fill the entire memory allocation of DE 2 and 1, respectively, to a unique value.

    if (localPet==0) then
      myFarray = 12345.6789d0
    else if (localPet==2) then
      myFarray = 6789.12345d0
    endif

Here synchronization is needed before any PETs that share access to the same DEs can safely access the data without race condition. The Field class provides a simple synchronization method that can be used.

    call ESMF_FieldSync(field, rc=rc) ! prevent race condition

Now it is safe for PETs 1 and 3 to access the shared DEs. We expect to find the data that was set above. For simplicity of the code only the first array element is inspected here.

    if (localPet==1) then
      if (abs(myFarray(1,1)-12345.6789d0)>1.d10) print *, "bad data detected"
    else if (localPet==3) then
      if (abs(myFarray(1,1)-6789.12345d0)>1.d10) print *, "bad data detected"
    endif

  endif ! ending the ssiSharedMemoryEnabled conditional


26.3.11 Create a 2D Field with a 2D Grid and a Fortran data array

A user can create an ESMF_Field directly from an ESMF_Grid and an intrinsic Fortran data array. This interface is overloaded for typekind and rank of the Fortran data array.

In the following example, each dimension size of the Fortran array is equal to the exclusive bounds of its corresponding Grid dimension queried from the Grid through ESMF_GridGet() public interface.

Formally let fa_shape(i) be the shape of i-th dimension of user supplied Fortran array, then rule 1 states:

   
    (1) fa_shape(i) = exclusiveCount(i)         
                  i = 1...GridDimCount

fa_shape(i) defines the shape of i-th dimension of the Fortran array. ExclusiveCount are the number of data elements of i-th dimension in the exclusive region queried from ESMF_GridGet interface. Rule 1 assumes that the Grid and the Fortran intrinsic array have same number of dimensions; and optional arguments of FieldCreate from Fortran array are left unspecified using default setup. These assumptions are true for most typical uses of FieldCreate from Fortran data array. This is the easiest way to create a Field from a Grid and a Fortran intrinsic data array.

Fortran array dimension sizes (called shape in most Fortran language books) are equivalent to the bounds and counts used in this manual. The following equation holds:

   
    fa_shape(i) = shape(i) = counts(i) = upper_bound(i) - lower_bound(i) + 1

These typically mean the same concept unless specifically explained to mean something else. For example, ESMF uses DimCount very often to mean number of dimensions instead of its meaning implied in the above equation. We'll clarify the meaning of a word when ambiguity could occur.

Rule 1 is most useful for a user working with Field creation from a Grid and a Fortran data array in most scenarios. It extends to higher dimension count, 3D, 4D, etc... Typically, as the code example demonstrates, a user first creates a Grid, then uses ESMF_GridGet() to retrieve the exclusive counts. Next the user calculates the shape of each Fortran array dimension according to rule 1. The Fortran data array is allocated and initialized based on the computed shape. A Field can either be created in one shot or created empty and finished using ESMF_FieldEmptyComplete.

There are important details that can be skipped but are good to know for ESMF_FieldEmptyComplete and ESMF_FieldCreate from a Fortran data array. 1) these methods require each PET contains exactly one DE. This implies that a code using FieldCreate from a data array or FieldEmptyComplete must have the same number of DEs and PETs, formally $n_{DE} = n_{PET}$. Violation of this condition will cause run time failures. 2) the bounds and counts retrieved from GridGet are DE specific or equivalently PET specific, which means that the Fortran array shape could be different from one PET to another.

    grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
          regDecomp=(/2,2/), name="atmgrid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_GridGet(grid, localDE=0, staggerloc=ESMF_STAGGERLOC_CENTER, &
        exclusiveCount=gec, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    allocate(farray(gec(1), gec(2)) )

    field = ESMF_FieldCreate(grid, farray, ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.12 Create a 2D Field with a 2D Grid and a Fortran data pointer

The setup of this example is similar to the previous section except that the Field is created from a data pointer instead of a data array. We highlight the ability to deallocate the internal Fortran data pointer queried from the Field. This gives a user more flexibility with memory management.

    allocate(farrayPtr(gec(1), gec(2)) )

    field = ESMF_FieldCreate(grid, farrayPtr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    call ESMF_FieldGet(field, farrayPtr=farrayPtr2, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    ! deallocate the retrieved Fortran array pointer
    deallocate(farrayPtr2)


26.3.13 Create a 3D Field with a 2D Grid and a 3D Fortran data array

This example demonstrates a typical use of ESMF_Field combining a 2D grid and a 3D Fortran native data array. One immediate problem follows: how does one define the bounds of the ungridded dimension? This is solved by the optional arguments ungriddedLBound and ungriddedUBound of the ESMF_FieldCreate interface. By definition, ungriddedLBound and ungriddedUBound are both 1 dimensional integer Fortran arrays.

Formally, let fa_shape(j=1...FieldDimCount-GridDimCount) be the shape of the ungridded dimensions of a Field relative to the Grid used in Field creation. The Field dimension count is equal to the number of dimensions of the Fortran array, which equals the number of dimensions of the resultant Field. GridDimCount is the number of dimensions of the Grid.

fa_shape(j) is computed as:

   
    fa_shape(j) = ungriddedUBound(j) - ungriddedLBound(j) + 1

fa_shape is easy to compute when the gridded and ungridded dimensions do not mix. However, it's conceivable that at higher dimension count, gridded and ungridded dimensions can interleave. To aid the computation of ungridded dimension shape we formally introduce the mapping concept.

Let $map_{A,B}(i=1...n_A) = i_B$, and $i_B \in [\phi, 1...n_B]$. $n_A$ is the number of elements in set A, $n_B$ is the number of elements in set B. $map_{A,B}(i)$ defines a mapping from i-th element of set A to $i_B$-th element in set B. $i_B = \phi$ indicates there does not exist a mapping from i-th element of set A to set B.

Suppose we have a mapping from dimension index of ungriddedLBound (or ungriddedUBound) to Fortran array dimension index, called ugb2fa. By definition, $n_A$ equals to the dimension count of ungriddedLBound (or ungriddedUBound), $n_B$ equals to the dimension count of the Fortran array. We can now formulate the computation of ungridded dimension shape as rule 2:

   
    (2) fa_shape(ugb2fa(j)) = ungriddedUBound(j) - ungriddedLBound(j) + 1 
                          j = 1..FortranArrayDimCount - GridDimCount

The mapping can be computed in linear time proportional to the Fortran array dimension count (or rank) using the following algorithm in pseudocode:

  
    map_index = 1
    do i = 1, farray_rank
        if i-th dimension of farray is ungridded
            ugb2fa(map_index) = i
            map_index = map_index + 1
        endif
    enddo

Here we use rank and dimension count interchangeably. These 2 terminologies are typically equivalent. But there are subtle differences under certain conditions. Rank is the total number of dimensions of a tensor object. Dimension count allows a finer description of the heterogeneous dimensions in that object. For example, a Field of rank 5 can have 3 gridded dimensions and 2 ungridded dimensions. Rank is precisely the summation of dimension count of all types of dimensions.

For example, if a 5D array is used with a 3D Grid, there are 2 ungridded dimensions: ungriddedLBound=(/1,2/) and ungriddedUBound=(/5,7/). Suppose the distribution of dimensions looks like (O, X, O, X, O), O means gridded, X means ungridded. Then the mapping from ungridded bounds to Fortran array is ugb2fa=(/2, 4/). The shape of 2nd and 4th dimension of Fortran array should equal (5, 8).

Back to our 3D Field created from a 2D Grid and 3D Fortran array example, suppose the 3rd Field dimension is ungridded, ungriddedLBound=(/3/), ungriddedUBound=(/9/). First we use rule 1 to compute shapes of the gridded Fortran array dimension, then we use rule 2 to compute shapes of the ungridded Fortran array dimension. In this example, we used the exclusive bounds obtained in the previous example.

    fa_shape(1) = gec(1) ! rule 1
    fa_shape(2) = gec(2)
    fa_shape(3) = 7 ! rule 2 9-3+1
    allocate(farray3d(fa_shape(1), fa_shape(2), fa_shape(3)))
    field = ESMF_FieldCreate(grid, farray3d, ESMF_INDEX_DELOCAL, &
        ungriddedLBound=(/3/), ungriddedUBound=(/9/), &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.14 Create a 3D Field with a 2D Grid and a 3D Fortran data array with gridToFieldMap argument

Building upon the previous example, we will create a 3D Field from a 2D grid and 3D array but with a slight twist. In this example, we introduce the gridToFieldMap argument that allows a user to map Grid dimension index to Field dimension index.

In this example, both dimensions of the Grid are distributed and the mapping from DistGrid to Grid is (/1,2/). We will introduce rule 3 assuming distgridToGridMap=(/1,2,3...gridDimCount/), and distgridDimCount equals to gridDimCount. This is a reasonable assumption in typical Field use.

We apply the mapping gridToFieldMap on rule 1 to create rule 3:

   
    (3) fa_shape(gridToFieldMap(i)) = exclusiveCount(i)        
                                  i = 1,..GridDimCount.

Back to our example, suppose the 2nd Field dimension is ungridded, ungriddedLBound=(/3/), ungriddedUBound=(/9/). gridToFieldMap=(/3,1/), meaning the 1st Grid dimension maps to 3rd Field dimension, and 2nd Grid dimension maps to 1st Field dimension.

First we use rule 3 to compute shapes of the gridded Fortran array dimension, then we use rule 2 to compute shapes of the ungridded Fortran array dimension. In this example, we use the exclusive bounds obtained in the previous example.

    gridToFieldMap2d(1) = 3
    gridToFieldMap2d(2) = 1
    do i = 1, 2
        fa_shape(gridToFieldMap2d(i)) = gec(i)
    end do
    fa_shape(2) = 7
    allocate(farray3d(fa_shape(1), fa_shape(2), fa_shape(3)))
    field = ESMF_FieldCreate(grid, farray3d, ESMF_INDEX_DELOCAL, &
        ungriddedLBound=(/3/), ungriddedUBound=(/9/), &
        gridToFieldMap=gridToFieldMap2d, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.15 Create a 3D Field with a 2D Grid and a 3D Fortran data array with halos

This example is similar to example 26.3.14. In addition, here we will show how a user can associate different halo widths to a Fortran array to create a Field through the totalLWidth and totalUWidth optional arguments. A diagram of the dimension configuration from Grid, halos, and Fortran data array is shown here.

Figure 12: Field dimension configuration from Grid, halos, and Fortran data array.
\scalebox{0.75}{\includegraphics{FieldParameterSetup}}

The ESMF_FieldCreate() interface supports creating a Field from a Grid and a Fortran array padded with halos on the distributed dimensions of the Fortran array. Using this technique one can avoid passing non-contiguous Fortran array slice to FieldCreate. It guarantees the same exclusive region, and by using halos, it also defines a bigger total region to contain the entire contiguous memory block of the Fortran array.

The elements of totalLWidth and totalUWidth are applied in the order distributed dimensions appear in the Fortran array. By definition, totalLWidth and totalUWidth are 1 dimensional arrays of non-negative integer values. The size of haloWidth arrays is equal to the number of distributed dimensions of the Fortran array, which is also equal to the number of distributed dimensions of the Grid used in the Field creation.

Because the order of totalWidth (representing both totalLWidth and totalUWidth) element is applied to the order distributed dimensions appear in the Fortran array dimensions, it's quite simple to compute the shape of distributed dimensions of the Fortran array. They are done in a similar manner when applying ungriddedLBound and ungriddedUBound to ungridded dimensions of the Fortran array defined by rule 2.

Assume we have the mapping from the dimension index of totalWidth to the dimension index of Fortran array, called mhw2fa; and we also have the mapping from dimension index of Fortran array to dimension index of the Grid, called fa2g. The shape of distributed dimensions of a Fortran array can be computed by rule 4:

  
    (4) fa_shape(mhw2fa(k)) = exclusiveCount(fa2g(mhw2fa(k)) + 
                              totalUWidth(k) + totalLWidth(k)
                          k = 1...size(totalWidth)

This rule may seem confusing but algorithmically the computation can be done by the following pseudocode:

  
    fa_index = 1
    do i = 1, farray_rank
       if i-th dimension of Fortran array is distributed
           fa_shape(i) = exclusiveCount(fa2g(i)) + 
                         totalUWidth(fa_index) + totalLWidth(fa_index)
           fa_index = fa_index + 1
       endif
    enddo

The only complication then is to figure out the mapping from Fortran array dimension index to Grid dimension index. This process can be done by computing the reverse mapping from Field to Grid.

Typically, we don't have to consider these complications if the following conditions are met: 1) All Grid dimensions are distributed. 2) DistGrid in the Grid has a dimension index mapping to the Grid in the form of natural order (/1,2,3,.../). This natural order mapping is the default mapping between various objects throughout ESMF. 3) Grid to Field mapping is in the form of natural order, i.e. default mapping. These seem like a lot of conditions but they are the default case in the interaction among DistGrid, Grid, and Field. When these conditions are met, which is typically true, the shape of distributed dimensions of Fortran array follows rule 5 in a simple form:

  
    (5) fa_shape(k) = exclusiveCount(k) + 
                      totalUWidth(k) + totalLWidth(k) 
                  k = 1...size(totalWidth)

Let's examine an example on how to apply rule 5. Suppose we have a 5D array and a 3D Grid that has its first 3 dimensions mapped to the first 3 dimensions of the Fortran array. totalLWidth=(/1,2,3/), totalUWidth=(/7,9,10/), then by rule 5, the following pseudo code can be used to compute the shape of the first 3 dimensions of the Fortran array. The shape of the remaining two ungridded dimensions can be computed according to rule 2.

  
    do k = 1, 3
        fa_shape(k) = exclusiveCount(k) + 
                      totalUWidth(k) + totalLWidth(k)) 
    enddo

Suppose now gridToFieldMap=(/2,3,4/) instead which says the first dimension of Grid maps to the 2nd dimension of Field (or Fortran array) and so on and so forth, we can obtain a more general form of rule 5 by introducing first_distdim_index shift when Grid to Field map (gridToFieldMap) is in the form of (/a,a+1,a+2.../).

  
    (6) fa_shape(k+first_distdim_index-1) = exclusiveCount(k) +
                                            totalUWidth(k) + totalLWidth(k)
                                        k = 1...size(totalWidth)

It's obvious that first_distdim_index=a. If the first dimension of the Fortran array is distributed, then rule 6 degenerates into rule 5, which is the typical case.

Back to our example creating a 3D Field from a 2D Grid and a 3D intrinsic Fortran array, we will use the Grid created from previous example that satisfies condition 1 and 2. We'll also use a simple gridToFieldMap (1,2) which is the default mapping that satisfies condition 3. First we use rule 5 to compute the shape of distributed dimensions then we use rule 2 to compute the shape of the ungridded dimensions.

    gridToFieldMap2d(1) = 1
    gridToFieldMap2d(2) = 2
    totalLWidth2d(1) = 3
    totalLWidth2d(2) = 4
    totalUWidth2d(1) = 3
    totalUWidth2d(2) = 5
    do k = 1, 2
        fa_shape(k) = gec(k) + totalLWidth2d(k) + totalUWidth2d(k)
    end do
    fa_shape(3) = 7          ! 9-3+1
    allocate(farray3d(fa_shape(1), fa_shape(2), fa_shape(3)))
    field = ESMF_FieldCreate(grid, farray3d, ESMF_INDEX_DELOCAL, &
        ungriddedLBound=(/3/), ungriddedUBound=(/9/), &
        totalLWidth=totalLWidth2d, totalUWidth=totalUWidth2d, &
        gridToFieldMap=gridToFieldMap2d, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.16 Create a Field from a LocStream, typekind, and rank

In this example, an ESMF_Field is created from an ESMF_LocStream and typekind/rank. The location stream object is uniformly distributed in a 1 dimensional space on 4 DEs. The rank is 1 dimensional. Please refer to LocStream examples section for more information on LocStream creation.

    locs = ESMF_LocStreamCreate(minIndex=1, maxIndex=16, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    field = ESMF_FieldCreate(locs, typekind=ESMF_TYPEKIND_I4, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.17 Create a Field from a LocStream and arrayspec

In this example, an ESMF_Field is created from an ESMF_LocStream and an ESMF_Arrayspec. The location stream object is uniformly distributed in a 1 dimensional space on 4 DEs. The arrayspec is 1 dimensional. Please refer to LocStream examples section for more information on LocStream creation.

    locs = ESMF_LocStreamCreate(minIndex=1, maxIndex=16, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_I4, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    field = ESMF_FieldCreate(locs, arrayspec, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.18 Create a Field from a Mesh, typekind, and rank

In this example, an ESMF_Field is created from an ESMF_Mesh and typekind/rank. The mesh object is on a Euclidean surface that is partitioned to a 2x2 rectangular space with 4 elements and 9 nodes. The nodal space is represented by a distgrid with 9 indices. A Field is created on locally owned nodes on each PET. Therefore, the created Field has 9 data points globally. The mesh object can be represented by the picture below. For more information on Mesh creation, please see Section 33.3.1.

                Mesh Ids
  
    2.0   7 ------- 8 -------- 9
          |         |          |
          |    3    |    4     |
          |         |          |
    1.0   4 ------- 5 -------- 6
          |         |          |
          |    1    |    2     |
          |         |          |
    0.0   1 ------- 2 -------- 3
  
         0.0       1.0        2.0 
  
        Node Ids at corners
        Element Ids in centers
   
  
               Mesh Owners
  
    2.0   2 ------- 2 -------- 3
          |         |          |
          |    2    |    3     |
          |         |          |
    1.0   0 ------- 0 -------- 1
          |         |          |
          |    0    |    1     |
          |         |          |
    0.0   0 ------- 0 -------- 1
  
         0.0       1.0        2.0 
  
        Node Owners at corners
        Element Owners in centers

      ! Create Mesh structure in 1 step
      mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
             nodeIds=nodeIds, nodeCoords=nodeCoords, &
             nodeOwners=nodeOwners, elementIds=elemIds,&
             elementTypes=elemTypes, elementConn=elemConn, &
             rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

      ! Field is created on the 1 dimensional nodal distgrid. On
      ! each PET, Field is created on the locally owned nodes.
      field = ESMF_FieldCreate(mesh, typekind=ESMF_TYPEKIND_I4, rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.19 Create a Field from a Mesh and arrayspec

In this example, an ESMF_Field is created from an ESMF_Mesh and an ESMF_Arrayspec. The mesh object is on a Euclidean surface that is partitioned to a 2x2 rectangular space with 4 elements and 9 nodes. The nodal space is represented by a distgrid with 9 indices. Field is created on locally owned nodes on each PET. Therefore, the created Field has 9 data points globally. The mesh object can be represented by the picture below. For more information on Mesh creation, please see Section 33.3.1.

      ! Create Mesh structure in 1 step
      mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
             nodeIds=nodeIds, nodeCoords=nodeCoords, &
             nodeOwners=nodeOwners, elementIds=elemIds,&
             elementTypes=elemTypes, elementConn=elemConn, &
             rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

      call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_I4, rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

      ! Field is created on the 1 dimensional nodal distgrid. On
      ! each PET, Field is created on the locally owned nodes.
      field = ESMF_FieldCreate(mesh, arrayspec, rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.20 Create a Field from a Mesh and an Array

In this example, an ESMF_Field is created from an ESMF_Mesh and an ESMF_Array. The mesh object is created in the previous example and the array object is retrieved from the field created in the previous example too.

    call ESMF_MeshGet(mesh, nodalDistgrid=distgrid, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    array = ESMF_ArrayCreate(distgrid=distgrid, arrayspec=arrayspec, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    ! query the array from the previous example
    call ESMF_FieldGet(field, array=array, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    ! create a Field from a mesh and an array
    field1 = ESMF_FieldCreate(mesh, array, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.21 Create a Field from a Mesh and an ArraySpec with optional features

In this example, an ESMF_Field is created from an ESMF_Mesh and an ESMF_ArraySpec. The mesh object is created in the previous example. The Field is also created with optional arguments such as ungridded dimensions and dimension mapping.

In this example, the mesh is mapped to the 2nd dimension of the ESMF_Field, with its first dimension being the ungridded dimension with bounds 1,3.

    call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_I4, rc=rc)
    field = ESMF_FieldCreate(mesh, arrayspec=arrayspec, gridToFieldMap=(/2/), &
        ungriddedLBound=(/1/), ungriddedUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.22 Create a Field with replicated dimensions

In this example an ESMF_Field with replicated dimension is created from an ESMF_Grid and an ESMF_Arrayspec. A user can also use other ESMF_FieldCreate() methods to create replicated dimension Field, this example illustrates the key concepts and use of a replicated dimension Field.

Normally gridToFieldMap argument in ESMF_FieldCreate() should not contain 0 value entries. However, for a Field with replicated dimension, a 0 entry in gridToFieldMap indicates the corresponding Grid dimension is replicated in the Field. In such a Field, the rank of the Field is no longer necessarily greater than its Grid rank. An example will make this clear. We will start by creating Distgrid and Grid.

    ! create 4D distgrid
    distgrid = ESMF_DistGridCreate(minIndex=(/1,1,1,1/), &
        maxIndex=(/6,4,6,4/), regDecomp=(/2,1,2,1/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create 4D grid on top of the 4D distgrid
    grid = ESMF_GridCreate(distgrid=distgrid, name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create 3D arrayspec
    call ESMF_ArraySpecSet(arrayspec, 3, ESMF_TYPEKIND_R8, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

In this example, a user creates a 3D Field with replicated dimension replicated along the 2nd and 4th dimension of its underlying 4D Grid. In addition, the 2nd dimension of the Field is ungridded (why?). The 1st and 3rd dimensions of the Field have halos.

    ! create field, 2nd and 4th dimensions of the Grid are replicated
    field = ESMF_FieldCreate(grid, arrayspec, indexflag=ESMF_INDEX_DELOCAL, &
        gridToFieldMap=(/1,0,2,0/), &
        ungriddedLBound=(/1/), ungriddedUBound=(/4/), &
        totalLWidth=(/1,1/), totalUWidth=(/4,5/), &
        staggerloc=ESMF_STAGGERLOC_CORNER, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! get basic information from the field
    call ESMF_FieldGet(field, grid=grid1, array=array, typekind=typekind, &
        dimCount=dimCount, staggerloc=lstaggerloc, &
        gridToFieldMap=lgridToFieldMap, ungriddedLBound=lungriddedLBound, &
        ungriddedUBound=lungriddedUBound, totalLWidth=ltotalLWidth, &
        totalUWidth=ltotalUWidth, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! get bounds information from the field
    call ESMF_FieldGet(field, localDe=0, farrayPtr=farray, &
        exclusiveLBound=felb, exclusiveUBound=feub, exclusiveCount=fec, &
        computationalLBound=fclb, computationalUBound=fcub, &
        computationalCount=fcc, totalLBound=ftlb, totalUBound=ftub, &
        totalCount=ftc, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

Next we verify that the field and array bounds agree with each other

    call ESMF_ArrayGet(array, rank=arank, dimCount=adimCount, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    gridrank_repdim = 0
    do i = 1, size(gridToFieldMap)
        if(gridToFieldMap(i) == 0) gridrank_repdim = gridrank_repdim + 1
    enddo

Number of undistributed dimension of the array X is computed from total rank of the array A, the dimension count of its underlying distgrid B and number of replicated dimension in the distgrid C. We have the following formula: X = A - (B - C)

    allocate(audlb(arank-adimCount+gridrank_repdim), &
        audub(arank-adimCount+gridrank_repdim))
    call ESMF_ArrayGet(array, exclusiveLBound=aelb, exclusiveUBound=aeub, &
        computationalLBound=aclb, computationalUBound=acub, &
        totalLBound=atlb, totalUBound=atub, &
        undistLBound=audlb, undistUBound=audub, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! verify the ungridded bounds from field match
    ! undistributed bounds from its underlying array
    do i = 1, arank-adimCount
        if(lungriddedLBound(i) .ne. audlb(i) ) &
            rc = ESMF_FAILURE
    enddo
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    do i = 1, arank-adimCount
        if(lungriddedUBound(i) .ne. audub(i) ) &
            rc = ESMF_FAILURE
    enddo
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

We then verify the data in the replicated dimension Field can be updated and accessed.

    do ik = ftlb(3), ftub(3)
     do ij = ftlb(2), ftub(2)
      do ii = ftlb(1), ftub(1)
        farray(ii,ij,ik) = ii+ij*2+ik
      enddo
     enddo
    enddo
    ! access and verify
    call ESMF_FieldGet(field, localDe=0, farrayPtr=farray1, &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    do ik = ftlb(3), ftub(3)
     do ij = ftlb(2), ftub(2)
      do ii = ftlb(1), ftub(1)
        n = ii+ij*2+ik
        if(farray1(ii,ij,ik) .ne. n ) rc = ESMF_FAILURE
      enddo
     enddo
    enddo
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! release resources
    call ESMF_FieldDestroy(field)
    call ESMF_GridDestroy(grid)
    call ESMF_DistGridDestroy(distgrid)


26.3.23 Create a Field on an arbitrarily distributed Grid

With the introduction of Field on arbitrarily distributed Grid, Field has two kinds of dimension count: one associated geometrical (or physical) dimensionality, the other one associated with its memory index space representation. Field and Grid dimCount reflect the physical index space of the objects. A new type of dimCount rank should be added to both of these entities. The rank gives the number of dimensions of the memory index space of the objects. This would be the dimension of the pointer pulled out of Field and the size of the bounds vector, for example.

For non-arbitrary Grids rank=dimCount, but for grids and fields with arbitrary dimensions rank = dimCount - (number of Arb dims) + 1 (Internally Field can use the Arb info from the grid to create the mapping from the Field Array to the DistGrid)

When creating a Field size(GridToFieldMap)=dimCount for both Arb and Non-arb grids This array specifies the mapping of Field to Grid identically for both Arb and Nonarb grids If a zero occurs in an entry corresponding to any arbitrary dimension, then a zero must occur in every entry corresponding to an arbitrary dimension (i.e. all arbitrary dimensions must either be all replicated or all not replicated, they can't be broken apart).

In this example an ESMF_Field is created from an arbitrarily distributed ESMF_Grid and an ESMF_Arrayspec. A user can also use other ESMF_FieldCreate() methods to create such a Field, this example illustrates the key concepts and use of Field on arbitrary distributed Grid.

The Grid is 3 dimensional in physics index space but the first two dimension are collapsed into a single memory index space. Thus the resulting Field is 3D in physics index space and 2D in memory index space. This is made obvious with the 2D arrayspec used to create this Field.

    ! create a 3D grid with the first 2 dimensions collapsed 
    ! and arbitrarily distributed
    grid3d = ESMF_GridCreateNoPeriDim(coordTypeKind=ESMF_TYPEKIND_R8, &
      minIndex=(/1,1,1/), maxIndex=(/xdim, ydim,zdim/), &
      arbIndexList=localArbIndex,arbIndexCount=localArbIndexCount, &
      name="arb3dgrid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create a 2D arrayspec
    call ESMF_ArraySpecSet(arrayspec2D, rank=2, typekind=ESMF_TYPEKIND_R4, &
         rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create a 2D Field using the Grid and the arrayspec
    field = ESMF_FieldCreate(grid3d, arrayspec2D, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  
    call ESMF_FieldGet(field, rank=rank, dimCount=dimCount, &
                       rc=rc)
    if (myPet .eq. 0) print *, 'Field rank, dimCount', &
                                rank, dimCount
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  
    ! verify that the dimension counts are correct
    if (rank .ne. 2) correct = .false.
    if (dimCount .ne. 3) correct = .false.


26.3.24 Create a Field on an arbitrarily distributed Grid with replicated dimensions & ungridded bounds

The next example is slightly more complicated in that the Field also contains one ungridded dimension and its gridded dimension is replicated on the arbitrarily distributed dimension of the Grid.

The same 3D Grid and 2D arrayspec in the previous example are used but a gridToFieldMap argument is supplied to the ESMF_FieldCreate() call. The first 2 entries of the map are 0, the last (3rd) entry is 1. The 3rd dimension of the Grid is mapped to the first dimension of the Field, this dimension is then replicated on the arbitrarily distributed dimensions of the Grid. In addition, the Field also has one ungridded dimension. Thus the final dimension count of the Field is 2 in both physics and memory index space.

    field = ESMF_FieldCreate(grid3d, arrayspec2D,gridToFieldMap=(/0,0,1/), &
            ungriddedLBound=(/1/), ungriddedUBound=(/10/),rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  
    call ESMF_FieldGet(field, rank=rank, dimCount=dimCount, &
                       rc=rc)
    if (myPet .eq. 0) print *, 'Field rank, dimCount', &
                                rank, dimCount
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  
    if (rank .ne. 2) correct = .false.
    if (dimCount .ne. 2) correct = .false.


26.3.25 Field regridding

This section describes the Field regrid methods. For an in depth description of ESMF regridding and the options available please see Section 24.2.

The basic flow of ESMF Field regridding is as follows. First a source and destination geometry object are created, depending on the regrid method they can be either a Grid, a Mesh, an XGrid, or a LocStream. Next Fields are built on the source and destination grid objects. These Fields are then passed into ESMF_FieldRegridStore(). The user can either get a sparse matrix from this call and/or a routeHandle. If the user gets the sparse matrix then they are responsible for deallocating it, but other than that can use it as they wish. The routeHandle can be used in the ESMF_FieldRegrid() call to perform the actual interpolation of data from the source to the destination field. This interpolation can be repeated for the same set of Fields as long as the coordinates at the staggerloc involved in the regridding in the associated grid object don't change. The same routeHandle can also be used between any pair of Fields that matches the original pari in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability. However, if you want the routehandle to be the same interpolation between the grid objects upon which the Fields are built as was calculated with the original ESMF_FieldRegridStore() call, then there are additional constraints on the grid objects. To be the same interpolation, the grid objects upon which the Fields are build must contain the same coordinates at the stagger locations involved in the regridding as the original source and destination Fields used in the ESMF_FieldRegridStore() call. The routehandle represents the interpolation between the grid objects as they were during the ESMF_FieldRegridStore() call. So if the coordinates at the stagger location in the grid objects change, a new call to ESMF_FieldRegridStore() is necessary to compute the interpolation between that new set of coordinates. When finished with the routeHandle ESMF_FieldRegridRelease() should be used to free the associated memory.

The following example demonstrates doing a regrid operation between two Fields.

  ! (Create source Grid, Mesh, XGrid, or LocStream.)
  ! (Create srcField on the above.)

  ! (Create destination Grid, Mesh, XGrid, or LocStream.)
  ! (Create dstField on the above.)
  
  ! Create the routeHandle which encodes the communication and
  ! information necessary for the regrid sparse matrix multiply.
  call ESMF_FieldRegridStore(srcField=srcField, dstField=dstField, &
                             routeHandle=routeHandle, rc=localrc)

 
  ! Can loop here regridding from srcField to dstField 
  ! do i=1,....

       ! (Put data into srcField)

       ! Use the routeHandle to regrid data from srcField to dstField.
       ! As described above, the same routeHandle can be used to 
       ! regrid a large class of different source and destination Fields. 
       call ESMF_FieldRegrid(srcField, dstField, routeHandle, rc=localrc)

  !    (Use data in dstField)

  ! enddo 


  ! Free the buffers and data associated with the routeHandle. 
  call ESMF_FieldRegridRelease(routeHandle, rc=localrc)

26.3.26 Field regrid with masking

As before, to create the sparse matrix regrid operator we call the ESMF_FieldRegridStore() routine. However, in this case we apply masking to the regrid operation. The mask value for each index location in the Grids may be set using the ESMF_GridAddItem() call (see Section 31.3.17 and Section 31.3.18). Mask values may be set independently for the source and destination Grids. If no mask values have been set in a Grid, then it is assumed no masking should be used for that Grid. The srcMaskValues parameter allows the user to set the list of values which indicate that a source location should be masked out. The dstMaskValues parameter allows the user to set the list of values which indicate that a destination location should be masked out. The absence of one of these parameters indicates that no masking should be used for that Field (e.g no srcMaskValue parameter indicates that source masking shouldn't occur). The unmappedaction flag may be used with or without masking and indicates what should occur if destination points can not be mapped to a source cell. Here the ESMF_UNMAPPEDACTION_IGNORE value indicates that unmapped destination points are to be ignored and no sparse matrix entries should be generated for them.

  call ESMF_FieldRegridStore(srcField=srcField, srcMaskValues=(/1/),       &
                             dstField=dstField, dstMaskValues=(/1/),       &
                             unmappedaction=ESMF_UNMAPPEDACTION_IGNORE, &
                             routeHandle=routeHandle,                      &
                             regridmethod=ESMF_REGRIDMETHOD_BILINEAR,     &
                             rc=localrc)

The ESMF_FieldRegrid and ESMF_FieldRegridRelease calls may then be applied as in the previous example.

26.3.27 Field regrid example: Mesh to Mesh

This example demonstrates the regridding process between Fields created on Meshes. First the Meshes are created. This example omits the setup of the arrays describing the Mesh, but please see Section 33.3.1 for examples of this. After creation Fields are constructed on the Meshes, and then ESMF_FieldRegridStore() is called to construct a RouteHandle implementing the regrid operation. Finally, ESMF_FieldRegrid() is called with the Fields and the RouteHandle to do the interpolation between the source Field and destination Field. Note the coordinates of the source and destination Mesh should be in degrees.

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! Create Source Mesh
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  ! Create the Mesh structure.
  ! For brevity's sake, the code to fill the Mesh creation 
  ! arrays is omitted from this example. However, here
  ! is a brief description of the arrays:
  ! srcNodeIds    - the global ids for the src nodes
  ! srcNodeCoords - the coordinates for the src nodes
  ! srcNodeOwners - which PET owns each src node
  ! srcElemIds    - the global ids of the src elements
  ! srcElemTypes  - the topological shape of each src element
  ! srcElemConn   - how to connect the nodes to form the elements
  !                 in the source mesh
  ! Several examples of setting up these arrays can be seen in
  ! the Mesh Section "Mesh Creation". 
  srcMesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
         nodeIds=srcNodeIds, nodeCoords=srcNodeCoords, &
         nodeOwners=srcNodeOwners, elementIds=srcElemIds,&
         elementTypes=srcElemTypes, elementConn=srcElemConn, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)



  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! Create and Fill Source Field
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  ! Set description of source Field
  call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_R8, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Create source Field
  srcField = ESMF_FieldCreate(srcMesh, arrayspec, &
                        name="source", rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Get source Field data pointer to put data into
  call ESMF_FieldGet(srcField, 0, fptr1D,  rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Get number of local nodes to allocate space
  ! to hold local node coordinates
  call ESMF_MeshGet(srcMesh, &
         numOwnedNodes=numOwnedNodes, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Allocate space to hold local node coordinates
  ! (spatial dimension of Mesh*number of local nodes)
  allocate(ownedNodeCoords(2*numOwnedNodes))

  ! Get local node coordinates
  call ESMF_MeshGet(srcMesh, &
         ownedNodeCoords=ownedNodeCoords, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Set the source Field to the function 20.0+x+y
  do i=1,numOwnedNodes
    ! Get coordinates
    x=ownedNodeCoords(2*i-1)
    y=ownedNodeCoords(2*i)

   ! Set source function
   fptr1D(i) = 20.0+x+y
  enddo

  ! Deallocate local node coordinates
  deallocate(ownedNodeCoords)


  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! Create Destination Mesh
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  ! Create the Mesh structure.
  ! For brevity's sake, the code to fill the Mesh creation 
  ! arrays is omitted from this example. However, here
  ! is a brief description of the arrays:
  ! dstNodeIds    - the global ids for the dst nodes
  ! dstNodeCoords - the coordinates for the dst nodes
  ! dstNodeOwners - which PET owns each dst node
  ! dstElemIds    - the global ids of the dst elements
  ! dstElemTypes  - the topological shape of each dst element
  ! dstElemConn   - how to connect the nodes to form the elements
  !                 in the destination mesh
  ! Several examples of setting up these arrays can be seen in
  ! the Mesh Section "Mesh Creation". 
  dstMesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
         nodeIds=dstNodeIds, nodeCoords=dstNodeCoords, &
         nodeOwners=dstNodeOwners, elementIds=dstElemIds,&
         elementTypes=dstElemTypes, elementConn=dstElemConn, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! Create Destination Field
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  ! Set description of source Field
  call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_R8, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Create destination Field
  dstField = ESMF_FieldCreate(dstMesh, arrayspec, &
                        name="destination", rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! Do Regrid
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  ! Compute RouteHandle which contains the regrid operation
  call ESMF_FieldRegridStore( &
          srcField, &
          dstField=dstField, &
          routeHandle=routeHandle, &
          regridmethod=ESMF_REGRIDMETHOD_BILINEAR, &
          rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Perform Regrid operation moving data from srcField to dstField
  call ESMF_FieldRegrid(srcField, dstField, routeHandle, rc=rc)


  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! dstField now contains the interpolated data.
  ! If the Meshes don't change, then routeHandle
  ! may be used repeatedly to interpolate from 
  ! srcField to dstField.  
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

   
  ! User code to use the routeHandle, Fields, and
  ! Meshes goes here before they are freed below.


  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! Free the objects created in the example.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  ! Free the RouteHandle
  call ESMF_FieldRegridRelease(routeHandle, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Free the Fields
  call ESMF_FieldDestroy(srcField, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  call ESMF_FieldDestroy(dstField, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  ! Free the Meshes
  call ESMF_MeshDestroy(dstMesh, rc=rc)

  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

  call ESMF_MeshDestroy(srcMesh, rc=rc)
 
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.28 Gather Field data onto root PET

User can use ESMF_FieldGather interface to gather Field data from multiple PETs onto a single root PET. This interface is overloaded by type, kind, and rank.

Note that the implementation of Scatter and Gather is not sequence index based. If the Field is built on arbitrarily distributed Grid, Mesh, LocStream or XGrid, Gather will not gather data to rootPet from source data points corresponding to the sequence index on the rootPet. Instead Gather will gather a contiguous memory range from source PET to rootPet. The size of the memory range is equal to the number of data elements on the source PET. Vice versa for the Scatter operation. In this case, the user should use ESMF_FieldRedist to achieve the same data operation result. For examples how to use ESMF_FieldRedist to perform Gather and Scatter, please refer to 26.3.32 and 26.3.31.

In this example, we first create a 2D Field, then use ESMF_FieldGather to collect all the data in this Field into a data pointer on PET 0.

    ! Get current VM and pet number
    call ESMF_VMGetCurrent(vm, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_VMGet(vm, localPet=lpe, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Create a 2D Grid and use this grid to create a Field
    ! farray is the Fortran data array that contains data on each PET.
    grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
        regDecomp=(/2,2/), &
        name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    field = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, rc=localrc)
    if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


    call ESMF_FieldGet(field, farrayPtr=fptr, rc=localrc)
    if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    !---------Initialize pet specific field data----------------
    !    1        5         10
    ! 1  +--------+---------+
    !    |        |         |
    !    |   0    |    1    |
    !    |        |         |
    ! 10 +--------+---------+
    !    |        |         |
    !    |   2    |    3    |
    !    |        |         |
    ! 20 +--------+---------+
    fptr = lpe

    ! allocate the Fortran data array on PET 0 to store gathered data
    if(lpe .eq. 0) then
      allocate (farrayDst(10,20))
    else
      allocate (farrayDst(0,0))
    end if
    call ESMF_FieldGather(field, farrayDst, rootPet=0, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! check that the values gathered on rootPet are correct
    if(lpe .eq. 0) then
       do i = 1, 5
          do j = 1, 10
             if(farrayDst(i, j) .ne. 0) localrc=ESMF_FAILURE
          enddo
       enddo
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
       do i = 6, 10
          do j = 1, 10
             if(farrayDst(i, j) .ne. 1) localrc=ESMF_FAILURE
          enddo
       enddo
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
       do i = 1, 5
          do j = 11, 20
             if(farrayDst(i, j) .ne. 2) localrc=ESMF_FAILURE
          enddo
       enddo
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
       do i = 6, 10
          do j = 11, 20
             if(farrayDst(i, j) .ne. 3) localrc=ESMF_FAILURE
          enddo
       enddo
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif

    ! destroy all objects created in this example to prevent memory leak
    call ESMF_FieldDestroy(field, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    call ESMF_GridDestroy(grid, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    if(lpe .eq. 0) deallocate(farrayDst)


26.3.29 Scatter Field data from root PET onto its set of joint PETs

User can use ESMF_FieldScatter interface to scatter Field data from root PET onto its set of joint PETs. This interface is overloaded by type, kind, and rank.

In this example, we first create a 2D Field, then use ESMF_FieldScatter to scatter the data from a data array located on PET 0 onto this Field.

    ! Create a 2D Grid and use this grid to create a Field
    ! farray is the Fortran data array that contains data on each PET.
    grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
        regDecomp=(/2,2/), &
        name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    field = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, rc=localrc)
    if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! initialize values to be scattered
    !    1        5         10
    ! 1  +--------+---------+
    !    |        |         |
    !    |   0    |    1    |
    !    |        |         |
    ! 10 +--------+---------+
    !    |        |         |
    !    |   2    |    3    |
    !    |        |         |
    ! 20 +--------+---------+
    if(lpe .eq. 0) then
        allocate(farraySrc(10,20))
        farraySrc(1:5,1:10) = 0
        farraySrc(6:10,1:10) = 1
        farraySrc(1:5,11:20) = 2
        farraySrc(6:10,11:20) = 3
    else
      allocate (farraySrc(0,0))
    endif

    ! scatter the data onto individual PETs of the Field
    call ESMF_FieldScatter(field, farraySrc, rootPet=0, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_FieldGet(field, localDe=0, farrayPtr=fptr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! verify that the scattered data is properly distributed
    do i = lbound(fptr, 1), ubound(fptr, 1)
        do j = lbound(fptr, 2), ubound(fptr, 2)
            if(fptr(i, j) .ne. lpe) localrc = ESMF_FAILURE
        enddo
        if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    enddo

    ! destroy all objects created in this example to prevent memory leak
    call ESMF_FieldDestroy(field, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    call ESMF_GridDestroy(grid, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    if(lpe .eq. 0) deallocate(farraySrc)


26.3.30 Redistribute data from source Field to destination Field

User can use ESMF_FieldRedist interface to redistribute data from source Field to destination Field. This interface is overloaded by type and kind; In the version of ESMF_FieldRedist without factor argument, a default value of 1 is used.

In this example, we first create two 1D Fields, a source Field and a destination Field. Then we use ESMF_FieldRedist to redistribute data from source Field to destination Field.

    ! Get current VM and pet number
    call ESMF_VMGetCurrent(vm, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_VMGet(vm, localPet=localPet, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create grid
    distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
            regDecomp=(/4/), &
            rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    grid = ESMF_GridCreate(distgrid=distgrid, &
        name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create srcField
    ! +--------+--------+--------+--------+
    !      0        1        2        3            ! value
    ! 1        4        8        12       16       ! bounds
    srcField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, &
      indexflag=ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_FieldGet(srcField, farrayPtr=srcfptr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    srcfptr(:) = localPet

    ! create dstField
    ! +--------+--------+--------+--------+
    !      0        0        0        0            ! value
    ! 1        4        8        12       16       ! bounds
    dstField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, &
      indexflag=ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_FieldGet(dstField, farrayPtr=dstfptr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  
    dstfptr(:) = 0

    ! perform redist
    ! 1. setup routehandle from source Field to destination Field
    call ESMF_FieldRedistStore(srcField, dstField, routehandle, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! 2. use precomputed routehandle to redistribute data
    call ESMF_FieldRedist(srcfield, dstField, routehandle, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! verify redist
    call ESMF_FieldGet(dstField, localDe=0, farrayPtr=fptr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Verify that the redistributed data in dstField is correct.
    ! Before the redist op, the dst Field contains all 0. 
    ! The redist op reset the values to the PE value, verify this is the case.
    do i = lbound(fptr, 1), ubound(fptr, 1)
        if(fptr(i) .ne. localPet) localrc = ESMF_FAILURE
    enddo
    if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

Field redistribution can also be performed between different Field pairs that match the original Fields in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

    call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=2, rc=rc)

Create two fields with ungridded dimensions using the Grid created previously. The new Field pair has matching number of elements. The ungridded dimension is mapped to the first dimension of either Field.

    srcFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
        ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)

    dstFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
        ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)

Using the previously computed routehandle, the Fields can be redistributed.

    call ESMF_FieldRedist(srcfieldA, dstFieldA, routehandle, rc=rc)

    call ESMF_FieldRedistRelease(routehandle, rc=rc)


26.3.31 FieldRedist as a form of scatter involving arbitrary distribution

User can use ESMF_FieldRedist interface to redistribute data from source Field to destination Field, where the destination Field is built on an arbitrarily distributed structure, e.g. ESMF_Mesh. The underlying mechanism is explained in section 28.2.19.

In this example, we will create 2 one dimensional Fields, the src Field has a regular decomposition and holds all its data on a single PET, in this case PET 0. The destination Field is built on a Mesh which is itself built on an arbitrarily distributed distgrid. Then we use ESMF_FieldRedist to redistribute data from source Field to destination Field, similar to a traditional scatter operation.

The src Field only has data on PET 0 where it is sequentially initialized, i.e. 1,2,3...This data will be redistributed (or scattered) from PET 0 to the destination Field arbitrarily distributed on all the PETs.

    ! a one dimensional grid whose elements are all located on PET 0
    distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/9/), &
        regDecomp=(/1/), &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    grid = ESMF_GridCreate(distgrid=distgrid, &
        indexflag=ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    srcField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! initialize the source data
    if (localPet == 0) then
        call ESMF_FieldGet(srcField, farrayPtr=srcfptr, rc=rc)
        if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
        do i = 1, 9
            srcfptr(i) = i
        enddo
    endif

For more information on Mesh creation, user can refer to Mesh examples section or Field creation on Mesh example for more details.

      ! Create Mesh structure
      mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
             nodeIds=nodeIds, nodeCoords=nodeCoords, &
             nodeOwners=nodeOwners, elementIds=elemIds,&
             elementTypes=elemTypes, elementConn=elemConn, &
             rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

Create the destination Field on the Mesh that is arbitrarily distributed on all the PETs.

      dstField = ESMF_FieldCreate(mesh, typekind=ESMF_TYPEKIND_I4, rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

Perform the redistribution from source Field to destination Field.

     call ESMF_FieldRedistStore(srcField, dstField, &
             routehandle=routehandle, rc=rc)
     if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
     call ESMF_FieldRedist(srcField, dstField, routehandle=routehandle, rc=rc)
     if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

We can now verify that the sequentially initialized source data is scattered on to the destination Field. The data has been scattered onto the destination Field with the following distribution.

  
   4 elements on PET 0:  1 2 4 5
   2 elements on PET 1:  3 6
   2 elements on PET 2:  7 8
   1 element  on PET 3:  9
Because the redistribution is index based, the elements also corresponds to the index space of Mesh in the destination Field.

    call ESMF_FieldGet(dstField, farrayPtr=dstfptr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

The scatter operation is successful. Since the routehandle computed with ESMF_FieldRedistStore can be reused, user can use the same routehandle to scatter multiple source Fields from a single PET to multiple destination Fields distributed on all PETs. The gathering operation is just the opposite of the demonstrated scattering operation, where a user would redist from a source Field distributed on multiple PETs to a destination Field that only has data storage on a single PET.

Now it's time to release all the resources.

    call ESMF_FieldRedistRelease(routehandle=routehandle, rc=rc)


26.3.32 FieldRedist as a form of gather involving arbitrary distribution

Similarly, one can use the same approach to gather the data from an arbitrary distribution to a non-arbitrary distribution. This concept is demonstrated by using the previous Fields but the data operation is reversed. This time data is gathered from the Field built on the mesh to the Field that has only data allocation on rootPet.

First a FieldRedist routehandle is created from the Field built on Mesh to the Field that has only data allocation on rootPet.

    call ESMF_FieldRedistStore(dstField, srcField, routehandle=routehandle, &
         rc=rc)

Perform FieldRedist, this will gather the data points from the Field built on mesh to the data pointer on the rootPet (default to 0) stored in the srcField.

    call ESMF_FieldRedist(dstField, srcField, routehandle=routehandle, rc=rc)

Release the routehandle used for the gather operation.

    call ESMF_FieldRedistRelease(routehandle=routehandle, rc=rc)


26.3.33 Sparse matrix multiplication from source Field to destination Field

The ESMF_FieldSMM() interface can be used to perform sparse matrix multiplication from source Field to destination Field. This interface is overloaded by type and kind;

In this example, we first create two 1D Fields, a source Field and a destination Field. Then we use ESMF_FieldSMM to perform sparse matrix multiplication from source Field to destination Field.

The source and destination Field data are arranged such that each of the 4 PETs has 4 data elements. Moreover, the source Field has all its data elements initialized to a linear function based on local PET number. Then collectively on each PET, a SMM according to the following formula is preformed:
$dstField(i) = i * srcField(i), i = 1 ... 4$

Because source Field data are initialized to a linear function based on local PET number, the formula predicts that the result destination Field data on each PET is 1,2,3,4. This is verified in the example.

Section 28.2.18 provides a detailed discussion of the sparse matrix multiplication operation implemented in ESMF.

    ! Get current VM and pet number
    call ESMF_VMGetCurrent(vm, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_VMGet(vm, localPet=lpe, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create distgrid and grid
    distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
        regDecomp=(/4/), &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    grid = ESMF_GridCreate(distgrid=distgrid, &
        name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_GridGetFieldBounds(grid, localDe=0, totalCount=fa_shape, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create src\_farray, srcArray, and srcField
    ! +--------+--------+--------+--------+
    !      1        2        3        4            ! value
    ! 1        4        8        12       16       ! bounds
    allocate(src_farray(fa_shape(1)) )
    src_farray = lpe+1
    srcArray = ESMF_ArrayCreate(distgrid, src_farray, &
                indexflag=ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    srcField = ESMF_FieldCreate(grid, srcArray, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! create dst_farray, dstArray, and dstField
    ! +--------+--------+--------+--------+
    !      0        0        0        0            ! value
    ! 1        4        8        12       16       ! bounds
    allocate(dst_farray(fa_shape(1)) )
    dst_farray = 0
    dstArray = ESMF_ArrayCreate(distgrid, dst_farray, &
                indexflag=ESMF_INDEX_DELOCAL, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    dstField = ESMF_FieldCreate(grid, dstArray, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! perform sparse matrix multiplication
    ! 1. setup routehandle from source Field to destination Field
    ! initialize factorList and factorIndexList
    allocate(factorList(4))
    allocate(factorIndexList(2,4))
    factorList = (/1,2,3,4/)
    factorIndexList(1,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
    factorIndexList(2,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)

    call ESMF_FieldSMMStore(srcField, dstField, routehandle, &
        factorList, factorIndexList, rc=localrc)
    if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! 2. use precomputed routehandle to perform SMM
    call ESMF_FieldSMM(srcfield, dstField, routehandle, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! verify sparse matrix multiplication
    call ESMF_FieldGet(dstField, localDe=0, farrayPtr=fptr, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! Verify that the result data in dstField is correct.
    ! Before the SMM op, the dst Field contains all 0.
    ! The SMM op reset the values to the index value, verify this is the case.
    ! +--------+--------+--------+--------+
    !  1 2 3 4  2 4 6 8  3 6 9 12  4 8 12 16       ! value
    ! 1        4        8        12       16       ! bounds
    do i = lbound(fptr, 1), ubound(fptr, 1)
        if(fptr(i) /= i*(lpe+1)) rc = ESMF_FAILURE
    enddo

Field sparse matrix multiplication can also be applied between Fields that matche the original Fields in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability

    call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=2, rc=rc)

Create two fields with ungridded dimensions using the Grid created previously. The new Field pair has matching number of elements. The ungridded dimension is mapped to the first dimension of either Field.

    srcFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
        ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)

    dstFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
        ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)

Using the previously computed routehandle, the sparse matrix multiplication can be performed between the Fields.

    call ESMF_FieldSMM(srcfieldA, dstFieldA, routehandle, rc=rc)

    ! release route handle
    call ESMF_FieldSMMRelease(routehandle, rc=rc)

In the following discussion, we demonstrate how to set up a SMM routehandle between a pair of Fields that are different in number of gridded dimensions and the size of those gridded dimensions. The source Field has a 1D decomposition with 16 total elements; the destination Field has a 2D decomposition with 12 total elements. For ease of understanding of the actual matrix calculation, a global indexing scheme is used.

    distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
        indexflag=ESMF_INDEX_GLOBAL, &
        regDecomp=(/4/), &
        rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    grid = ESMF_GridCreate(distgrid=distgrid, &
        indexflag=ESMF_INDEX_GLOBAL, &
        name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    call ESMF_GridGetFieldBounds(grid, localDe=0, totalLBound=tlb, &
                       totalUBound=tub, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

create 1D src_farray, srcArray, and srcField

   +  PET0  +  PET1  +  PET2  +  PET3  +
   +--------+--------+--------+--------+
        1        2        3        4            ! value
   1        4        8        12       16       ! bounds of seq indices

    allocate(src_farray2(tlb(1):tub(1)) )
    src_farray2 = lpe+1
    srcArray = ESMF_ArrayCreate(distgrid, src_farray2, &
                  indexflag=ESMF_INDEX_GLOBAL, &
      rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    !print *, lpe, '+', tlb, tub, '+', src_farray2

    srcField = ESMF_FieldCreate(grid, srcArray, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

Create 2D dstField on the following distribution (numbers are the sequence indices):

   +  PET0  +  PET1  +  PET2  +  PET3  +
   +--------+--------+--------+--------+
   |        |        |        |        |
   |   1    |   4    |   7    |   10   |
   |        |        |        |        |
   +--------+--------+--------+--------+
   |        |        |        |        |
   |   2    |   5    |   8    |   11   |
   |        |        |        |        |
   +--------+--------+--------+--------+
   |        |        |        |        |
   |   3    |   6    |   9    |   12   |
   |        |        |        |        |
   +--------+--------+--------+--------+

    ! Create the destination Grid
    dstGrid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/3,4/), &
      indexflag = ESMF_INDEX_GLOBAL, &
      regDecomp = (/1,4/), &
      rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    dstField = ESMF_FieldCreate(dstGrid, typekind=ESMF_TYPEKIND_R4, &
      indexflag=ESMF_INDEX_GLOBAL, &
      rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

Perform sparse matrix multiplication $dst_i$ = $M_{i,j}$ * $src_j$ First setup routehandle from source Field to destination Field with prescribed factorList and factorIndexList.

The sparse matrix is of size 12x16, however only the following entries are filled:

   M(3,1) = 0.1
   M(3,10) = 0.4
   M(8,2) = 0.25
   M(8,16) = 0.5
   M(12,1) = 0.3
   M(12,16) = 0.7

By the definition of matrix calculation, the 8th element on PET2 in the dstField equals to 0.25*srcField(2) + 0.5*srcField(16) = 0.25*1+0.5*4=2.25 For simplicity, we will load the factorList and factorIndexList on PET 0 and 1, the SMMStore engine will load balance the parameters on all 4 PETs internally for optimal performance.

    if(lpe == 0) then
      allocate(factorList(3), factorIndexList(2,3))
      factorList=(/0.1,0.4,0.25/)
      factorIndexList(1,:)=(/1,10,2/)
      factorIndexList(2,:)=(/3,3,8/)
      call ESMF_FieldSMMStore(srcField, dstField, routehandle=routehandle, &
          factorList=factorList, factorIndexList=factorIndexList, rc=localrc)
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    else if(lpe == 1) then
      allocate(factorList(3), factorIndexList(2,3))
      factorList=(/0.5,0.3,0.7/)
      factorIndexList(1,:)=(/16,1,16/)
      factorIndexList(2,:)=(/8,12,12/)
      call ESMF_FieldSMMStore(srcField, dstField, routehandle=routehandle, &
          factorList=factorList, factorIndexList=factorIndexList, rc=localrc)
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    else
      call ESMF_FieldSMMStore(srcField, dstField, routehandle=routehandle, &
          rc=localrc)
      if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif

    ! 2. use precomputed routehandle to perform SMM
    call ESMF_FieldSMM(srcfield, dstField, routehandle=routehandle, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


26.3.34 Field Halo solving a domain decomposed heat transfer problem

The ESMF_FieldHalo() interface can be used to perform halo updates for a Field. This eases communication programming from a user perspective. By definition, the user program only needs to update locally owned exclusive region in each domain, then call FieldHalo to communicate the values in the halo region from/to neighboring domain elements. In this example, we solve a 1D heat transfer problem: $u_t = \alpha^2 u_{xx}$ with the initial condition $u(0, x) = 20$ and boundary conditions $u(t, 0) = 10, u(t, 1) = 40$. The temperature field $u$ is represented by a ESMF_Field. A finite difference explicit time stepping scheme is employed. During each time step, FieldHalo update is called to communicate values in the halo region to neighboring domain elements. The steady state (as $t \rightarrow \infty$) solution is a linear temperature profile along $x$. The numerical solution is an approximation of the steady state solution. It can be verified to represent a linear temperature profile.

Section 28.2.15 provides a discussion of the halo operation implemented in ESMF_Array.

! create 1D distgrid and grid decomposed according to the following diagram:
! +------------+   +----------------+   +---------------+   +--------------+
! |   DE 0  |  |   |  |   DE 1   |  |   |  |   DE 2  |  |   |  |   DE 3    |
! |  1 x 16 |  |   |  |  1 x 16  |  |   |  |  1 x 16 |  |   |  |  1 x 16   |
! |         | 1|<->|1 |          | 1|<->|1 |         | 1|<->|1 |           |
! |         |  |   |  |          |  |   |  |         |  |   |  |           |
! +------------+   +----------------+   +---------------+   +--------------+
    distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/npx/), &
        regDecomp=(/4/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    grid = ESMF_GridCreate(distgrid=distgrid, name="grid", rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    ! set up initial condition and boundary conditions of the
    ! temperature Field
    if(lpe == 0) then
        allocate(fptr(17), tmp_farray(17))
        fptr = 20.
        fptr(1) = 10.
        tmp_farray(1) = 10.
        startx = 2
        endx = 16

        field = ESMF_FieldCreate(grid, fptr, totalUWidth=(/1/), &
                name="temperature", rc=rc)
        if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    else if(lpe == 3) then
        allocate(fptr(17), tmp_farray(17))
        fptr = 20.
        fptr(17) = 40.
        tmp_farray(17) = 40.
        startx = 2
        endx = 16

        field = ESMF_FieldCreate(grid, fptr, totalLWidth=(/1/), &
                name="temperature", rc=rc)
        if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    else
        allocate(fptr(18), tmp_farray(18))
        fptr = 20.
        startx = 2
        endx = 17

        field = ESMF_FieldCreate(grid, fptr, &
            totalLWidth=(/1/), totalUWidth=(/1/), name="temperature", rc=rc)
        if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    endif

    ! compute the halo update routehandle of the decomposed temperature Field
    call ESMF_FieldHaloStore(field, routehandle=routehandle, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

    dt = 0.01
    dx = 1./npx
    alpha = 0.1

    ! Employ explicit time stepping
    ! Solution converges after about 9000 steps based on apriori knowledge.
    ! The result is a linear temperature profile stored in field.
    do iter = 1, 9000
     ! only elements in the exclusive region are updated locally
     ! in each domain
     do i = startx, endx
       tmp_farray(i) = &
       fptr(i)+alpha*alpha*dt/dx/dx*(fptr(i+1)-2.*fptr(i)+fptr(i-1))
      enddo
      fptr = tmp_farray
     ! call halo update to communicate the values in the halo region to
     ! neighboring domains
     call ESMF_FieldHalo(field, routehandle=routehandle, rc=rc)
     if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
    enddo

    ! release the halo routehandle
    call ESMF_FieldHaloRelease(routehandle, rc=rc)

26.4 Restrictions and Future Work

  1. CAUTION: It depends on the specific entry point of ESMF_FieldCreate() used during Field creation, which Fortran operations are supported on the Fortran array pointer farrayPtr, returned by ESMF_FieldGet(). Only if the ESMF_FieldCreate() from pointer variant was used, will the returned farrayPtr variable contain the original bounds information, and be suitable for the Fortran deallocate() call. This limitation is a direct consequence of the Fortran 95 standard relating to the passing of array arguments.

  2. No mathematical operators. The Fields class does not currently support advanced operations on fields, such as differential or other mathematical operators.

26.5 Design and Implementation Notes

  1. Some methods which have a Field interface are actually implemented at the underlying Grid or Array level; they are inherited by the Field class. This allows the user API (Application Programming Interface) to present functions at the level which is most consistent to the application without restricting where inside the ESMF the actual implementation is done.

  2. The Field class is implemented in Fortran, and as such is defined inside the framework by a Field derived type and a set of subprograms (functions and subroutines) which operate on that derived type. The Field class itself is very thin; it is a container class which groups a Grid and an Array object together.

  3. Fields follow the framework-wide convention of the unison creation and operation rule: All PETs which are part of the currently executing VM must create the same Fields at the same point in their execution. Since an early user request was that global object creation not impose the overhead of a barrier or synchronization point, Field creation does no inter-PET communication. For this to work, each PET must query the total number of PETs in this VM, and which local PET number it is. It can then compute which DE(s) are part of the local decomposition, and any global information can be computed in unison by all PETs independently of the others. In this way the overhead of communication is avoided, at the cost of more difficulty in diagnosing program bugs which result from not all PETs executing the same create calls.

  4. Related to the item above, the user request to not impose inter-PET communication at object creation time means that requirement FLD 1.5.1, that all Fields will have unique names, and if not specified, the framework will generate a unique name for it, is difficult or impossible to support. A part of this requirement has been implemented; a unique object counter is maintained in the Base object class, and if a name is not given at create time a name such as "Field003" is generated which is guaranteed to not be repeated by the framework. However, it is impossible to error check that the user has not replicated a name, and it is possible under certain conditions that if not all PETs have created the same number of objects, that the counters on different PETs may not stay synchronized. This remains an open issue.

26.6 Class API

26.6.1 ESMF_FieldAssignment(=) - Field assignment


INTERFACE:

   interface assignment(=)
   field1 = field2
ARGUMENTS:
   type(ESMF_Field) :: field1
   type(ESMF_Field) :: field2
STATUS:

DESCRIPTION:

Assign field1 as an alias to the same ESMF Field object in memory as field2. If field2 is invalid, then field1 will be equally invalid after the assignment.

The arguments are:

field1
The ESMF_Field object on the left hand side of the assignment.
field2
The ESMF_Field object on the right hand side of the assignment.

26.6.2 ESMF_FieldOperator(==) - Field equality operator


INTERFACE:

   interface operator(==)
   if (field1 == field2) then ... endif
   OR
   result = (field1 == field2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_Field), intent(in) :: field1
   type(ESMF_Field), intent(in) :: field2
STATUS:

DESCRIPTION:

Test whether field1 and field2 are valid aliases to the same ESMF Field object in memory. For a more general comparison of two ESMF Fields, going beyond the simple alias test, the ESMF_FieldMatch() function (not yet implemented) must be used.

The arguments are:

field1
The ESMF_Field object on the left hand side of the equality operation.
field2
The ESMF_Field object on the right hand side of the equality operation.

26.6.3 ESMF_FieldOperator(/=) - Field not equal operator


INTERFACE:

   interface operator(/=)
   if (field1 /= field2) then ... endif
   OR
   result = (field1 /= field2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_Field), intent(in) :: field1
   type(ESMF_Field), intent(in) :: field2
STATUS:

DESCRIPTION:

Test whether field1 and field2 are not valid aliases to the same ESMF Field object in memory. For a more general comparison of two ESMF Fields, going beyond the simple alias test, the ESMF_FieldMatch() function (not yet implemented) must be used.

The arguments are:

field1
The ESMF_Field object on the left hand side of the non-equality operation.
field2
The ESMF_Field object on the right hand side of the non-equality operation.

26.6.4 ESMF_FieldCopy - Copy data from one Field object to another


INTERFACE:

   subroutine ESMF_FieldCopy(fieldOut, fieldIn, rc)
ARGUMENTS:
        type(ESMF_Field), intent(inout) :: fieldOut
        type(ESMF_Field), intent(in) :: fieldIn
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer, intent(out), optional :: rc
DESCRIPTION:

Copy data from one ESMF_Field object to another.

The arguments are:

fieldOut
ESMF_Field object into which to copy the data. The incoming fieldOut must already references a matching memory allocation.
fieldIn
ESMF_Field object that holds the data to be copied.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.5 ESMF_FieldCreate - Create a Field from Geom and typekind


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
 function ESMF_FieldCreateGBTKR(geom, typekind, &
      gridToFieldMap, ungriddedLBound, ungriddedUBound, &
      totalLWidth, totalUWidth, pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateGBTKR
ARGUMENTS:
     type(ESMF_Geom), intent(in) :: geom
     type(ESMF_TypeKind_Flag),intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     integer, intent(in), optional :: totalLWidth(:)
     integer, intent(in), optional :: totalUWidth(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.4.

The arguments are:

geom
ESMF_Geom object.
typekind
The typekind of the Field. See section 54.61 for a list of valid typekind options.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.6 ESMF_FieldCreate - Create a Field from Geom and ArraySpec


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateGBArraySpec(geom, arrayspec, gridToFieldMap, &
     ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, &
     pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateGBArraySpec
ARGUMENTS:
     type(ESMF_Geom) :: geom
     type(ESMF_ArraySpec), intent(in) :: arrayspec
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     integer, intent(in), optional :: totalLWidth(:)
     integer, intent(in), optional :: totalUWidth(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.5.

The arguments are:

geom
ESMF_Geom object.
arrayspec
Data type and kind specification.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.7 ESMF_FieldCreate - Create a Field from Geom and Array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateGBArray(geom, array, datacopyflag, &
     gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     totalLWidth, totalUWidth, name, vm, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateGBArray
ARGUMENTS:
     type(ESMF_Geom), intent(in) :: geom
     type(ESMF_Array), intent(in) :: array
     type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     integer, intent(in), optional :: totalLWidth(:)
     integer, intent(in), optional :: totalUWidth(:)
     character (len = *), intent(in), optional :: name
     type(ESMF_VM), intent(in), optional :: vm
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.

The arguments are:

geom
ESMF_Geom object.
array
ESMF_Array object.
[datacopyflag]
Indicates whether to copy the contents of the array or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[name]
Field name.
[vm]
If present, the Field object is created on the specified ESMF_VM object. The default is to create on the VM of the current component context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.8 ESMF_FieldCreate - Create a Field from Geom and Fortran array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateGBData<rank><type><kind>(geom, & 
   farray, indexflag, datacopyflag, gridToFieldMap, ungriddedLBound, & 
   ungriddedUBound, totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateGBData<rank><type><kind>
ARGUMENTS:
   type(ESMF_Geom) :: geom 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), target :: farray 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data array and ESMF_Geom. The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed. The arguments are:

geom
ESMF_Geom object. The dimCount of the Geom must be smaller than or equal to the rank of the farray.
farray
Native Fortran data array to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Whether to copy the contents of the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farray rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farray dimensions less the total (distributed + undistributed) dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the farray. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farray. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the farray. That is, for each gridded dimension the farray size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farray. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the farray. That is, for each gridded dimension the farray size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.9 ESMF_FieldCreate - Create a Field from Geom and Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateGBDataPtr<rank><type><kind>(geom, & 
   farrayPtr, datacopyflag, gridToFieldMap, & 
   totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateGBDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_Geom) :: geom 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: farrayPtr 
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data pointer and ESMF_Geom. The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

geom
ESMF_Geom object. The dimCount of the Geom must be smaller than or equal to the rank of the farrayPtr.
farrayPtr
Native Fortran data pointer to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Whether to copy the contents of the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farrayPtr. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the farrayPtr. That is, for each gridded dimension the farrayPtr size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farrayPtr. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the farrayPtr. That is, for each gridded dimension the farrayPtr size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.10 ESMF_FieldCreate - Create a Field from Grid and typekind


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateGridTKR(grid, typekind, &
     indexflag, staggerloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     totalLWidth, totalUWidth, pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateGridTKR
ARGUMENTS:
     type(ESMF_Grid), intent(in) :: grid
     type(ESMF_TypeKind_Flag),intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Index_Flag), intent(in), optional :: indexflag
     type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     integer, intent(in), optional :: totalLWidth(:)
     integer, intent(in), optional :: totalUWidth(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.4.

The arguments are:

grid
ESMF_Grid object.
typekind
The typekind of the Field. See section 54.61 for a list of valid typekind options.
[indexflag]
Indicate how DE-local indices are defined. By default each DE's exclusive region is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated Grid. See section 54.29 for a list of valid indexflag options. The default indexflag value is the one stored in then ESMF_Grid object. Currently it is erroneous to specify an indexflag different from the one stored in the ESMF_Grid object. The default value is ESMF_INDEX_DELOCAL
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.11 ESMF_FieldCreate - Create a Field from Grid and ArraySpec


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateGridArraySpec(grid, arrayspec, &
     indexflag, staggerloc, gridToFieldMap, ungriddedLBound, &
     ungriddedUBound, totalLWidth, totalUWidth, pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateGridArraySpec
ARGUMENTS:
     type(ESMF_Grid), intent(in) :: grid
     type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Index_Flag), intent(in), optional :: indexflag
     type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     integer, intent(in), optional :: totalLWidth(:)
     integer, intent(in), optional :: totalUWidth(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.5.

The arguments are:

grid
ESMF_Grid object.
arrayspec
Data type and kind specification.
[indexflag]
Indicate how DE-local indices are defined. By default each DE's exclusive region is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated Grid. See section 54.29 for a list of valid indexflag options. The default indexflag value is the one stored in then ESMF_Grid object. Currently it is erroneous to specify an indexflag different from the one stored in the ESMF_Grid object.The default value is ESMF_INDEX_DELOCAL
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.12 ESMF_FieldCreate - Create a Field from Grid and Array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateGridArray(grid, array, datacopyflag, &
     staggerloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     totalLWidth, totalUWidth, name, vm, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateGridArray
ARGUMENTS:
     type(ESMF_Grid), intent(in) :: grid
     type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
     type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     integer, intent(in), optional :: totalLWidth(:)
     integer, intent(in), optional :: totalUWidth(:)
     character (len = *), intent(in), optional :: name
     type(ESMF_VM), intent(in), optional :: vm
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.

The arguments are:

grid
ESMF_Grid object.
array
ESMF_Array object.
[datacopyflag]
Indicates whether to copy the contents of the array or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[name]
Field name.
[vm]
If present, the Field object is constructed on the specified ESMF_VM object. The default is to construct on the VM of the current component context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.13 ESMF_FieldCreate - Create a Field from Grid and Fortran array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateGridData<rank><type><kind>(grid, & 
   farray, indexflag, datacopyflag, staggerloc, & 
   gridToFieldMap, ungriddedLBound, ungriddedUBound, & 
   totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateGridData<rank><type><kind>
ARGUMENTS:
   type(ESMF_Grid), intent(in) :: grid 
   <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_StaggerLoc), intent(in), optional :: staggerloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Field from a Fortran data array and ESMF_Grid. The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed. For examples and associated documentation regarding this method see section 26.3.11, 26.3.13, 26.3.14, 26.3.15, and 26.3.9.

The arguments are:

grid
ESMF_Grid object.
farray
Native Fortran data array to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options. Currently it is erroneous to specify an indexflag different from the one stored in the ESMF_Grid object.
[datacopyflag]
Whether to copy the contents of the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farray rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farray dimensions less the total (distributed + undistributed) dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the farray. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farray. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the farray. That is, for each gridded dimension the farray size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farray. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the farray. That is, for each gridded dimension the farray size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.14 ESMF_FieldCreate - Create a Field from Grid and Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateGridDataPtr<rank><type><kind>(grid, & 
   farrayPtr, datacopyflag, staggerloc, gridToFieldMap, & 
   totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateGridDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_Grid), intent(in) :: grid 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_StaggerLoc), intent(in), optional :: staggerloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Field from a Fortran data pointer and ESMF_Grid. The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

For examples and associated documentation regarding this method see section 26.3.12, 26.3.13, 26.3.14, 26.3.15, and 26.3.9.

The arguments are:

grid
ESMF_Grid object.
farrayPtr
Native Fortran data pointer to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Whether to copy the contents of the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farrayPtr. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the farrayPtr. That is, for each gridded dimension the farrayPtr size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the farrayPtr. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the farrayPtr. That is, for each gridded dimension the farrayPtr size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.15 ESMF_FieldCreate - Create a Field from LocStream and typekind


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateLSTKR(locstream, typekind, &
     gridToFieldMap, ungriddedLBound, ungriddedUBound, pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateLSTKR
ARGUMENTS:
     type(ESMF_LocStream), intent(in) :: locstream
     type(ESMF_TypeKind_Flag),intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.16.

The arguments are:

locstream
ESMF_LocStream object.
typekind
The typekind of the Field. See section 54.61 for a list of valid typekind options.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.16 ESMF_FieldCreate - Create a Field from LocStream and ArraySpec


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateLSArraySpec(locstream, arrayspec, &
     gridToFieldMap, ungriddedLBound, ungriddedUBound, pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateLSArraySpec
ARGUMENTS:
     type(ESMF_LocStream), intent(in) :: locstream
     type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.17.

The arguments are:

locstream
ESMF_LocStream object.
arrayspec
Data type and kind specification.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.17 ESMF_FieldCreate - Create a Field from LocStream and Array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateLSArray(locstream, array, &
     datacopyflag, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateLSArray
ARGUMENTS:
     type(ESMF_LocStream), intent(in) :: locstream
     type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     character (len = *), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.

The arguments are:

locstream
ESMF_LocStream object.
array
ESMF_Array object.
[datacopyflag]
Indicates whether to copy the contents of the array or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.18 ESMF_FieldCreate - Create a Field from LocStream and Fortran array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateLSData<rank><type><kind>(locstream, farray, & 
   indexflag, datacopyflag, gridToFieldMap, ungriddedLBound, & 
   ungriddedUBound, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateLSData<rank><type><kind>
ARGUMENTS:
   type(ESMF_LocStream), intent(in) :: locstream 
   <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data array and ESMF_LocStream. The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

locstream
ESMF_LocStream object.
farray
Native Fortran data array to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Whether to copy the contents of the farray or reference directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the locstream's dimCount. The list elements map each dimension of the locstream to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the locstream's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farray rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farray dimensions less the total (distributed + undistributed) dimensions in the locstream. Unlocstreamded dimensions must be in the same order they are stored in the farray. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than locstream dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than locstream dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.19 ESMF_FieldCreate - Create a Field from LocStream and Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateLSDataPtr<rank><type><kind>(locstream, & 
   farrayPtr, datacopyflag, gridToFieldMap, & 
   name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateLSDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_LocStream), intent(in) :: locstream 
   <type> (ESMF_KIND_<kind>),pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data pointer and ESMF_LocStream. The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

locstream
ESMF_LocStream object.
farrayPtr
Native Fortran data pointer to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Whether to copy the contents of the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the locstream's dimCount. The list elements map each dimension of the locstream to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the locstream's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the locstream. Unlocstreamded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.20 ESMF_FieldCreate - Create a Field from Mesh and typekind


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateMeshTKR(mesh, typekind, indexflag, &
     meshloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateMeshTKR
ARGUMENTS:
     type(ESMF_Mesh), intent(in) :: mesh
     type(ESMF_TypeKind_Flag), intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Index_Flag), intent(in), optional :: indexflag
     type(ESMF_MeshLoc), intent(in), optional :: meshloc
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.18.

The arguments are:

mesh
ESMF_Mesh object.
typekind
The typekind of the Field. See section 54.61 for a list of valid typekind options.
[indexflag]
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.21 ESMF_FieldCreate - Create a Field from Mesh and ArraySpec


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateMeshArraySpec(mesh, arrayspec, &
     indexflag, meshloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateMeshArraySpec
ARGUMENTS:
     type(ESMF_Mesh), intent(in) :: mesh
     type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Index_Flag),intent(in), optional :: indexflag
     type(ESMF_MeshLoc), intent(in), optional :: meshloc
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.19 and 26.3.21.

The arguments are:

mesh
ESMF_Mesh object.
arrayspec
Data type and kind specification.
[indexflag]
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.22 ESMF_FieldCreate - Create a Field from Mesh and Array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateMeshArray(mesh, array, &
     datacopyflag, meshloc, &
     gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     name, vm, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateMeshArray
ARGUMENTS:
     type(ESMF_Mesh), intent(in) :: mesh
     type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
     type(ESMF_MeshLoc), intent(in), optional :: meshloc
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     character (len = *), intent(in), optional :: name
     type(ESMF_VM), intent(in), optional :: vm
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.20.

The arguments are:

mesh
ESMF_Mesh object.
array
ESMF_Array object.
[datacopyflag]
Indicates whether to copy the contents of the array or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[name]
Field name.
[vm]
If present, the Field object is constructed on the specified ESMF_VM object. The default is to construct on the VM of the current component context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.23 ESMF_FieldCreate - Create a Field from Mesh and Fortran array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateMeshData<rank><type><kind>(mesh, & 
   farray, indexflag, datacopyflag, meshloc, & 
   gridToFieldMap, ungriddedLBound, ungriddedUBound, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateMeshData<rank><type><kind>
ARGUMENTS:
   type(ESMF_Mesh), intent(in) :: mesh 
   <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_MeshLoc), intent(in), optional :: meshloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data array and ESMF_Mesh. The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

mesh
ESMF_Mesh object.
farray
Native Fortran data array to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Whether to copy the contents of the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the mesh's dimCount. The list elements map each dimension of the mesh to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the mesh's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farray rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farray dimensions less the total (distributed + undistributed) dimensions in the mesh. Unmeshded dimensions must be in the same order they are stored in the farray. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than mesh dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than mesh dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.24 ESMF_FieldCreate - Create a Field from Mesh and Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateMeshDataPtr<rank><type><kind>(mesh, & 
   farrayPtr, datacopyflag, meshloc, gridToFieldMap, & 
   name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateMeshDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_Mesh), intent(in) :: mesh 
   <type> (ESMF_KIND_<kind>),pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_MeshLoc), intent(in), optional :: meshloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data pointer and ESMF_Mesh. The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

mesh
ESMF_Mesh object.
farrayPtr
Native Fortran data pointer to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Whether to copy the contents of the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the mesh's dimCount. The list elements map each dimension of the mesh to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the mesh's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the mesh. Unmeshded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.25 ESMF_FieldCreate - Create a Field from XGrid and typekind


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateXGTKR(xgrid, typekind, xgridside, &
     gridindex, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateXGTKR
ARGUMENTS:
     type(ESMF_XGrid), intent(in) :: xgrid
     type(ESMF_TypeKind_Flag), intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside
     integer, intent(in), optional :: gridindex
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.16.

The arguments are:

xgrid
ESMF_XGrid object.
typekind
The typekind of the Field. See section 54.61 for a list of valid typekind options.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridSide is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the XGrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular XGrid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.26 ESMF_FieldCreate - Create a Field from XGrid and ArraySpec


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateXGArraySpec(xgrid, arrayspec, &
     xgridside, gridindex, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     pinflag, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateXGArraySpec
ARGUMENTS:
     type(ESMF_XGrid), intent(in) :: xgrid
     type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_XGridSide_Flag), intent(in), optional :: xgridSide
     integer, intent(in), optional :: gridIndex
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     type(ESMF_Pin_Flag), intent(in), optional :: pinflag
     character (len=*), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.17.

The arguments are:

xgrid
ESMF_XGrid object.
arrayspec
Data type and kind specification.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridside is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the XGrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular XGrid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.27 ESMF_FieldCreate - Create a Field from XGrid and Array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateXGArray(xgrid, array, &
     datacopyflag, xgridside, gridindex, &
     gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateXGArray
ARGUMENTS:
     type(ESMF_XGrid), intent(in) :: xgrid
     type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
     type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside
     integer, intent(in), optional :: gridindex
     integer, intent(in), optional :: gridToFieldMap(:)
     integer, intent(in), optional :: ungriddedLBound(:)
     integer, intent(in), optional :: ungriddedUBound(:)
     character (len = *), intent(in), optional :: name
     integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.

The arguments are:

xgrid
ESMF_XGrid object.
array
ESMF_Array object.
[datacopyflag]
Indicates whether to copy the contents of the array or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridSide is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the XGrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular XGrid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.28 ESMF_FieldCreate - Create a Field from XGrid and Fortran array


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateXGData<rank><type><kind>(xgrid, & 
   farray, indexflag, datacopyflag, xgridside, gridindex, & 
   gridToFieldMap, ungriddedLBound, ungriddedUBound, name,& 
   rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateXGData<rank><type><kind>
ARGUMENTS:
   type(ESMF_XGrid), intent(in) :: xgrid 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside 
   integer, intent(in), optional :: gridindex 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data array and ESMF_Xgrid. The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

xgrid
ESMF_XGrid object.
farray
Native Fortran data array to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Whether to copy the contents of the farray or reference directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridside is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the xgrid's dimCount. The list elements map each dimension of the xgrid to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the xgrid's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farray rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farray dimensions less the total (distributed + undistributed) dimensions in the xgrid. Unxgridded dimensions must be in the same order they are stored in the farray. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the Xgrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Xgrid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than xgrid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than xgrid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the farray.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.29 ESMF_FieldCreate - Create a Field from XGrid and Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_FieldCreate() 
   function ESMF_FieldCreateXGDataPtr<rank><type><kind>(xgrid, farrayPtr, & 
   datacopyflag, xgridside, & 
   gridindex, gridToFieldMap, name, rc)
RETURN VALUE:
   type(ESMF_Field) :: ESMF_FieldCreateXGDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_XGrid), intent(in) :: xgrid 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside 
   integer, intent(in), optional :: gridindex 
   integer, intent(in), optional :: gridToFieldMap(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field from a Fortran data pointer and ESMF_Xgrid. The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

xgrid
ESMF_XGrid object.
farrayPtr
Native Fortran data pointer to be copied/referenced in the Field The Field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Whether to copy the contents of the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridside is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the xgrid's dimCount. The list elements map each dimension of the xgrid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the xgrid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the farrayPtr rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total farrayPtr dimensions less the total (distributed + undistributed) dimensions in the xgrid. Unxgridded dimensions must be in the same order they are stored in the farrayPtr. Permutations of the order of dimensions are handled via individual communication methods. For example, an undistributed dimension can be remapped to a distributed dimension as part of the ESMF_ArrayRedist() operation. If the Field dimCount is less than the Xgrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Xgrid dimension will be replicating the Field across the DEs along this direction.
[name]
Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.30 ESMF_FieldCreate - Create Field object from an existing Field object


INTERFACE:

   ! Private name; call using ESMF_FieldCreate()
   function ESMF_FieldCreateFromField(field, datacopyflag, &
     trailingUngridSlice, name, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldCreateFromField
ARGUMENTS:
        type(ESMF_Field), intent(in) :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
        integer, intent(in), optional :: trailingUngridSlice(:)
        character (len=*), intent(in), optional :: name
        integer, intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_Field object from an existing Field.

The return value is the newly created ESMF_Field object. Supports array slicing.

The arguments are:

field
ESMF_Field object to be used as originator.
[datacopyflag]
Specifies whether the created Field object references the memory allocation provided by field directly or copies the data from field into a new memory allocation. For valid values see 54.13. The default is ESMF_DATACOPY_VALUE.
[trailingUngridSlice]
If present, the returned Field refers to a slice of field created by removing some or all of the trailing (i.e. rightmost) ungridded dimensions from the incoming Field object. The number of trailing dimensions removed is equal to the size $n$ of trailingUngridSlice, which must be less than or equal to the total number of trailing ungridded dimensions in field. The trailingUngridSlice argument is interpreted as an index tuple for the $n$ trailing ungridded dimensions in field in order, and the Field slice at that location is returned.
[name]
Name of the Field object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.31 ESMF_FieldDestroy - Release resources associated with a Field


INTERFACE:

   subroutine ESMF_FieldDestroy(field, noGarbage, rc)
ARGUMENTS:
     type(ESMF_Field), intent(inout) :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: noGarbage
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Destroys the ESMF_Field, releasing the resources associated with the object.

If an ESMF_Grid is associated with field, it will not be released.

By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.

The arguments are:

field
ESMF_Field object.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.32 ESMF_FieldEmptyComplete - Complete a Field from arrayspec


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete()
 subroutine ESMF_FieldEmptyCompAS(field, arrayspec, indexflag, &
  gridToFieldMap, ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, &
  pinflag, rc)
ARGUMENTS:
  type(ESMF_Field), intent(inout) :: field
  type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
  type(ESMF_Index_Flag), intent(in), optional :: indexflag
  integer, intent(in), optional :: gridToFieldMap(:)
  integer, intent(in), optional :: ungriddedLBound(:)
  integer, intent(in), optional :: ungriddedUBound(:)
  integer, intent(in), optional :: totalLWidth(:)
  integer, intent(in), optional :: totalUWidth(:)
  type(ESMF_Pin_Flag), intent(in), optional :: pinflag
  integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Complete an ESMF_Field and allocate space internally for an ESMF_Array based on arrayspec. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.

The arguments are:

field
The input ESMF_Field with a status of ESMF_FIELDSTATUS_GRIDSET.
arrayspec
Data type and kind specification.
[indexflag]
Indicate how DE-local indices are defined. By default each DE's exclusive region is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated Grid. See section 54.29 for a list of valid indexflag options.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.33 ESMF_FieldEmptyComplete - Complete a Field from typekind


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete()
 subroutine ESMF_FieldEmptyCompTK(field, typekind, indexflag, &
  gridToFieldMap, ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, &
  pinflag, rc)
ARGUMENTS:
  type(ESMF_Field), intent(inout) :: field
  type(ESMF_TypeKind_Flag), intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
  type(ESMF_Index_Flag), intent(in), optional :: indexflag
  integer, intent(in), optional :: gridToFieldMap(:)
  integer, intent(in), optional :: ungriddedLBound(:)
  integer, intent(in), optional :: ungriddedUBound(:)
  integer, intent(in), optional :: totalLWidth(:)
  integer, intent(in), optional :: totalUWidth(:)
  type(ESMF_Pin_Flag), intent(in), optional :: pinflag
  integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Complete an ESMF_Field and allocate space internally for an ESMF_Array based on typekind. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.

For an example and associated documentation using this method see section 26.3.7.

The arguments are:

field
The input ESMF_Field with a status of ESMF_FIELDSTATUS_GRIDSET.
typekind
Data type and kind specification.
[indexflag]
Indicate how DE-local indices are defined. By default each DE's exclusive region is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated Grid. See section 54.29 for a list of valid indexflag options.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the Field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.34 ESMF_FieldEmptyComplete - Complete a Field from Fortran array


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyComp<rank><type><kind>(field, & 
   farray, indexflag, datacopyflag, gridToFieldMap, & 
   ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Complete an ESMF_Field and allocate space internally for an ESMF_Array based on typekind. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.

The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

For an example and associated documentation using this method see section 26.3.8.

The arguments are:

field
The input ESMF_Field with a status of ESMF_FIELDSTATUS_GRIDSET. The ESMF_Field will have the same dimension (dimCount) as the rank of the farray.
farray
Native Fortran data array to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Indicates whether to copy the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farray dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.35 ESMF_FieldEmptyComplete - Complete a Field from Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompPtr<rank><type><kind>(field, & 
   farrayPtr, datacopyflag, gridToFieldMap, & 
   totalLWidth, totalUWidth, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Complete an ESMF_Field and allocate space internally for an ESMF_Array based on typekind. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.

The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

field
The input ESMF_Field with a status of ESMF_FIELDSTATUS_GRIDSET. The ESMF_Field will have the same dimension (dimCount) as the rank of the farrayPtr.
farrayPtr
Native Fortran data pointer to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Indicates whether to copy the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farrayPtr dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.36 ESMF_FieldEmptyComplete - Complete a Field from Geom started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompGB<rank><type><kind>(field, geom, & 
   farray, indexflag, datacopyflag, gridToFieldMap, ungriddedLBound, & 
   ungriddedUBound, totalLWidth, totalUWidth, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_Geom), intent(in) :: geom 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), target :: farray 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call. For an example and associated documentation using this method see section 26.3.7.

The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farray.
geom
The ESMF_Geom object to complete the Field.
farray
Native Fortran data array to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Indicates whether to copy the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the geom's dimCount. The list elements map each dimension of the geom to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the geom's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farray dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.37 ESMF_FieldEmptyComplete - Complete a Field from Geom started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompGBPtr<rank><type><kind>(field, geom, & 
   farrayPtr, datacopyflag, gridToFieldMap, & 
   totalLWidth, totalUWidth, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inou) :: field 
   type(ESMF_Geom), intent(in) :: geom 
   <type> (ESMF_KIND_<kind>), dimension(<rank>), pointer :: farrayPtr 
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call. For an example and associated documentation using this method see section 26.3.7.

The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farrayPtr.
geom
The ESMF_Geom object to complete the Field. The dimCount of the Geom must be smaller than or equal to the rank of the farrayPtr.
farrayPtr
Native Fortran data pointer to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Indicates whether to copy the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the geom's dimCount. The list elements map each dimension of the geom to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the geom's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farrayPtr dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.38 ESMF_FieldEmptyComplete - Complete a Field from Grid started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompGrid<rank><type><kind>(field, grid, & 
   farray, indexflag, datacopyflag, staggerloc, gridToFieldMap, & 
   ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_Grid), intent(in) :: grid 
   <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_STAGGERLOC), intent(in), optional :: staggerloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farray.
grid
The ESMF_Grid object to complete the Field.
farray
Native Fortran data array to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Indicates whether to copy the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farray dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.39 ESMF_FieldEmptyComplete - Complete a Field from Grid started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompGridPtr<rank><type><kind>(field, grid, & 
   farrayPtr, datacopyflag, staggerloc, gridToFieldMap, & 
   totalLWidth, totalUWidth, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_Grid), intent(in) :: grid 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_STAGGERLOC), intent(in), optional :: staggerloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management. The Fortran data pointer inside ESMF_Field can be queried and deallocated when

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farrayPtr.
grid
The ESMF_Grid object to complete the Field.
farrayPtr
Native Fortran data pointer to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Indicates whether to copy the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farrayPtr dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Grid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Grid dimension will be replicating the Field across the DEs along this direction.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.40 ESMF_FieldEmptyComplete - Complete a Field from LocStream started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompLS<rank><type><kind>(field, locstream, & 
   farray, indexflag, datacopyflag, gridToFieldMap, & 
   ungriddedLBound, ungriddedUBound, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_LocStream), intent(in) :: locstream 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farray.
locstream
The ESMF_LocStream object to complete the Field.
farray
Native Fortran data array to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Indicates whether to copy the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the locstream's dimCount. The list elements map each dimension of the locstream to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the locstream's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farray dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than locstream dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than locstream dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.41 ESMF_FieldEmptyComplete - Complete a Field from LocStream started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompLSPtr<rank><type><kind>(field, locstream, & 
   farrayPtr, datacopyflag, gridToFieldMap, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_LocStream), intent(in) :: locstream 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farrayPtr.
locstream
The ESMF_LocStream object to complete the Field.
farrayPtr
Native Fortran data pointer to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Indicates whether to copy the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[gridToFieldMap]
List with number of elements equal to the locstream's dimCount. The list elements map each dimension of the locstream to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the locstream's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farrayPtr dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the LocStream dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular LocStream dimension will be replicating the Field across the DEs along this direction.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.42 ESMF_FieldEmptyComplete - Complete a Field from Mesh started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompMesh<rank><type><kind>(field, mesh, & 
   farray, indexflag, datacopyflag, meshloc, & 
   gridToFieldMap, ungriddedLBound, ungriddedUBound, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_Mesh), intent(in) :: mesh 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_MeshLoc), intent(in), optional :: meshloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farray.
mesh
The ESMF_Mesh object to complete the Field.
farray
Native Fortran data array to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Indicates whether to copy the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[meshloc]
Which part of the mesh to build the Field on. Can be set to either ESMF_MESHLOC_NODE or ESMF_MESHLOC_ELEMENT. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the mesh's dimCount. The list elements map each dimension of the mesh to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the mesh's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farray dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than Mesh dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than Mesh dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.43 ESMF_FieldEmptyComplete - Complete a Field from Mesh started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompMeshPtr<rank><type><kind>(field, mesh, & 
   farrayPtr, datacopyflag, meshloc, gridToFieldMap, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_Mesh), intent(in) :: mesh 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_MeshLoc), intent(in), optional :: meshloc 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farrayPtr.
mesh
The ESMF_Mesh object to complete the Field.
farrayPtr
Native Fortran data pointer to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Indicates whether to copy the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[meshloc]
Which part of the mesh to build the Field on. Can be set to either ESMF_MESHLOC_NODE or ESMF_MESHLOC_ELEMENT. If not set, defaults to ESMF_MESHLOC_NODE.
[gridToFieldMap]
List with number of elements equal to the mesh's dimCount. The list elements map each dimension of the mesh to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the mesh's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farrayPtr dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the Mesh dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular Mesh dimension will be replicating the Field across the DEs along this direction.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.44 ESMF_FieldEmptyComplete - Complete a Field from XGrid started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompXG<rank><type><kind>(field, xgrid, & 
   farray, indexflag, datacopyflag, xgridside, gridindex, & 
   gridToFieldMap, & 
   ungriddedLBound, ungriddedUBound, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_XGrid), intent(in) :: xgrid 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside 
   integer, intent(in), optional :: gridindex 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(in), optional :: ungriddedLBound(:) 
   integer, intent(in), optional :: ungriddedUBound(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farray.
xgrid
The ESMF_XGrid object to complete the Field.
farray
Native Fortran data array to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farray.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Indicates whether to copy the farray or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridSide is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the xgrid's dimCount. The list elements map each dimension of the xgrid to a dimension in the farray by specifying the appropriate farray dimension index. The default is to map all of the xgrid's dimensions against the lowest dimensions of the farray in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farray dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the XGrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular XGrid dimension will be replicating the Field across the DEs along this direction.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than XGrid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than XGrid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.45 ESMF_FieldEmptyComplete - Complete a Field from XGrid started with FieldEmptyCreate


INTERFACE:

   ! Private name; call using ESMF_FieldEmptyComplete() 
   subroutine ESMF_FieldEmptyCompXGPtr<rank><type><kind>(field, xgrid, & 
   farrayPtr, xgridside, gridindex, & 
   datacopyflag, gridToFieldMap, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   type(ESMF_XGrid), intent(in) :: xgrid 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside 
   integer, intent(in), optional :: gridindex 
   integer, intent(in), optional :: gridToFieldMap(:) 
   integer, intent(out), optional :: rc
DESCRIPTION:

This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.

The Fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the Fortran data pointer in this case. This gives user more flexibility over memory management.

The arguments are:

field
The ESMF_Field object to be completed and committed in this call. The field will have the same dimension (dimCount) as the rank of the farrayPtr.
xgrid
The ESMF_XGrid object to complete the Field.
farrayPtr
Native Fortran data pointer to be copied/referenced in the field. The field dimension (dimCount) will be the same as the dimCount for the farrayPtr.
[datacopyflag]
Indicates whether to copy the farrayPtr or reference it directly. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridside is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[gridToFieldMap]
List with number of elements equal to the xgrid's dimCount. The list elements map each dimension of the xgrid to a dimension in the farrayPtr by specifying the appropriate farrayPtr dimension index. The default is to map all of the xgrid's dimensions against the lowest dimensions of the farrayPtr in sequence, i.e. gridToFieldMap = (/1,2,3,.../). Unmapped farrayPtr dimensions are undistributed Field dimensions. All gridToFieldMap entries must be greater than or equal to zero and smaller than or equal to the Field dimCount. It is erroneous to specify the same entry multiple times unless it is zero. If the Field dimCount is less than the XGrid dimCount then the default gridToFieldMap will contain zeros for the rightmost entries. A zero entry in the gridToFieldMap indicates that the particular XGrid dimension will be replicating the Field across the DEs along this direction.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.46 ESMF_FieldEmptyCreate - Create an empty Field


INTERFACE:

   function ESMF_FieldEmptyCreate(name, vm, rc)
RETURN VALUE:
     type(ESMF_Field) :: ESMF_FieldEmptyCreate
ARGUMENTS:
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character (len = *), intent(in), optional :: name
     type(ESMF_VM), intent(in), optional :: vm
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

This version of ESMF_FieldCreate builds an empty ESMF_Field and depends on later calls to add an ESMF_Grid and ESMF_Array to it. The empty ESMF_Field can be completed in one more step or two more steps by the ESMF_FieldEmptySet and ESMF_FieldEmptyComplete methods. Attributes can be added to an empty Field object. For an example and associated documentation using this method see section 26.3.8 and 26.3.7.

The arguments are:

[name]
Field name.
[vm]
If present, the Field object is created on the specified ESMF_VM object. The default is to create on the VM of the current component context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.47 ESMF_FieldEmptySet - Set a Geom in an empty Field


INTERFACE:

   ! Private name; call using ESMF_FieldEmptySet()
   subroutine ESMF_FieldEmptySetGeom(field, geom, vm, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field
   type(ESMF_Geom), intent(in) :: geom
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_VM), intent(in), optional :: vm
   integer, intent(out), optional :: rc
DESCRIPTION:

Set a geom in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_Geom internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.

For an example and associated documentation using this method see section 26.3.7.

The arguments are:

field
Empty ESMF_Field. After this operation, the ESMF_Field contains the ESMF_Grid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET.
geom
ESMF_Geom to be set in the ESMF_Field.
[vm]
If present, the Field object will only be accessed, and the Grid object set, on those PETs contained in the specified ESMF_VM object. The default is to assume the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.48 ESMF_FieldEmptySet - Set a Grid in an empty Field


INTERFACE:

   ! Private name; call using ESMF_FieldEmptySet()
   subroutine ESMF_FieldEmptySetGrid(field, grid, StaggerLoc, &
     vm, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field
   type(ESMF_Grid), intent(in) :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_STAGGERLOC), intent(in), optional :: StaggerLoc
   type(ESMF_VM), intent(in), optional :: vm
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Set a grid and an optional staggerloc (default to center stagger ESMF_STAGGERLOC_CENTER) in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_Grid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.

For an example and associated documentation using this method see section 26.3.7.

The arguments are:

field
Empty ESMF_Field. After this operation, the ESMF_Field contains the ESMF_Grid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET.
grid
ESMF_Grid to be set in the ESMF_Field.
[StaggerLoc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. To create a custom stagger location see section 31.3.25. The default value is ESMF_STAGGERLOC_CENTER.
[vm]
If present, the Field object will only be accessed, and the Grid object set, on those PETs contained in the specified ESMF_VM object. The default is to assume the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.49 ESMF_FieldEmptySet - Set a Mesh in an empty Field


INTERFACE:

   ! Private name; call using ESMF_FieldEmptySet()
   subroutine ESMF_FieldEmptySetMesh(field, mesh, indexflag, meshloc, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field
   type(ESMF_Mesh), intent(in) :: mesh
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_Index_Flag),intent(in), optional :: indexflag
   type(ESMF_MeshLoc), intent(in), optional :: meshloc
   integer, intent(out), optional :: rc
DESCRIPTION:

Set a mesh and an optional meshloc (default to center stagger ESMF_MESHLOC_NODE) in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_Mesh internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.

The arguments are:

field
Empty ESMF_Field. After this operation, the ESMF_Field contains the ESMF_Mesh internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET.
mesh
ESMF_Mesh to be set in the ESMF_Field.
[indexflag]
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[meshloc]
Which part of the mesh to build the Field on. Can be set to either ESMF_MESHLOC_NODE or ESMF_MESHLOC_ELEMENT. If not set, defaults to ESMF_MESHLOC_NODE.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.50 ESMF_FieldEmptySet - Set a LocStream in an empty Field


INTERFACE:

   ! Private name; call using ESMF_FieldEmptySet()
   subroutine ESMF_FieldEmptySetLocStream(field, locstream, &
     vm, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field
   type(ESMF_LocStream), intent(in) :: locstream
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_VM), intent(in), optional :: vm
   integer, intent(out), optional :: rc
DESCRIPTION:

Set a ESMF_LocStream in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_LocStream internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.

The arguments are:

field
Empty ESMF_Field. After this operation, the ESMF_Field contains the ESMF_LocStream internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET.
locstream
ESMF_LocStream to be set in the ESMF_Field.
[vm]
If present, the Field object will only be accessed, and the Grid object set, on those PETs contained in the specified ESMF_VM object. The default is to assume the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.51 ESMF_FieldEmptySet - Set an XGrid in an empty Field


INTERFACE:

   ! Private name; call using ESMF_FieldEmptySet()
   subroutine ESMF_FieldEmptySetXGrid(field, xgrid, xgridside, gridindex, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field
   type(ESMF_XGrid), intent(in) :: xgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside
   integer, intent(in), optional :: gridindex
   integer, intent(out), optional :: rc
DESCRIPTION:

Set a xgrid and optional xgridside (default to balanced side ESMF_XGRIDSIDE_Balanced) and gridindex (default to 1) in a non-complete ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_XGrid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.

The arguments are:

field
Empty ESMF_Field. After this operation, the ESMF_Field contains the ESMF_XGrid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET.
xgrid
ESMF_XGrid to be set in the ESMF_Field.
[xgridside]
Side of XGrid to retrieve a DistGrid. For valid predefined values see section 34.2.1. The default value is ESMF_XGRIDSIDE_BALANCED.
[gridindex]
Index to specify which DistGrid when on side A or side B. The default value is 1.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.52 ESMF_FieldFill - Fill data into a Field


INTERFACE:

   subroutine ESMF_FieldFill(field, dataFillScheme, &
     const1, member, step, &
     param1I4, param2I4, param3I4, &
     param1R4, param2R4, param3R4, &
     param1R8, param2R8, param3R8, &
     rc)
ARGUMENTS:
     type(ESMF_Field), intent(inout) :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(len=*), intent(in), optional :: dataFillScheme
     real(ESMF_KIND_R8), intent(in), optional :: const1
     integer, intent(in), optional :: member
     integer, intent(in), optional :: step
     integer(ESMF_KIND_I4), intent(in), optional :: param1I4
     integer(ESMF_KIND_I4), intent(in), optional :: param2I4
     integer(ESMF_KIND_I4), intent(in), optional :: param3I4
     real(ESMF_KIND_R4), intent(in), optional :: param1R4
     real(ESMF_KIND_R4), intent(in), optional :: param2R4
     real(ESMF_KIND_R4), intent(in), optional :: param3R4
     real(ESMF_KIND_R8), intent(in), optional :: param1R8
     real(ESMF_KIND_R8), intent(in), optional :: param2R8
     real(ESMF_KIND_R8), intent(in), optional :: param3R8
     integer, intent(out), optional :: rc
DESCRIPTION:

Fill field with data according to dataFillScheme. Depending on the chosen fill scheme, the member and step arguments are used to provide differing fill data patterns.

The arguments are:

field
The ESMF_Field object to fill with data.
[dataFillScheme]
The fill scheme. The available options are "sincos", "one", "const", "random", "nan", and "snan". Defaults to "sincos".
[const1]
Constant of real type. Defaults to 0.
[member]
Member incrementor. Defaults to 1.
[step]
Step incrementor. Defaults to 1.
[param1I4]
Optional parameter of typekind I4. The default depends on the specified dataFillScheme.
[param2I4]
Optional parameter of typekind I4. The default depends on the specified dataFillScheme.
[param3I4]
Optional parameter of typekind I4. The default depends on the specified dataFillScheme.
[param1R4]
Optional parameter of typekind R4. The default depends on the specified dataFillScheme.
[param2R4]
Optional parameter of typekind R4. The default depends on the specified dataFillScheme.
[param3R4]
Optional parameter of typekind R4. The default depends on the specified dataFillScheme.
[param1R8]
Optional parameter of typekind R8. The default depends on the specified dataFillScheme.
[param2R8]
Optional parameter of typekind R8. The default depends on the specified dataFillScheme.
[param3R8]
Optional parameter of typekind R8. The default depends on the specified dataFillScheme.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.53 ESMF_FieldGather - Gather a Fortran array from an ESMF_Field


INTERFACE:

   subroutine ESMF_FieldGather<rank><type><kind>(field, farray, & 
   rootPet, tile, vm, rc)
ARGUMENTS:
   type(ESMF_Field), intent(in) :: field 
   <type>(ESMF_KIND_<kind>), intent(out), target :: farray(<rank>) 
   integer, intent(in) :: rootPet 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(in), optional :: tile 
   type(ESMF_VM), intent(in), optional :: vm 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Gather the data of an ESMF_Field object into the farray located on rootPET. A single DistGrid tile of array must be gathered into farray. The optional tile argument allows selection of the tile. For Fields defined on a single tile DistGrid the default selection (tile 1) will be correct. The shape of farray must match the shape of the tile in Field.

If the Field contains replicating DistGrid dimensions data will be gathered from the numerically higher DEs. Replicated data elements in numericaly lower DEs will be ignored.

The implementation of Scatter and Gather is not sequence index based. If the Field is built on arbitrarily distributed Grid, Mesh, LocStream or XGrid, Gather will not gather data to rootPet from source data points corresponding to the sequence index on rootPet. Instead Gather will gather a contiguous memory range from source PET to rootPet. The size of the memory range is equal to the number of data elements on the source PET. Vice versa for the Scatter operation. In this case, the user should use ESMF_FieldRedist to achieve the same data operation result. For examples how to use ESMF_FieldRedist to perform Gather and Scatter, please refer to 26.3.32 and 26.3.31.

This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.

For examples and associated documentation regarding this method see Section 26.3.28.

The arguments are:

field
The ESMF_Field object from which data will be gathered.
{farray}
The Fortran array into which to gather data. Only root must provide a valid farray, the other PETs may treat farray as an optional argument.
rootPet
PET that holds the valid destination array, i.e. farray.
[tile]
The DistGrid tile in field from which to gather farray. By default farray will be gathered from tile 1.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.54 ESMF_FieldGet - Get object-wide Field information


INTERFACE:

   ! Private name; call using ESMF_FieldGet()
   subroutine ESMF_FieldGetDefault(field, arrayspec, status, &
     geomtype, geom, grid, mesh, locstream, xgrid, array, localarrayList, &
     typekind, rank, dimCount, geomDimCount, ungriddedDimCount, &
     replicatedDimCount, staggerloc, meshloc, xgridside, gridindex, &
     gridToFieldMap, ungriddedLBound, ungriddedUBound, &
     totalLWidth, totalUWidth, localDeCount, ssiLocalDeCount, &
     localDeToDeMap, minIndex, maxIndex, elementCount, &
     localMinIndex, localMaxIndex, localElementCount, isESMFAllocated, &
     name, vm, rc)
ARGUMENTS:
     type(ESMF_Field), intent(in) :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_ArraySpec), intent(out), optional :: arrayspec
     type(ESMF_FieldStatus_Flag),intent(out), optional :: status
     type(ESMF_GeomType_Flag), intent(out), optional :: geomtype
     type(ESMF_Geom), intent(out), optional :: geom
     type(ESMF_Grid), intent(out), optional :: grid
     type(ESMF_Mesh), intent(out), optional :: mesh
     type(ESMF_LocStream), intent(out), optional :: locstream
     type(ESMF_XGrid), intent(out), optional :: xgrid
     type(ESMF_Array), intent(out), optional :: array
     type(ESMF_LocalArray), target, intent(out), optional :: localarrayList(:)
     type(ESMF_TypeKind_Flag), intent(out), optional :: typekind
     integer, intent(out), optional :: rank
     integer, intent(out), optional :: dimCount
     integer, intent(out), optional :: geomDimCount
     integer, intent(out), optional :: ungriddedDimCount
     integer, intent(out), optional :: replicatedDimCount
     type(ESMF_StaggerLoc), intent(out), optional :: staggerloc
     type(ESMF_MeshLoc), intent(out), optional :: meshloc
     type(ESMF_XGridSide_Flag), intent(out), optional :: xgridside
     integer, intent(out), optional :: gridindex
     integer, intent(out), optional :: gridToFieldMap(:)
     integer, intent(out), optional :: ungriddedLBound(:)
     integer, intent(out), optional :: ungriddedUBound(:)
     integer, intent(out), optional :: totalLWidth(:,:)
     integer, intent(out), optional :: totalUWidth(:,:)
     integer, intent(out), optional :: localDeCount
     integer, intent(out), optional :: ssiLocalDeCount
     integer, intent(out), optional :: localDeToDeMap(:)
     integer, intent(out), optional :: minIndex(:)
     integer, intent(out), optional :: maxIndex(:)
     integer, intent(out), optional :: elementCount(:)
     integer, intent(out), optional :: localMinIndex(:)
     integer, intent(out), optional :: localMaxIndex(:)
     integer, intent(out), optional :: localElementCount(:)
     logical, intent(out), optional :: isESMFAllocated
     character(len=*), intent(out), optional :: name
     type(ESMF_VM), intent(out), optional :: vm
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Query an ESMF_Field object for various pieces of information. All arguments after the field argument are optional. To select individual items use the named_argument=value syntax. For an example and associated documentation using this method see section 26.3.3.

The arguments are:

field
ESMF_Field object to query.
[arrayspec]
ESMF_ArraySpec object containing the type/kind/rank information of the Field object.
[status]
The status of the Field. See section 26.2.1 for a complete list of values.
[geomtype]
The type of geometry on which the Field is built. See section 54.23 for the range of values.
[geom]
ESMF_Geom.
[grid]
ESMF_Grid.
[mesh]
STATUS:This argument is excluded from the backward compatibility statement.
ESMF_Mesh.
[locstream]
STATUS:This argument is excluded from the backward compatibility statement.
ESMF_LocStream.
[xgrid]
STATUS:This argument is excluded from the backward compatibility statement.
ESMF_XGrid.
[array]
ESMF_Array.
[localarrayList]
Upon return this holds a list of the associated ESMC_LocalArray objects. localarrayList must be allocated to be of size localDeCount or ssiLocalDeCount.
[typekind]
TypeKind specifier for Field. See section 54.61 for a complete list of values.
[rank]
Number of dimensions in the physical memory of the field data. It is identical to dimCount when the corresponding grid is a non-arbitrary grid. It is less than dimCount when the grid is arbitrarily distributed. For an detailed discussion of this parameter, please see section 26.3.23 and section 26.3.24.
[dimCount]
Number of geometrical dimensions in field. For an detailed discussion of this parameter, please see section 26.3.23 and section 26.3.24.
[geomDimCount]
Number of dimensions in the underlying Geom object.
[ungriddedDimCount]
Number of ungridded dimensions in the Field object.
[replicatedDimCount]
Number of replicated dimensions in the Field object.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values and interpretation of results see section 31.2.6.
[meshloc]
STATUS:This argument is excluded from the backward compatibility statement.
The part of the mesh to build the Field on. Can be either ESMF_MESHLOC_NODE or ESMF_MESHLOC_ELEMENT. If not set, defaults to ESMF_MESHLOC_NODE.
[xgridside]
STATUS:This argument is excluded from the backward compatibility statement.
The side of the XGrid that the Field was created on. See section 34.2.1 for a complete list of values.
[gridIndex]
STATUS:This argument is excluded from the backward compatibility statement.
If xgridside is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid/Mesh on that side the Field was created on.
[gridToFieldMap]
List with number of elements equal to the grid's dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid's dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound argument must be equal or greater to ungriddedDimCount, the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. Note that the ordering of the ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound argument must be equal or greater to ungriddedDimCount, the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. Note that the ordering of the ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of the first dimension of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ). The size of the 2nd dimension of this array is localDeCount.
[totalUWidth]
Upper bound of halo region. The size of the first dimension of this array is the number of gridded dimensions in the field. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ). The size of the 2nd dimension of this array is localDeCount.
[localDeCount]
Upon return this holds the number of PET-local DEs defined in the DELayout associated with the Field object.
[ssiLocalDeCount]
The number of DEs in the Field available to the local PET. This includes DEs that are local to other PETs on the same SSI, that are accessible via shared memory.
[localDeToDeMap]
Mapping between localDe indices and the (global) DEs associated with the local PET. The localDe index variables are discussed in sections 50.3.7 and 28.2.5. The provided actual argument must be of size localDeCount, or ssiLocalDeCount, and will be filled accordingly.
[minIndex]
Upon return this holds the global lower bounds of the field data across all PETs. This information will be identical across all PETs. minIndex must be allocated to be of size equal to the field rank.
[maxIndex]
Upon return this holds the global upper bounds of the field data across all PETs. This information will be identical across all PETs. maxIndex must be allocated to be of size equal to the field rank.
[elementCount]
Upon return this holds the global number of items of the field data across all PETs. This information will be identical across all PETs. elementCount must be allocated to be of size equal to the field rank.
[localMinIndex]
Upon return this holds the PET local lower bounds globally indexed of the field data. localMinIndex must be allocated to be of size equal to the field rank.
[localMaxIndex]
Upon return this holds the PET local upper bounds globally indexed of the field data. localMaxIndex must be allocated to be of size equal to the field rank.
[localElementCount]
Upon return this holds the PET local number of items of the field data. localElementCount must be allocated to be of size equal to the field rank.
[isESMFAllocated]
Set to .true. for data allocations held by ESMF. Set to .false. otherwise.
[name]
Name of the Field object.
[vm]
The VM on which the Field object was created.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.55 ESMF_FieldGet - Get a DE-local Fortran array pointer from a Field


INTERFACE:

   ! Private name; call using ESMF_FieldGet() 
   subroutine ESMF_FieldGetDataPtr<rank><type><kind>(field, localDe, & 
   farrayPtr, exclusiveLBound, exclusiveUBound, exclusiveCount, & 
   computationalLBound, computationalUBound, computationalCount, & 
   totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
   type(ESMF_Field), intent(in) :: field 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(in), optional :: localDe 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
   integer, intent(out), optional :: exclusiveLBound(:) 
   integer, intent(out), optional :: exclusiveUBound(:) 
   integer, intent(out), optional :: exclusiveCount(:) 
   integer, intent(out), optional :: computationalLBound(:) 
   integer, intent(out), optional :: computationalUBound(:) 
   integer, intent(out), optional :: computationalCount(:) 
   integer, intent(out), optional :: totalLBound(:) 
   integer, intent(out), optional :: totalUBound(:) 
   integer, intent(out), optional :: totalCount(:) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get a Fortran pointer to DE-local memory allocation within field. For convenience DE-local bounds can be queried at the same time. For an example and associated documentation using this method see section 26.3.2.

The arguments are:

field
ESMF_Field object.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
farrayPtr
Fortran array pointer which will be pointed at DE-local memory allocation. It depends on the specific entry point of ESMF_FieldCreate() used during field creation, which Fortran operations are supported on the returned farrayPtr. See 26.4 for more details.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[exclusiveCount]
Upon return this holds the number of items, exclusiveUBound-exclusiveLBound+1, in the exclusive region per dimension. exclusiveCount must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the computational region. computationalLBound must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the lower bounds of the computational region. computationalUBound must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[totalLBound]
Upon return this holds the lower bounds of the total region. totalLBound must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[totalUBound]
Upon return this holds the lower bounds of the total region. totalUBound must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[totalCount]
Upon return this holds the number of items in the total region per dimension (i.e. totalUBound-totalLBound+1). computationalCount must be allocated to be of size equal to field's dimCount. See section 28.2.6 for a description of the regions and their associated bounds and counts.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.56 ESMF_FieldGetBounds - Get DE-local Field data bounds


INTERFACE:

   ! Private name; call using ESMF_FieldGetBounds()
   subroutine ESMF_FieldGetBounds(field, localDe, &
     exclusiveLBound, exclusiveUBound, exclusiveCount, computationalLBound, &
     computationalUBound, computationalCount, totalLBound, &
     totalUBound, totalCount, rc)
ARGUMENTS:
     type(ESMF_Field), intent(in) :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: localDe
     integer, intent(out), optional :: exclusiveLBound(:)
     integer, intent(out), optional :: exclusiveUBound(:)
     integer, intent(out), optional :: exclusiveCount(:)
     integer, intent(out), optional :: computationalLBound(:)
     integer, intent(out), optional :: computationalUBound(:)
     integer, intent(out), optional :: computationalCount(:)
     integer, intent(out), optional :: totalLBound(:)
     integer, intent(out), optional :: totalUBound(:)
     integer, intent(out), optional :: totalCount(:)
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method returns the bounds information of a field that consists of a internal grid and a internal array. The exclusive and computational bounds are shared between the grid and the array but the total bounds are the array bounds plus the halo width. The count is the number of elements between each bound pair.

The arguments are:

field
Field to get the information from.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to the field rank. Please see section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to the field rank. Please see section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveCount]
Upon return this holds the number of items, exclusiveUBound-exclusiveLBound+1, in the exclusive region per dimension. exclusiveCount must be allocated to be of size equal to the field rank. Please see section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the stagger region. computationalLBound must be allocated to be of size equal to the field rank. Please see section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the upper bounds of the stagger region. computationalUBound must be allocated to be of size equal to the field rank. Please see section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to the field rank. Please see section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalLBound]
Upon return this holds the lower bounds of the total region. totalLBound must be allocated to be of size equal to the field rank.
[totalUBound]
Upon return this holds the upper bounds of the total region. totalUBound must be allocated to be of size equal to the field rank.
[totalCount]
Upon return this holds the number of items in the total region per dimension (i.e. totalUBound-totalLBound+1). totalCount must be allocated to be of size equal to the field rank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.57 ESMF_FieldHalo - Execute a FieldHalo operation


INTERFACE:

   subroutine ESMF_FieldHalo(field, routehandle,  &
                             routesyncflag, finishedflag, checkflag, rc)
ARGUMENTS:
     type(ESMF_Field),          intent(inout)          :: field
     type(ESMF_RouteHandle),    intent(inout)          :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_RouteSync_Flag), intent(in),  optional  :: routesyncflag
     logical,                   intent(out), optional  :: finishedflag
     logical,                   intent(in),  optional  :: checkflag
     integer,                   intent(out), optional  :: rc
STATUS:

DESCRIPTION:

Execute a precomputed Field halo operation for field. The field argument must match the Field used during ESMF_FieldHaloStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

See ESMF_FieldHaloStore() on how to precompute routehandle.

This call is collective across the current VM.

field
ESMF_Field containing data to be haloed.
routehandle
Handle to the precomputed Route.
[routesyncflag]
Indicate communication option. Default is ESMF_ROUTESYNC_BLOCKING, resulting in a blocking operation. See section 54.53 for a complete list of valid settings.
[finishedflag]
Used in combination with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH. Returned finishedflag equal to .true. indicates that all operations have finished. A value of .false. indicates that there are still unfinished operations that require additional calls with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH, or a final call with routesyncflag = ESMF_ROUTESYNC_NBWAITFINISH. For all other routesyncflag settings the returned value in finishedflag is always .true..
[checkflag]
If set to .TRUE. the input Field pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.58 ESMF_FieldHaloRelease - Release resources associated with a Field halo operation


INTERFACE:

   subroutine ESMF_FieldHaloRelease(routehandle, noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional  :: noGarbage
     integer,                intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

Release resources associated with a Field halo operation. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.59 ESMF_FieldHaloStore - Store a FieldHalo operation


INTERFACE:

     subroutine ESMF_FieldHaloStore(field, routehandle,  &
       startregion, haloLDepth, haloUDepth, rc)
ARGUMENTS:
     type(ESMF_Field),            intent(inout)           :: field
     type(ESMF_RouteHandle),      intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_StartRegion_Flag), intent(in),    optional :: startregion
     integer,                     intent(in),    optional :: haloLDepth(:)
     integer,                     intent(in),    optional :: haloUDepth(:)
     integer,                     intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Store a Field halo operation over the data in field. By default, i.e. without specifying startregion, haloLDepth and haloUDepth, all elements in the total Field region that lie outside the exclusive region will be considered potential destination elements for halo. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.

Specifying startregion allows to change the shape of the effective halo region from the inside. Setting this flag to ESMF_STARTREGION_COMPUTATIONAL means that only elements outside the computational region of the Field are considered for potential destination elements for the halo operation. The default is ESMF_STARTREGION_EXCLUSIVE.

The haloLDepth and haloUDepth arguments allow to reduce the extent of the effective halo region. Starting at the region specified by startregion, the haloLDepth and haloUDepth define a halo depth in each direction. Note that the maximum halo region is limited by the total Field region, independent of the actual haloLDepth and haloUDepth setting. The total Field region is local DE specific. The haloLDepth and haloUDepth are interpreted as the maximum desired extent, reducing the potentially larger region available for the halo operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldHalo() on any Field that matches field in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

field
ESMF_Field containing data to be haloed. The data in this Field may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[startregion]
The start of the effective halo region on every DE. The default setting is ESMF_STARTREGION_EXCLUSIVE, rendering all non-exclusive elements potential halo destination elements. See section 54.56 for a complete list of valid settings.
[haloLDepth]
This vector specifies the lower corner of the effective halo region with respect to the lower corner of startregion. The size of haloLDepth must equal the number of distributed Array dimensions.
[haloUDepth]
This vector specifies the upper corner of the effective halo region with respect to the upper corner of startregion. The size of haloUDepth must equal the number of distributed Array dimensions.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.60 ESMF_FieldIsCreated - Check whether a Field object has been created


INTERFACE:

   function ESMF_FieldIsCreated(field, rc)
RETURN VALUE:
     logical :: ESMF_FieldIsCreated
ARGUMENTS:
     type(ESMF_Field), intent(in)            :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,             intent(out), optional :: rc
DESCRIPTION:

Return .true. if the field has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

field
ESMF_Field queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.61 ESMF_FieldPrint - Print Field information


INTERFACE:

   subroutine ESMF_FieldPrint(field, rc)
ARGUMENTS:
     type(ESMF_Field), intent(in)            :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,          intent(out), optional :: rc
STATUS:

DESCRIPTION:

Prints information about the field to stdout. This subroutine goes through the internal data members of a field data type and prints information of each data member.

The arguments are:

field
An ESMF_Field object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.62 ESMF_FieldRead - Read Field data from a file


INTERFACE:

   subroutine ESMF_FieldRead(field, fileName,        &
       variableName, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_Field),      intent(inout)          :: field
     character(*),          intent(in)             :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(*),          intent(in),  optional  :: variableName
     integer,               intent(in),  optional  :: timeslice
     type(ESMF_IOFmt_Flag), intent(in),  optional  :: iofmt
     integer,               intent(out), optional  :: rc
DESCRIPTION:

Read Field data from a file and put it into an ESMF_Field object. For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

Limitations:

The arguments are:

field
The ESMF_Field object in which the read data is returned.
fileName
The name of the file from which Field data is read. If the Field is a multi-tile Array, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being read from a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be read from "myfile1.nc", tile 2 from "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[variableName]
Variable name in the file; default is the "name" of Field. Use this argument only in the I/O format (such as NetCDF) that supports variable name. If the I/O format does not support this (such as binary format), ESMF will return an error code.
[timeslice]
Number of slices to be read from file, starting from the 1st slice
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.63 ESMF_FieldRedist - Execute a Field redistribution


INTERFACE:

   subroutine ESMF_FieldRedist(srcField, dstField, routehandle,  &
     checkflag, rc)
ARGUMENTS:
         type(ESMF_Field),       intent(in),optional     :: srcField
         type(ESMF_Field),       intent(inout),optional  :: dstField
         type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical,                intent(in),   optional  :: checkflag
         integer,                intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

Execute a precomputed Field redistribution from srcField to dstField. Both srcField and dstField must match the respective Fields used during ESMF_FieldRedistStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcField and dstField arguments are optional in support of the situation where srcField and/or dstField are not defined on all PETs. The srcField and dstField must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

See ESMF_FieldRedistStore() on how to precompute routehandle.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.30.

[srcField]
ESMF_Field with source data.
[dstField]
ESMF_Field with destination data.
routehandle
Handle to the precomputed Route.
[checkflag]
If set to .TRUE. the input Field pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.64 ESMF_FieldRedistRelease - Release resources associated with Field redistribution


INTERFACE:

   subroutine ESMF_FieldRedistRelease(routehandle, noGarbage, rc)
ARGUMENTS:
         type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical,                intent(in),   optional  :: noGarbage
         integer,                intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

Release resources associated with a Field redistribution. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.65 ESMF_FieldRedistStore - Precompute Field redistribution with a local factor argument


INTERFACE:

   ! Private name; call using ESMF_FieldRedistStore() 
   subroutine ESMF_FieldRedistStore<type><kind>(srcField, dstField, & 
          routehandle, factor, srcToDstTransposeMap, &
          ignoreUnmatchedIndices, rc)
ARGUMENTS:
     type(ESMF_Field),         intent(in)            :: srcField  
     type(ESMF_Field),         intent(inout)         :: dstField  
     type(ESMF_RouteHandle),   intent(inout)         :: routehandle
     <type>(ESMF_KIND_<kind>), intent(in)            :: factor 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                  intent(in),  optional :: srcToDstTransposeMap(:) 
     logical,                  intent(in),  optional :: ignoreUnmatchedIndices
     integer,                  intent(out), optional :: rc
STATUS:

DESCRIPTION:

ESMF_FieldRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_FieldRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_FieldRedistStore() method, as provided through the separate entry points shown in 26.6.65 and 26.6.66, is described in the following paragraphs as a whole.

Store a Field redistribution operation from srcField to dstField. Interface 26.6.65 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 26.6.66. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.

Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source Field, destination Field, and the factor may be of different <type><kind>. Further, source and destination Fields may differ in shape, however, the number of elements must match.

The default redistribution operation, when srcToDstTransposeMap is not specified, corresponds to the identity mapping: each element of the sequentialized source Field is copied to the sequentialized destination Field element in order.

If the srcToDstTransposeMap argument is provided it must be identical across all PETs. The srcToDstTransposeMap allows source and destination Field dimensions to be transposed during the redistribution. To support this option, the number of source and destination Field dimensions must be equal and the size of the associated dimensions must match. See section 28.2.17 for more details about the use of the srcToDstTransposeMap argument.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldRedist() on any pair of Fields that matches srcField and dstField in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.30.

The arguments are:

srcField
ESMF_Field with source data.
dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.
routehandle
Handle to the precomputed Route.
factor
Factor by which to multiply data. Default is 1. See full method description above for details on the interplay with other PETs.
[srcToDstTransposeMap]
A list with as many entries as there are dimensions in srcField, or tileCount times this many entries. Each entry maps the corresponding srcField dimension against the specified dstField dimension. Mixing distributed and undistributed dimensions is supported. Negative entries reverse the order of elements along the specified dimension when going from source to destination. When providing $rank \times tileCount$ elements in srcToDstTransposeMap, each block of size rank is associated with the corresponding tile (in order), and interpreted as the tile-specific transpose map.
[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when not all elements match between the srcField and dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores unmatched indices.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.66 ESMF_FieldRedistStore - Precompute Field redistribution without a local factor argument


INTERFACE:

   ! Private name; call using ESMF_FieldRedistStore() 
     subroutine ESMF_FieldRedistStoreNF(srcField, dstField, & 
         routehandle, srcToDstTransposeMap, &
         ignoreUnmatchedIndices, rc)
ARGUMENTS:
         type(ESMF_Field),       intent(in)            :: srcField  
         type(ESMF_Field),       intent(inout)         :: dstField  
         type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         integer,                intent(in), optional  :: srcToDstTransposeMap(:) 
         logical,                intent(in), optional  :: ignoreUnmatchedIndices
         integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

ESMF_FieldRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_FieldRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_FieldRedistStore() method, as provided through the separate entry points shown in 26.6.65 and 26.6.66, is described in the following paragraphs as a whole.

Store a Field redistribution operation from srcField to dstField. Interface 26.6.65 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 26.6.66. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.

Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source Field, destination Field, and the factor may be of different <type><kind>. Further, source and destination Fields may differ in shape, however, the number of elements must match.

The default redistribution operation, when srcToDstTransposeMap is not specified, corresponds to the identity mapping: each element of the sequentialized source Field is copied to the sequentialized destination Field element in order.

If the srcToDstTransposeMap argument is provided it must be identical across all PETs. The srcToDstTransposeMap allows source and destination Field dimensions to be transposed during the redistribution. To support this option, the number of source and destination Field dimensions must be equal and the size of the associated dimensions must match. See section 28.2.17 for more details about the use of the srcToDstTransposeMap argument.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldRedist() on any pair of Fields that matches srcField and dstField in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.30.

The arguments are:

srcField
ESMF_Field with source data.
dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[srcToDstTransposeMap]
A list with as many entries as there are dimensions in srcField, or tileCount times this many entries. Each entry maps the corresponding srcField dimension against the specified dstField dimension. Mixing distributed and undistributed dimensions is supported. Negative entries reverse the order of elements along the specified dimension when going from source to destination. When providing $rank \times tileCount$ elements in srcToDstTransposeMap, each block of size rank is associated with the corresponding tile (in order), and interpreted as the tile-specific transpose map.
[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when not all elements match between the srcField and dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores unmatched indices.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.67 ESMF_FieldRegrid - Compute a regridding operation


INTERFACE:

   subroutine ESMF_FieldRegrid(srcField, dstField, routehandle, &
     zeroregion, termorderflag, checkflag, dynamicMask, rc)
ARGUMENTS:
       type(ESMF_Field),               intent(in),    optional :: srcField
       type(ESMF_Field),               intent(inout), optional :: dstField
       type(ESMF_RouteHandle),         intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_Region_Flag),         intent(in),    optional :: zeroregion
       type(ESMF_TermOrder_Flag),      intent(in),    optional :: termorderflag
       logical,                        intent(in),    optional :: checkflag
       type(ESMF_DynamicMask), target, intent(in),    optional :: dynamicMask
       integer,                        intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Execute the precomputed regrid operation stored in routehandle to interpolate from srcField to dstField. See ESMF_FieldRegridStore() on how to precompute the routehandle.

Both srcField and dstField must match the respective Fields used during ESMF_FieldRegridStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcField and dstField arguments are optional in support of the situation where srcField and/or dstField are not defined on all PETs. The srcField and dstField must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

This call is collective across the current VM.

[srcField]
ESMF_Field with source data.
[dstField]
ESMF_Field with destination data.
routehandle
Handle to the precomputed Route.
[zeroregion]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in dstField will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in dstField will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination Array that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings.
[termorderflag]
Specifies the order of the source side terms in all of the destination sums. The termorderflag only affects the order of terms during the execution of the RouteHandle. See the 37.2.1 section for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods. See 54.60 for a full list of options. The default setting depends on whether the dynamicMask argument is present or not. With dynamicMask argument present, the default of termorderflag is ESMF_TERMORDER_SRCSEQ. This ensures that all source terms are present on the destination side, and the interpolation can be calculated as a single sum. When dynamicMask is absent, the default of termorderflag is ESMF_TERMORDER_FREE, allowing maximum flexibility and partial sums for optimum performance.
[checkflag]
If set to .TRUE. the input Array pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[dynamicMask]
Object holding dynamic masking information. See section 37.2.6 for a discussion of dynamic masking.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.68 ESMF_FieldRegridRelease - Free resources used by a regridding operation


INTERFACE:

       subroutine ESMF_FieldRegridRelease(routehandle, &
         noGarbage, rc)
ARGUMENTS:
       type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical,                intent(in),  optional :: noGarbage
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with a regrid operation. After this call routehandle becomes invalid.

The arguments are:

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.69 ESMF_FieldRegridStore - Precompute a Field regridding operation and return a RouteHandle and weights


INTERFACE:

   !   Private name; call using ESMF_FieldRegridStore()
       subroutine ESMF_FieldRegridStoreNX(srcField, dstField, &
                     srcMaskValues, dstMaskValues, &
                     regridmethod, &
                     polemethod, regridPoleNPnts, & 
                     lineType, &
                     normType, &
                     vectorRegrid, & 
                     extrapMethod, &
                     extrapNumSrcPnts, &
                     extrapDistExponent, &
                     extrapNumLevels, &
                     unmappedaction, ignoreDegenerate, &
                     srcTermProcessing, & 
                     pipeLineDepth, &
                     routehandle, &
                     factorList, factorIndexList, & 
                     weights, indices, &  ! DEPRECATED ARGUMENTS
                     srcFracField, dstFracField, &
                     dstStatusField, &
                     unmappedDstList, &
                     checkFlag, &
                     rc)
ARGUMENTS:
       type(ESMF_Field),               intent(in)              :: srcField
       type(ESMF_Field),               intent(inout)           :: dstField
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer(ESMF_KIND_I4),          intent(in),    optional :: srcMaskValues(:)
       integer(ESMF_KIND_I4),          intent(in),    optional :: dstMaskValues(:)
       type(ESMF_RegridMethod_Flag),   intent(in),    optional :: regridmethod
       type(ESMF_PoleMethod_Flag),     intent(in),    optional :: polemethod
       integer,                        intent(in),    optional :: regridPoleNPnts
       type(ESMF_LineType_Flag),       intent(in),    optional :: lineType
       type(ESMF_NormType_Flag),       intent(in),    optional :: normType
       logical,                        intent(in),    optional :: vectorRegrid
       type(ESMF_ExtrapMethod_Flag),   intent(in),    optional :: extrapMethod
       integer,                        intent(in),    optional :: extrapNumSrcPnts
       real(ESMF_KIND_R4),             intent(in),    optional :: extrapDistExponent
       integer,                        intent(in),    optional :: extrapNumLevels
       type(ESMF_UnmappedAction_Flag), intent(in),    optional :: unmappedaction
       logical,                        intent(in),    optional :: ignoreDegenerate
       integer,                        intent(inout), optional :: srcTermProcessing
       integer,                        intent(inout), optional :: pipeLineDepth
       type(ESMF_RouteHandle),         intent(inout), optional :: routehandle
       real(ESMF_KIND_R8),             pointer,       optional :: factorList(:)
       integer(ESMF_KIND_I4),          pointer,       optional :: factorIndexList(:,:)
       real(ESMF_KIND_R8),    pointer, optional :: weights(:)   ! DEPRECATED ARG
       integer(ESMF_KIND_I4), pointer, optional :: indices(:,:) ! DEPRECATED ARG
       type(ESMF_Field),               intent(inout), optional :: srcFracField
       type(ESMF_Field),               intent(inout), optional :: dstFracField
       type(ESMF_Field),               intent(inout), optional :: dstStatusField
       integer(ESMF_KIND_I4),          pointer,       optional :: unmappedDstList(:)
       logical,                        intent(in),    optional :: checkFlag
       integer,                        intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Creates a sparse matrix operation (stored in routehandle) that contains the calculations and communications necessary to interpolate from srcField to dstField. The routehandle can then be used in the call ESMF_FieldRegrid() to interpolate between the Fields. The user may also get the interpolation matrix in sparse matrix form via the optional arguments factorList and factorIndexList.

The routehandle generated by this call is based just on the coordinates in the spatial class (e.g. Grid) contained in the Fields. If those coordinates don't change the routehandle can be used repeatedly to interpolate from the source Field to the destination Field. This is true even if the data in the Fields changes. The routehandle may also be used to interpolate between any source and destination Field which are created on the same location in the same Grid, LocStream, XGrid, or Mesh as the original Fields.

When it's no longer needed the routehandle should be destroyed by using ESMF_FieldRegridRelease() to free the memory it's using.

Note, as a side effect, that this call may change the data in dstField. If this is undesirable, then an easy work around is to create a second temporary field with the same structure as dstField and pass that in instead.

The arguments are:

srcField
Source Field.
dstField
Destination Field. The data in this Field may be overwritten by this call.
[srcMaskValues]
Mask information can be set in the Grid (see 31.3.17) or Mesh (see 33.3.11) upon which the srcField is built. The srcMaskValues argument specifies the values in that mask information which indicate a source point should be masked out. In other words, a locati on is masked if and only if the value for that location in the mask information matches one of the values listed in srcMaskValues. If srcMaskValues is not specified, no masking will occur.
[dstMaskValues]
Mask information can be set in the Grid (see 31.3.17) or Mesh (see 33.3.11) upon which the dstField is built. The dstMaskValues argument specifies the values in that mask information which indicate a destination point should be masked out. In other words, a location is masked if and only if the value for that location in the mask information matches one of the values listed in dstMaskValues. If dstMaskValues is not specified, no masking will occur.
[regridmethod]
The type of interpolation. Please see Section 54.51 for a list of valid options. If not specified, defaults to ESMF_REGRIDMETHOD_BILINEAR.
[polemethod]
Specifies the type of pole to construct on the source Grid during regridding. Please see Section 54.48 for a list of valid options. If not specified, defaults to ESMF_POLEMETHOD_ALLAVG for non-conservative regrid methods, and ESMF_POLEMETHOD_NONE for conservative methods.
[regridPoleNPnts]
If polemethod is ESMF_POLEMETHOD_NPNTAVG, then this parameter indicates the number of points over which to average. If polemethod is not ESMF_POLEMETHOD_NPNTAVG and regridPoleNPnts is specified, then it will be ignored. This subroutine will return an error if polemethod is ESMF_POLEMETHOD_NPNTAVG and regridPoleNPnts is not specified.
[lineType]
This argument controls the path of the line which connects two points on a sphere surface. This in turn controls the path along which distances are calculated and the shape of the edges that make up a cell. Both of these quantities can influence how interpolation weights are calculated. As would be expected, this argument is only applicable when srcField and dstField are built on grids which lie on the surface of a sphere. Section 54.36 shows a list of valid options for this argument. Figure 24.2.16 shows which line types are supported for each regrid method as well as showing the default line type by regrid method. If not specified, defaults to ESMF_LINETYPE_CART for non-conservative regrid methods, and ESMF_LINETYPE_GREAT_CIRCLE for conservative methods.

[normType]
This argument controls the type of normalization used when generating conservative weights. This option only applies to weights generated with regridmethod=ESMF_REGRIDMETHOD_CONSERVE or regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND Please see Section 54.45 for a list of valid options. If not specified normType defaults to ESMF_NORMTYPE_DSTAREA.
[vectorRegrid]
If true, treat a single ungridded dimension in the source and destination Fields as the components of a vector. If true and there is more than one ungridded dimension in either the source or destination, then an error will be returned. Currently, only undistributed (vector) dimensions of size 2 are supported. In the vector dimension, the first entry is interpreted as the east component and the second as the north component. In addition, this functionality presently only works when both the source and destination Fields are build on a geometry (e.g. an ESMF Grid) with a spherical coordinate system (e.g. ESMF_COORDSYS_SPH_DEG). We expect these restrictions to be loosened over time as new requirements come in from users. See section 24.2.17 for further information on this functionality. If not specified, this argument defaults to false.
[extrapMethod]
The type of extrapolation. Please see Section 54.18 for a list of valid options. If not specified, defaults to ESMF_EXTRAPMETHOD_NONE.
[extrapNumSrcPnts]
The number of source points to use for the extrapolation methods that use more than one source point (e.g. ESMF_EXTRAPMETHOD_NEAREST_IDAVG). If not specified, defaults to 8.
[extrapDistExponent]
The exponent to raise the distance to when calculating weights for the ESMF_EXTRAPMETHOD_NEAREST_IDAVG extrapolation method. A higher value reduces the influence of more distant points. If not specified, defaults to 2.0.
[extrapNumLevels]
The number of levels to output for the extrapolation methods that fill levels (e.g. ESMF_EXTRAPMETHOD_CREEP). When a method is used that requires this, then an error will be returned, if it is not specified.
[unmappedaction]
Specifies what should happen if there are destination points that can't be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR.
[ignoreDegenerate]
Ignore degenerate cells when checking for errors. If this is set to true, then the regridding proceeds, but degenerate cells will be skipped. If set to false, a degenerate cell produces an error. If not specified, ignoreDegenerate defaults to false.
[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldRegridStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldRegridStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[routehandle]
The communication handle that implements the regrid operation and that can be used later in the ESMF_FieldRegrid() call. The routehandle is optional so that if the user doesn't need it, then they can indicate that by not requesting it. The time to compute the routehandle can be a significant fraction of the time taken by this method, so if it's not needed then not requesting it is worthwhile.
[factorList]
The list of coefficients for a sparse matrix which interpolates from srcField to dstField. The array coming out of this variable is in the appropriate format to be used in other ESMF sparse matrix multiply calls, for example ESMF_FieldSMMStore(). The factorList array is allocated by the method and the user is responsible for deallocating it.
[factorIndexList]
The indices for a sparse matrix which interpolates from srcField to dstField. This argument is a 2D array containing pairs of source and destination sequence indices corresponding to the coefficients in the factorList argument. The first dimension of factorIndexList is of size 2. factorIndexList(1,:) specifies the sequence index of the source element in the srcField. factorIndexList(2,:) specifies the sequence index of the destination element in the dstField. The se cond dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2)==size(factorList). The array coming out of this variable is in the appropriate format to be used in other ESMF sparse matrix multiply calls, for example ESMF_FieldSMMStore(). The factorIndexList array is allocated by the method and the user is responsible for deallocating it.
[weights]
DEPRECATED ARGUMENT! Please use the argument factorList instead.
[indices]
DEPRECATED ARGUMENT! Please use the argument factorIndexList instead.
[srcFracField]
The fraction of each source cell participating in the regridding. Only valid when regridmethod is ESMF_REGRIDMETHOD_CONSERVE or regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND. This Field needs to be created on the same location (e.g staggerloc) as the srcField.
[dstFracField]
The fraction of each destination cell participating in the regridding. Only valid when regridmethod is ESMF_REGRIDMETHOD_CONSERVE or regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND. This Field needs to be created on the same location (e.g staggerloc) as the dstField. It is important to note that the current implementation of conservative regridding doesn't normalize the interpolation weights by the destination fraction. This means that for a destination grid which only partially overlaps the source grid the destination field which is output from the regrid operation should be divided by the corresponding destination fraction to yield the true interpolated values for cells which are only partially covered by the source grid.
[dstStatusField]
An ESMF Field which outputs a regrid status value for each destination location. Section 54.52 indicates the meaning of each value. The Field needs to be built on the same location (e.g. staggerloc) in the same Grid, Mesh, XGrid, or LocStream as the dstField argument. The Field also needs to be of typekind ESMF_TYPEKIND_I4. This option currently doesn't work with the ESMF_REGRIDMETHOD_NEAREST_DTOS regrid method.
[unmappedDstList]
The list of the sequence indices for locations in dstField which couldn't be mapped the srcField. The list on each PET only contains the unmapped locations for the piece of the dstField on that PET. If a destination point is masked, it won't be put in this list. This option currently doesn't work with the ESMF_REGRIDMETHOD_NEAREST_DTOS regrid method.
[checkFlag]
If set to .FALSE. (default) only quick error checking will be performed. If set to .TRUE. more expensive error checking will be performed, possibly catching more errors. Set checkFlag to .FALSE. to achieve highest performance. The checkFlag currently only turns on checking for conservative regrid methods (e.g. ESMF_REGRIDMETHOD_CONSERVE).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.70 ESMF_FieldRegridStore - Precompute a Field regridding operation between an XGrid and one of its side Grids or Meshes


INTERFACE:

   !   Private name; call using ESMF_FieldRegridStore()
     subroutine ESMF_FieldRegridStoreX(xgrid, srcField, dstField, &
                     regridmethod, &
                     srcTermProcessing, pipeLineDepth, &
                     routehandle, &
                     srcFracField, dstFracField, &
                     srcMergeFracField, dstMergeFracField, rc)
ARGUMENTS:
       type(ESMF_XGrid),       intent(in)                      :: xgrid
       type(ESMF_Field),       intent(in)                      :: srcField
       type(ESMF_Field),       intent(inout)                   :: dstField
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_RegridMethod_Flag),   intent(in),    optional :: regridmethod
       integer,                        intent(inout), optional :: srcTermProcessing
       integer,                        intent(inout), optional :: pipeLineDepth
       type(ESMF_RouteHandle), intent(inout), optional         :: routehandle
       type(ESMF_Field),       intent(inout), optional         :: srcFracField
       type(ESMF_Field),       intent(inout), optional         :: dstFracField
       type(ESMF_Field),       intent(inout), optional         :: srcMergeFracField
       type(ESMF_Field),       intent(inout), optional         :: dstMergeFracField
       integer,                intent(out),   optional         :: rc
STATUS:

DESCRIPTION:

This method creates a RouteHandle to do conservative interpolation specifically between a Field built on an XGrid and a Field build on one of the Grids or Meshes used to create that XGrid. (To do more general interpolation use the ESMF_FieldRegridStore() method in section 26.6.69.) The RouteHandle produced by this method can then be used in the call ESMF_FieldRegrid() to interpolate from the srcField to the dstField.

The RouteHandle generated by this call is based just on the coordinates in the Grids, XGrids, or Meshes contained in the Fields. If those coordinates don't change the RouteHandle can be used repeatedly to interpolate from the source Field to the destination Field. This is true even if the data in the Fields changes. The RouteHandle may also be used to interpolate between any source and destination Field which are created on the same Grid, XGrid, or Mesh as the original Fields.

When it's no longer needed the RouteHandle should be destroyed by using ESMF_FieldRegridRelease() to free the memory it's using.

Note, as a side effect, that this call may change the data in dstField. If this is undesirable, then an easy work around is to create a second temporary Field with the same structure as dstField and pass that in instead.

The arguments are:

xgrid
Exchange Grid.
srcField
Source Field built on either xgrid or one of the Grids or Meshes used to create xgrid.
dstField
Destination Field built on either xgrid or one of the Grids or Meshes used to create xgrid. The data in this Field may be overwritten by this call.
[regridmethod]
The type of interpolation. For this method only ESMF_REGRIDMETHOD_CONSERVE and ESMF_REGRIDMETHOD_CONSERVE_2ND are supported. If not specified, defaults to ESMF_REGRIDMETHOD_CONSERVE.
[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldRegridStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldRegridStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[routehandle]
The handle that implements the regrid and that can be used in later ESMF_FieldRegrid.
[srcFracField]
The fraction of each source cell participating in the regridding returned from this call. This Field needs to be created on the same Grid and location (e.g staggerloc) as the srcField.
[dstFracField]
The fraction of each destination cell participating in the regridding returned from this call. This Field needs to be created on the same Grid and location (e.g staggerloc) as the dstField.
[srcMergeFracField]
The fraction of each source cell as a result of Grid merge returned from this call. This Field needs to be created on the same Grid and location (e.g staggerloc) as the srcField.
[dstMergeFracField]
The fraction of each destination cell as a result of Grid merge returned from this call. This Field needs to be created on the same Grid and location (e.g staggerloc) as the dstField.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.71 ESMF_FieldRegridGetArea - Get the area of the cells used for conservative interpolation


INTERFACE:

       subroutine ESMF_FieldRegridGetArea(areaField, rc)
RETURN VALUE:
ARGUMENTS:
       type(ESMF_Field), intent(inout)                 :: areaField
       integer, intent(out), optional                  :: rc
DESCRIPTION:

This subroutine gets the area of the cells used for conservative interpolation for the grid object associated with areaField and puts them into areaField. If created on a 2D Grid, it must be built on the ESMF_STAGGERLOC_CENTER stagger location. If created on a 3D Grid, it must be built on the ESMF_STAGGERLOC_CENTER_VCENTER stagger location. If created on a Mesh, it must be built on the ESMF_MESHLOC_ELEMENT mesh location.

If the user has set the area in the Grid, Mesh, or XGrid under areaField, then that's the area that's returned in the units that the user set it in. If the user hasn't set the area, then the area is calculated and returned. If the Grid, Mesh, or XGrid is on the surface of a sphere, then the calculated area is in units of square radians. If the Grid, Mesh, or XGrid is Cartesian, then the calculated area is in square units of whatever unit the coordinates are in.

The arguments are:

areaField
The Field to put the area values in.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.72 ESMF_FieldScatter - Scatter a Fortran array across the ESMF_Field


INTERFACE:

   subroutine ESMF_FieldScatter<rank><type><kind>(field, farray, & 
   rootPet, tile, vm, rc)
ARGUMENTS:
   type(ESMF_Field), intent(inout) :: field 
   mtype (ESMF_KIND_mtypekind),intent(in), target :: farray(mdim) 
   integer, intent(in) :: rootPet 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(in), optional :: tile 
   type(ESMF_VM), intent(in), optional :: vm 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Scatter the data of farray located on rootPET across an ESMF_Field object. A single farray must be scattered across a single DistGrid tile in Field. The optional tile argument allows selection of the tile. For Fields defined on a single tile DistGrid the default selection (tile 1) will be correct. The shape of farray must match the shape of the tile in Field.

If the Field contains replicating DistGrid dimensions data will be scattered across all of the replicated pieces.

The implementation of Scatter and Gather is not sequence index based. If the Field is built on arbitrarily distributed Grid, Mesh, LocStream or XGrid, Scatter will not scatter data from rootPet to the destination data points corresponding to the sequence index on the rootPet. Instead Scatter will scatter a contiguous memory range from rootPet to destination PET. The size of the memory range is equal to the number of data elements on the destination PET. Vice versa for the Gather operation. In this case, the user should use ESMF_FieldRedist to achieve the same data operation result. For examples how to use ESMF_FieldRedist to perform Gather and Scatter, please refer to 26.3.32 and 26.3.31.

This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.

For examples and associated documentation regarding this method see Section 26.3.29.

The arguments are:

field
The ESMF_Field object across which data will be scattered.
{farray}
The Fortran array that is to be scattered. Only root must provide a valid farray, the other PETs may treat farray as an optional argument.
rootPet
PET that holds the valid data in farray.
[tile]
The DistGrid tile in field into which to scatter farray. By default farray will be scattered into tile 1.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.73 ESMF_FieldSet - Set object-wide Field information


INTERFACE:

   subroutine ESMF_FieldSet(field, name, rc)
ARGUMENTS:
     type(ESMF_Field),   intent(inout)         :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(len = *), intent(in),  optional :: name
     integer,            intent(out), optional :: rc
DESCRIPTION:

Sets adjustable settings in an ESMF_Field object.

The arguments are:

field
ESMF_Field object for which to set properties.
[name]
The Field name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.74 ESMF_FieldSync - Synchronize DEs across the Field in case of sharing


INTERFACE:

   subroutine ESMF_FieldSync(field, rc)
ARGUMENTS:
     type(ESMF_Field), intent(in)            :: field
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,          intent(out), optional :: rc
DESCRIPTION:

Synchronizes access to DEs across field to make sure PETs correctly access the data for read and write when DEs are shared.

The arguments are:

field
Specified ESMF_Field object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.75 ESMF_FieldSMM - Execute a Field sparse matrix multiplication


INTERFACE:

   subroutine ESMF_FieldSMM(srcField, dstField, routehandle, &
              zeroregion, termorderflag, checkflag, rc)
ARGUMENTS:
         type(ESMF_Field),          intent(in),    optional  :: srcField
         type(ESMF_Field),          intent(inout), optional  :: dstField
         type(ESMF_RouteHandle),    intent(inout)            :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         type(ESMF_Region_Flag),    intent(in),    optional  :: zeroregion
         type(ESMF_TermOrder_Flag), intent(in),    optional  :: termorderflag
         logical,                   intent(in),    optional  :: checkflag
         integer,                   intent(out),   optional  :: rc
STATUS:

DESCRIPTION:

Execute a precomputed Field sparse matrix multiplication from srcField to dstField. Both srcField and dstField must match the respective Fields used during ESMF_FieldSMMStore() in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcField and dstField arguments are optional in support of the situation where srcField and/or dstField are not defined on all PETs. The srcField and dstField must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

See ESMF_FieldSMMStore() on how to precompute routehandle.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.33.

[srcField]
ESMF_Field with source data.
[dstField]
ESMF_Field with destination data.
routehandle
Handle to the precomputed Route.
[zeroregion]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in dstField will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in dstField will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination Field that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings.
[termorderflag]
Specifies the order of the source side terms in all of the destination sums. The termorderflag only affects the order of terms during the execution of the RouteHandle. See the 37.2.1 section for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods. See 54.60 for a full list of options. The default is ESMF_TERMORDER_FREE, allowing maximum flexibility in the order of terms for optimum performance.
[checkflag]
If set to .TRUE. the input Field pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.76 ESMF_FieldSMMRelease - Release resources associated with Field

sparse matrix multiplication


INTERFACE:

   subroutine ESMF_FieldSMMRelease(routehandle, noGarbage, rc)
ARGUMENTS:
         type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical,                intent(in),   optional  :: noGarbage
         integer,                intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

Release resources associated with a Field sparse matrix multiplication. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.77 ESMF_FieldSMMStore - Precompute Field sparse matrix multiplication with local factors


INTERFACE:

   ! Private name; call using ESMF_FieldSMMStore() 
   subroutine ESMF_FieldSMMStore<type><kind>(srcField, dstField, & 
          routehandle, factorList, factorIndexList, &
          ignoreUnmatchedIndices, srcTermProcessing, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Field),         intent(in)              :: srcField  
     type(ESMF_Field),         intent(inout)           :: dstField  
     type(ESMF_RouteHandle),   intent(inout)           :: routehandle
     <type>(ESMF_KIND_<kind>), intent(in)              :: factorList(:) 
     integer,                  intent(in),             :: factorIndexList(:,:) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                  intent(in),    optional :: ignoreUnmatchedIndices
     integer,                  intent(inout), optional :: srcTermProcessing
     integer,                  intent(inout), optional :: pipeLineDepth
     integer,                  intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Store a Field sparse matrix multiplication operation from srcField to dstField. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source Field vector to the destination Field vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldSMM() on any pair of Fields that matches srcField and dstField in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.33.

The arguments are:

srcField
ESMF_Field with source data.

dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.

routehandle
Handle to the precomputed Route.

factorList
List of non-zero coefficients.

factorIndexList
Pairs of sequence indices for the factors stored in factorList.

The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4. The second dimension of factorIndexList steps through the list of

In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcField while factorIndexList(2,:) specifies the sequence index of the destination element in dstField. For this format to be a valid option source and destination Fields must have matching number of tensor elements (the product of the sizes of all Field tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.

The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcField. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstField.

See section 28.2.18 for details on the definition of Field sequence indices and tensor sequence indices.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcField or dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.78 ESMF_FieldSMMStore - Precompute Field sparse matrix multiplication and transpose with local factors


INTERFACE:

   ! Private name; call using ESMF_FieldSMMStore() 
   subroutine ESMF_FieldSMMStore<type><kind>TR(srcField, dstField, & 
          routehandle, transposeRoutehandle, factorList, factorIndexList, &
          ignoreUnmatchedIndices, srcTermProcessing, &
          pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Field),         intent(inout)           :: srcField  
     type(ESMF_Field),         intent(inout)           :: dstField  
     type(ESMF_RouteHandle),   intent(inout)           :: routehandle
     type(ESMF_RouteHandle),   intent(inout)           :: transposeRoutehandle
     <type>(ESMF_KIND_<kind>), intent(in)              :: factorList(:) 
     integer,                  intent(in),             :: factorIndexList(:,:) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                  intent(in),    optional :: ignoreUnmatchedIndices
     integer,                  intent(inout), optional :: srcTermProcessing
     integer,                  intent(inout), optional :: pipeLineDepth
     integer,                  intent(out),   optional :: rc
DESCRIPTION:

Store a Field sparse matrix multiplication operation from srcField to dstField. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source Field vector to the destination Field vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldSMM() on any pair of Fields that matches srcField and dstField in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.33.

The arguments are:

srcField
ESMF_Field with source data. The data in this Array may be destroyed by this call.

dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.

routehandle
Handle to the precomputed Route.

transposeRoutehandle
A handle to the transposed matrix operation is returned. The transposed operation goes from dstArray to srcArray.

factorList
List of non-zero coefficients.

factorIndexList
Pairs of sequence indices for the factors stored in factorList.

The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4. The second dimension of factorIndexList steps through the list of

In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcField while factorIndexList(2,:) specifies the sequence index of the destination element in dstField. For this format to be a valid option source and destination Fields must have matching number of tensor elements (the product of the sizes of all Field tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.

The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcField. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstField.

See section 28.2.18 for details on the definition of Field sequence indices and tensor sequence indices.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcField or dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.79 ESMF_FieldSMMStore - Precompute Field sparse matrix multiplication without local factors


INTERFACE:

   ! Private name; call using ESMF_FieldSMMStore() 
     subroutine ESMF_FieldSMMStoreNF(srcField, dstField, &
         routehandle, ignoreUnmatchedIndices, &
         srcTermProcessing, pipelineDepth, rc)
ARGUMENTS:
         type(ESMF_Field),       intent(in)              :: srcField  
         type(ESMF_Field),       intent(inout)           :: dstField  
         type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical,                intent(in),    optional :: ignoreUnmatchedIndices
         integer,                intent(inout), optional :: srcTermProcessing
         integer,                intent(inout), optional :: pipeLineDepth
         integer,                intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Store a Field sparse matrix multiplication operation from srcField to dstField. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source Field vector to the destination Field vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldSMM() on any pair of Fields that matches srcField and dstField in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.33.

The arguments are:

srcField
ESMF_Field with source data.

dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.

routehandle
Handle to the precomputed Route.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcField or dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.80 ESMF_FieldSMMStore - Precompute Field sparse matrix multiplication and transpose without local factors


INTERFACE:

   ! Private name; call using ESMF_FieldSMMStore() 
     subroutine ESMF_FieldSMMStoreNFTR(srcField, dstField, &
         routehandle, transposeRoutehandle, ignoreUnmatchedIndices, &
         srcTermProcessing, pipelineDepth, rc)
ARGUMENTS:
         type(ESMF_Field),       intent(inout)           :: srcField  
         type(ESMF_Field),       intent(inout)           :: dstField  
         type(ESMF_RouteHandle), intent(inout)           :: routehandle
         type(ESMF_RouteHandle), intent(inout)           :: transposeRoutehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
         logical,                intent(in),    optional :: ignoreUnmatchedIndices
         integer,                intent(inout), optional :: srcTermProcessing
         integer,                intent(inout), optional :: pipeLineDepth
         integer,                intent(out),   optional :: rc
DESCRIPTION:

Store a Field sparse matrix multiplication operation from srcField to dstField. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source Field vector to the destination Field vector.

Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.

It is erroneous to specify the identical Field object for srcField and dstField arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldSMM() on any pair of Fields that matches srcField and dstField in type, kind, and memory layout of the gridded dimensions. However, the size, number, and index order of ungridded dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

For examples and associated documentation regarding this method see Section 26.3.33.

The arguments are:

srcField
ESMF_Field with source data. The data in this Field may be destroyed by this call.

dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.

routehandle
Handle to the precomputed Route.

transposeRoutehandle
A handle to the transposed matrix operation is returned. The transposed operation goes from dstArray to srcArray.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcField or dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.81 ESMF_FieldSMMStore - Precompute sparse matrix multiplication using factors read from file


INTERFACE:

   ! Private name; call using ESMF_FieldSMMStore()
     subroutine ESMF_FieldSMMStoreFromFile(srcField, dstField, filename, &
       routehandle, ignoreUnmatchedIndices, &
       srcTermProcessing, pipelineDepth, rc)
 
   ! ARGUMENTS:
       type(ESMF_Field),       intent(in)              :: srcField  
       type(ESMF_Field),       intent(inout)           :: dstField
       character(len=*),       intent(in)              :: filename
       type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical,                intent(in),    optional :: ignoreUnmatchedIndices
       integer,                intent(inout), optional :: srcTermProcessing
       integer,                intent(inout), optional :: pipeLineDepth
       integer,                intent(out),   optional :: rc
DESCRIPTION:

Compute an ESMF_RouteHandle using factors read from file.

The arguments are:

srcField
ESMF_Field with source data.

dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.

filename
Path to the file containing weights for creating an ESMF_RouteHandle. See (12.9) for a description of the SCRIP weight file format. Only "row", "col", and "S" variables are required. They must be one-dimensionsal with dimension "n_s".

routehandle
Handle to the ESMF_RouteHandle.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcField or dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange. The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.82 ESMF_FieldSMMStore - Precompute sparse matrix multiplication and transpose using factors read from file


INTERFACE:

   ! Private name; call using ESMF_FieldSMMStore()
     subroutine ESMF_FieldSMMStoreFromFileTR(srcField, dstField, filename, &
       routehandle, transposeRoutehandle, &
       ignoreUnmatchedIndices, srcTermProcessing, pipelineDepth, rc)
 
   ! ARGUMENTS:
       type(ESMF_Field),       intent(inout)           :: srcField  
       type(ESMF_Field),       intent(inout)           :: dstField
       character(len=*),       intent(in)              :: filename
       type(ESMF_RouteHandle), intent(inout)           :: routehandle
       type(ESMF_RouteHandle), intent(inout)           :: transposeRoutehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical,                intent(in),    optional :: ignoreUnmatchedIndices
       integer,                intent(inout), optional :: srcTermProcessing
       integer,                intent(inout), optional :: pipeLineDepth
       integer,                intent(out),   optional :: rc
DESCRIPTION:

Compute an ESMF_RouteHandle using factors read from file.

The arguments are:

srcField
ESMF_Field with source data. The data in this Array may be destroyed by this call.

dstField
ESMF_Field with destination data. The data in this Field may be destroyed by this call.

filename
Path to the file containing weights for creating an ESMF_RouteHandle. See (12.9) for a description of the SCRIP weight file format. Only "row", "col", and "S" variables are required. They must be one-dimensionsal with dimension "n_s".

routehandle
Handle to the ESMF_RouteHandle.

transposeRoutehandle
A handle to the transposed matrix operation is returned. The transposed operation goes from dstArray to srcArray.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcField or dstField side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange. The ESMF_FieldSMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.6.83 ESMF_FieldValidate - Check validity of a Field


INTERFACE:

       subroutine ESMF_FieldValidate(field, rc)
ARGUMENTS:
       type(ESMF_Field), intent(in)            :: field 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,          intent(out), optional :: rc
STATUS:

DESCRIPTION:

Validates that the field is internally consistent. Currently this method determines if the field is uninitialized or already destroyed. It validates the contained array and grid objects. The code also checks if the array and grid sizes agree. This check compares the distgrid contained in array and grid; then it proceeds to compare the computational bounds contained in array and grid.

The method returns an error code if problems are found.

The arguments are:

field
ESMF_Field to validate.
[rc]
Return code; equals ESMF_SUCCESS if the field is valid.

26.6.84 ESMF_FieldWrite - Write Field data into a file


INTERFACE:

   subroutine ESMF_FieldWrite(field, fileName,   &
       variableName, convention, purpose, overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_Field),           intent(in)             :: field 
     character(*),               intent(in)             :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(*),               intent(in),  optional  :: variableName
     character(*),               intent(in),  optional  :: convention
     character(*),               intent(in),  optional  :: purpose
     logical,                    intent(in),  optional  :: overwrite
     type(ESMF_FileStatus_Flag), intent(in),  optional  :: status
     integer,                    intent(in),  optional  :: timeslice
     type(ESMF_IOFmt_Flag),      intent(in),  optional  :: iofmt
     integer,                    intent(out), optional  :: rc
DESCRIPTION:

Write Field data into a file. For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

When convention and purpose arguments are specified, a NetCDF variable can be created with user-specified dimension labels and attributes. Dimension labels may be defined for both gridded and ungridded dimensions. Dimension labels for gridded dimensions are specified at the Grid level by attaching an ESMF Attribute package to it. The Attribute package must contain an attribute named by the pre-defined ESMF parameter ESMF_ATT_GRIDDED_DIM_LABELS. The corresponding value is an array of character strings specifying the desired names of the dimensions. Likewise, for ungridded dimensions, an Attribute package is attached at the Field level. The name of the name must be ESMF_ATT_UNGRIDDED_DIM_LABELS.

NetCDF attributes for the variable can also be specified. As with dimension labels, an Attribute package is added to the Field with the desired names and values. A value may be either a scalar character string, or a scalar or array of type integer, real, or double precision. Dimension label attributes can co-exist with variable attributes within a common Attribute package.

Limitations:

The arguments are:

field
The ESMF_Field object that contains data to be written.
fileName
The name of the output file to which Field data is written. If the Field is a multi-tile Array, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being written to a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be written to "myfile1.nc", tile 2 to "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[variableName]
Variable name in the output file; default is the "name" of field. Use this argument only in the I/O format (such as NetCDF) that supports variable name. If the I/O format does not support this (such as binary format), ESMF will return an error code.
[convention]
Specifies an Attribute package associated with the Field, used to create NetCDF dimension labels and attributes for the variable in the file. When this argument is present, the purpose argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[purpose]
Specifies an Attribute package associated with the Field, used to create NetCDF dimension labels and attributes for the variable in the file. When this argument is present, the convention argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[overwrite]
A logical flag, the default is .false., i.e., existing field data may not be overwritten. If .true., only the data corresponding to the field's name will be overwritten. Note that it is always an error to attempt to overwrite a NetCDF variable with data which has a different shape.
[status]
The file status. Please see Section 54.22 for the list of options. If not present, defaults to ESMF_FILESTATUS_UNKNOWN.
[timeslice]
Some I/O formats (e.g. NetCDF) support the output of data in form of time slices. An unlimited dimension called time is defined in the file variable for this capability. The timeslice argument provides access to the time dimension, and must have a positive value. The behavior of this option may depend on the setting of the overwrite flag:
overwrite = .false.:
If the timeslice value is less than the maximum time already in the file, the write will fail.
overwrite = .true.:
Any positive timeslice value is valid.
By default, i.e. by omitting the timeslice argument, no provisions for time slicing are made in the output file, however, if the file already contains a time axis for the variable, a timeslice one greater than the maximum will be written.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.7 Class API: Field Utilities

26.7.1 ESMF_GridGetFieldBounds - Get precomputed DE-local Fortran data array bounds for creating a Field from a Grid and Fortran array


INTERFACE:

     subroutine ESMF_GridGetFieldBounds(grid, &
         localDe, staggerloc, gridToFieldMap, &
         ungriddedLBound, ungriddedUBound, &
         totalLWidth, totalUWidth, &
         totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
     type(ESMF_Grid),       intent(in)            :: grid     
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,               intent(in),  optional :: localDe
     type(ESMF_StaggerLoc), intent(in),  optional :: staggerloc 
     integer,               intent(in),  optional :: gridToFieldMap(:)    
     integer,               intent(in),  optional :: ungriddedLBound(:)
     integer,               intent(in),  optional :: ungriddedUBound(:)
     integer,               intent(in),  optional :: totalLWidth(:)
     integer,               intent(in),  optional :: totalUWidth(:)
     integer,               intent(out), optional :: totalLBound(:)
     integer,               intent(out), optional :: totalUBound(:)
     integer,               intent(out), optional :: totalCount(:)
     integer,               intent(out), optional :: rc
STATUS:

DESCRIPTION:

Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_Grid and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.

The arguments are:

grid
ESMF_Grid.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values and interpretation of results see section 31.2.6.
[gridToFieldMap]
List with number of elements equal to the grid|s dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid|s dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLWidth]
Lower bound of halo region. The size of this array is the number of dimensions in the grid. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalLWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should be max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalUWidth]
Upper bound of halo region. The size of this array is the number of dimensions in the grid. However, ordering of the elements needs to be the same as they appear in the field. Values default to 0. If values for totalUWidth are specified they must be reflected in the size of the field. That is, for each gridded dimension the field size should max( totalLWidth + totalUWidth + computationalCount, exclusiveCount ).
[totalLBound]
The relative lower bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_Grid and Fortran data array. This is an output variable from this user interface. The relative lower bounds of Fortran data array to be used
[totalUBound]
The relative upper bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_Grid and Fortran data array. This is an output variable from this user interface.
[totalCount]
Number of elements need to be allocated for Fortran data array to be used later in ESMF_FieldCreate from ESMF_Grid and Fortran data array. This is an output variable from this user interface.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.7.2 ESMF_LocStreamGetFieldBounds - Get precomputed DE-local Fortran data array bounds for creating a Field from a LocStream and Fortran array


INTERFACE:

     subroutine ESMF_LocStreamGetFieldBounds(locstream, &
         localDe, gridToFieldMap, &
         ungriddedLBound, ungriddedUBound, &
         totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
     type(ESMF_LocStream), intent(in)            :: locstream     
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(in),  optional :: localDe
     integer,              intent(in),  optional :: gridToFieldMap(:)    
     integer,              intent(in),  optional :: ungriddedLBound(:)
     integer,              intent(in),  optional :: ungriddedUBound(:)
     integer,              intent(out), optional :: totalLBound(:)
     integer,              intent(out), optional :: totalUBound(:)
     integer,              intent(out), optional :: totalCount(:)
     integer,              intent(out), optional :: rc
DESCRIPTION:

Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_LocStream and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.

The arguments are:

locstream
ESMF_LocStream.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
[gridToFieldMap]
List with number of elements equal to 1. The list elements map the dimension of the locstream to a dimension in the field by specifying the appropriate field dimension index. The default is to map the locstream|s dimension against the lowest dimension of the field in sequence, i.e. gridToFieldMap = (/1/). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than 1, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than 1, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLBound]
The relative lower bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_LocStream and Fortran data array. This is an output variable from this user interface.
[totalUBound]
The relative upper bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_LocStream and Fortran data array. This is an output variable from this user interface.
[totalCount]
Number of elements need to be allocated for Fortran data array to be used later in ESMF_FieldCreate from ESMF_LocStream and Fortran data array. This is an output variable from this user interface.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.7.3 ESMF_MeshGetFieldBounds - Get precomputed DE-local Fortran data array bounds for creating a Field from a Mesh and a Fortran array


INTERFACE:

     subroutine ESMF_MeshGetFieldBounds(mesh, &
         meshloc, &
         localDe, gridToFieldMap, &
         ungriddedLBound, ungriddedUBound, &
         totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
     type(ESMF_Mesh), intent(in)            :: mesh     
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_MeshLoc),intent(in),optional :: meshloc
     integer,         intent(in),  optional :: localDe
     integer,         intent(in),  optional :: gridToFieldMap(:)    
     integer,         intent(in),  optional :: ungriddedLBound(:)
     integer,         intent(in),  optional :: ungriddedUBound(:)
     integer,         intent(out), optional :: totalLBound(:)
     integer,         intent(out), optional :: totalUBound(:)
     integer,         intent(out), optional :: totalCount(:)
     integer,         intent(out), optional :: rc
DESCRIPTION:

Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_Mesh and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.

The arguments are:

mesh
ESMF_Mesh.
[meshloc]
Which part of the mesh to build the Field on. Can be set to either ESMF_MESHLOC_NODE or ESMF_MESHLOC_ELEMENT. If not set, defaults to ESMF_MESHLOC_NODE.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
[gridToFieldMap]
List with number of elements equal to the grid|s dimCount. The list elements map each dimension of the grid to a dimension in the field by specifying the appropriate field dimension index. The default is to map all of the grid|s dimensions against the lowest dimensions of the field in sequence, i.e. gridToFieldMap = (/1,2,3,.../). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. It is erroneous to specify the same gridToFieldMap entry multiple times. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than grid dimension count, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLBound]
The relative lower bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_Mesh and Fortran data array. This is an output variable from this user interface.
[totalUBound]
The relative upper bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_Mesh and Fortran data array. This is an output variable from this user interface.
[totalCount]
Number of elements need to be allocated for Fortran data array to be used later in ESMF_FieldCreate from ESMF_Mesh and Fortran data array. This is an output variable from this user interface.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

26.7.4 ESMF_XGridGetFieldBounds - Get precomputed DE-local Fortran data array bounds for creating a Field from an XGrid and a Fortran array


INTERFACE:

     subroutine ESMF_XGridGetFieldBounds(xgrid, &
         xgridside, gridindex, localDe, gridToFieldMap, &
         ungriddedLBound, ungriddedUBound, &
         totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
     type(ESMF_XGrid),          intent(in)            :: xgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_XGridSide_Flag), intent(in),  optional :: xgridside
     integer,                   intent(in),  optional :: gridindex
     integer,                   intent(in),  optional :: localDe
     integer,                   intent(in),  optional :: gridToFieldMap(:)    
     integer,                   intent(in),  optional :: ungriddedLBound(:)
     integer,                   intent(in),  optional :: ungriddedUBound(:)
     integer,                   intent(out), optional :: totalLBound(:)
     integer,                   intent(out), optional :: totalUBound(:)
     integer,                   intent(out), optional :: totalCount(:)
     integer,                   intent(out), optional :: rc
DESCRIPTION:

Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_XGrid and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.

The arguments are:

xgrid
ESMF_XGrid object.
[xgridside]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not passed in then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridside is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid on that side to create the Field on. If not provided, defaults to 1.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
[gridToFieldMap]
List with number of elements equal to 1. The list elements map the dimension of the locstream to a dimension in the field by specifying the appropriate field dimension index. The default is to map the locstream|s dimension against the lowest dimension of the field in sequence, i.e. gridToFieldMap = (/1/). The values of all gridToFieldMap entries must be greater than or equal to one and smaller than or equal to the field rank. The total ungridded dimensions in the field are the total field dimensions less the dimensions in the grid. Ungridded dimensions must be in the same order they are stored in the field.
[ungriddedLBound]
Lower bounds of the ungridded dimensions of the field. The number of elements in the ungriddedLBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than 1, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[ungriddedUBound]
Upper bounds of the ungridded dimensions of the field. The number of elements in the ungriddedUBound is equal to the number of ungridded dimensions in the field. All ungridded dimensions of the field are also undistributed. When field dimension count is greater than 1, both ungriddedLBound and ungriddedUBound must be specified. When both are specified the values are checked for consistency. Note that the the ordering of these ungridded dimensions is the same as their order in the field.
[totalLBound]
The relative lower bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_LocStream and Fortran data array. This is an output variable from this user interface.
[totalUBound]
The relative upper bounds of Fortran data array to be used later in ESMF_FieldCreate from ESMF_LocStream and Fortran data array. This is an output variable from this user interface.
[totalCount]
Number of elements need to be allocated for Fortran data array to be used later in ESMF_FieldCreate from ESMF_LocStream and Fortran data array. This is an output variable from this user interface.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27 ArrayBundle Class

27.1 Description

The ESMF_ArrayBundle class allows a set of Arrays to be bundled into a single object. The Arrays in an ArrayBundle may be of different type, kind, rank and distribution. Besides ease of use resulting from bundling, the ArrayBundle class offers the opportunity for performance optimization when operating on a bundle of Arrays as a single entity. Communication methods are especially good candidates for performance optimization. Best optimization results are expected for ArrayBundles that contain Arrays that share a common distribution, i.e. DistGrid, and are of same type, kind and rank.

ArrayBundles are one of the data objects that can be added to States, which are used for providing to or receiving data from other Components.

27.2 Use and Examples

Examples of creating, destroying and accessing ArrayBundles and their constituent Arrays are provided in this section, along with some notes on ArrayBundle methods.

27.2.1 Creating an ArrayBundle from a list of Arrays

An ArrayBundle is created from a list of ESMF_Array objects.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), rc=rc)

  allocate(arrayList(2))
  arrayList(1) = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
                 rc=rc)

  arrayList(2) = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
                 rc=rc)

Now arrayList is used to create an ArrayBundle object.

  arraybundle = ESMF_ArrayBundleCreate(arrayList=arrayList, &
    name="MyArrayBundle", rc=rc)

Here the temporary arrayList can be deallocated. This will not affect the ESMF Array objects inside the ArrayBundle. However, the Array objects must not be deallocated while the ArrayBundle references them.

  deallocate(arrayList)

27.2.2 Adding, removing, replacing Arrays in the ArrayBundle

Individual Arrays can be added using the Fortran array constructor syntax (/ ... /). Here an ESMF_Array is created on the fly and immediately added to the ArrayBundle.

  call ESMF_ArrayBundleAdd(arraybundle, arrayList=(/ &
    ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, name="AonFly")/), &
    rc=rc)

Items in the ArrayBundle can be replaced by items with the same name.

  call ESMF_ArraySpecSet(arrayspec2, typekind=ESMF_TYPEKIND_R4, rank=2, rc=rc)

  call ESMF_ArrayBundleReplace(arraybundle, arrayList=(/ &
    ESMF_ArrayCreate(arrayspec=arrayspec2, distgrid=distgrid, name="AonFly")/), &
    rc=rc)

Items can be removed from the ArrayBundle by providing their name.

  call ESMF_ArrayBundleRemove(arraybundle, arrayNameList=(/"AonFly"/), rc=rc)

The ArrayBundle AddReplace() method can be used to conveniently add an item to the ArrayBundle, or replacing an existing item of the same name.

  call ESMF_ArrayBundleAddReplace(arraybundle, arrayList=(/ &
    ESMF_ArrayCreate(arrayspec=arrayspec2, distgrid=distgrid, name="AonFly")/), &
    rc=rc)

The ArrayBundle object can be printed at any time to list its contents by name.

  call ESMF_ArrayBundlePrint(arraybundle, rc=rc)

27.2.3 Accessing Arrays inside the ArrayBundle

Individual items in the ArrayBundle can be accessed directly by their name.

  call ESMF_ArrayBundleGet(arraybundle, arrayName="AonFly", array=arrayOut, &
    rc=rc)

A list containing all of the Arrays in the ArrayBundle can also be requested in a single call. This requires that a large enough list argument is passed into the ESMF_ArrayBundleGet() method. The exact number of items in the ArrayBundle can be queried using the arrayCount argument first.

  call ESMF_ArrayBundleGet(arraybundle, arrayCount=arrayCount, rc=rc)

Then use arrayCount to correctly allocate the arrayList variable for a second call to ESMF_ArrayBundleGet().

  allocate(arrayList(arrayCount))
  call ESMF_ArrayBundleGet(arraybundle, arrayList=arrayList, rc=rc)

Now the arrayList variable can be used to access the individual Arrays, e.g. to print them.

  do i=1, arrayCount
    call ESMF_ArrayPrint(arrayList(i), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  enddo

By default the arrayList returned by ESMF_ArrayBundleGet() contains the items in alphabetical order. To instead return the items in the same order in which they were added to the ArrayBundle, the itemorderflag argument is passed with a value of ESMF_ITEMORDER_ADDORDER.

  call ESMF_ArrayBundleGet(arraybundle, arrayList=arrayList, &
    itemorderflag=ESMF_ITEMORDER_ADDORDER, rc=rc)

27.2.4 Destroying an ArrayBundle and its constituents

Destroying an ArrayBundle does not destroy the Arrays. In fact, it leaves the Arrays totally unchanged.

  call ESMF_ArrayBundleDestroy(arraybundle, rc=rc)

The Arrays must be destroyed separately.

  call ESMF_ArrayDestroy(arrayList(1), rc=rc)

  call ESMF_ArrayDestroy(arrayList(2), rc=rc)

  deallocate(arrayList)
  
  call ESMF_DistGridDestroy(distgrid, rc=rc)

27.2.5 Halo communication

One of the most fundamental communication pattern in domain decomposition codes is the halo operation. The ESMF Array class supports halos by allowing memory for extra elements to be allocated on each DE. See section 28.2.15 for a discussion of the Array level halo operation. The ArrayBundle level extents the Array halo operation to bundles of Arrays.

First create an ESMF_ArrayBundle object containing a set of ESMF Arrays.

  arraybundle = ESMF_ArrayBundleCreate(arrayList=arrayList, &
    name="MyArrayBundle", rc=rc)

The ArrayBundle object can be treated as a single entity. The ESMF_ArrayBundleHaloStore() call determines the most efficient halo exchange pattern for all Arrays that are part of arraybundle.

  call ESMF_ArrayBundleHaloStore(arraybundle=arraybundle, &
    routehandle=haloHandle, rc=rc)

The halo exchange pattern stored in haloHandle can now be applied to the arraybundle object, or any other ArrayBundle that is compatible to the one used during the ESMF_ArrayBundleHaloStore() call.

  call ESMF_ArrayBundleHalo(arraybundle=arraybundle, routehandle=haloHandle, &
    rc=rc)

Finally, when no longer needed, the resources held by haloHandle need to be returned to the system by calling ESMF_ArrayBundleHaloRelease().

  call ESMF_ArrayBundleHaloRelease(routehandle=haloHandle, rc=rc)

Finally the ArrayBundle object can be destroyed.

  call ESMF_ArrayBundleDestroy(arraybundle, rc=rc)

27.3 Restrictions and Future Work

27.4 Design and Implementation Notes

The following is a list of implementation specific details about the current ESMF ArrayBundle.

27.5 Class API

27.5.1 ESMF_ArrayBundleAssignment(=) - ArrayBundle assignment


INTERFACE:

     interface assignment(=)
     arraybundle1 = arraybundle2
ARGUMENTS:
     type(ESMF_ArrayBundle) :: arraybundle1
     type(ESMF_ArrayBundle) :: arraybundle2
STATUS:

DESCRIPTION:

Assign arraybundle1 as an alias to the same ESMF ArrayBundle object in memory as arraybundle2. If arraybundle2 is invalid, then arraybundle1 will be equally invalid after the assignment.

The arguments are:

arraybundle1
The ESMF_ArrayBundle object on the left hand side of the assignment.
arraybundle2
The ESMF_ArrayBundle object on the right hand side of the assignment.

27.5.2 ESMF_ArrayBundleOperator(==) - ArrayBundle equality operator


INTERFACE:

   interface operator(==)
     if (arraybundle1 == arraybundle2) then ... endif
               OR
     result = (arraybundle1 == arraybundle2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in) :: arraybundle1
     type(ESMF_ArrayBundle), intent(in) :: arraybundle2
STATUS:

DESCRIPTION:

Test whether arraybundle1 and arraybundle2 are valid aliases to the same ESMF ArrayBundle object in memory. For a more general comparison of two ESMF ArrayBundles, going beyond the simple alias test, the ESMF_ArrayBundleMatch() function (not yet implemented) must be used.

The arguments are:

arraybundle1
The ESMF_ArrayBundle object on the left hand side of the equality operation.
arraybundle2
The ESMF_ArrayBundle object on the right hand side of the equality operation.

27.5.3 ESMF_ArrayBundleOperator(/=) - ArrayBundle not equal operator


INTERFACE:

   interface operator(/=)
     if (arraybundle1 /= arraybundle2) then ... endif
               OR
     result = (arraybundle1 /= arraybundle2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in) :: arraybundle1
     type(ESMF_ArrayBundle), intent(in) :: arraybundle2
STATUS:

DESCRIPTION:

Test whether arraybundle1 and arraybundle2 are not valid aliases to the same ESMF ArrayBundle object in memory. For a more general comparison of two ESMF ArrayBundles, going beyond the simple alias test, the ESMF_ArrayBundleMatch() function (not yet implemented) must be used.

The arguments are:

arraybundle1
The ESMF_ArrayBundle object on the left hand side of the non-equality operation.
arraybundle2
The ESMF_ArrayBundle object on the right hand side of the non-equality operation.

27.5.4 ESMF_ArrayBundleAdd - Add Arrays to an ArrayBundle


INTERFACE:

     subroutine ESMF_ArrayBundleAdd(arraybundle, arrayList, &
       multiflag, relaxedflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)         :: arraybundle
     type(ESMF_Array),       intent(in)            :: arrayList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: multiflag
     logical,                intent(in),  optional :: relaxedflag
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Add Array(s) to an ArrayBundle. It is an error if arrayList contains Arrays that match by name Arrays already contained in arraybundle.

arraybundle
ESMF_ArrayBundle to be added to.
arrayList
List of ESMF_Array objects to be added.
[multiflag]
A setting of .true. allows multiple items with the same name to be added to arraybundle. For .false. added items must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "add" under multiflag=.false. mode, where it is not an error if arrayList contains items with names that are also found in arraybundle. The arraybundle is left unchanged for these items. For .false. this is treated as an error condition. The default setting is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.5 ESMF_ArrayBundleAddReplace - Conditionally add or replace Arrays in an ArrayBundle


INTERFACE:

     subroutine ESMF_ArrayBundleAddReplace(arraybundle, arrayList, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)         :: arraybundle
     type(ESMF_Array),       intent(in)            :: arrayList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Arrays in arrayList that do not match any Arrays by name in arraybundle are added to the ArrayBundle. Arrays in arraybundle that match by name Arrays in arrayList are replaced by those Arrays.

arraybundle
ESMF_ArrayBundle to be manipulated.
arrayList
List of ESMF_Array objects to be added or used as replacement.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.6 ESMF_ArrayBundleCreate - Create an ArrayBundle from a list of Arrays


INTERFACE:

   function ESMF_ArrayBundleCreate(arrayList, multiflag, &
     relaxedflag, name, rc)
RETURN VALUE:
     type(ESMF_ArrayBundle) :: ESMF_ArrayBundleCreate
ARGUMENTS:
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Array), intent(in),  optional :: arrayList(:)
     logical,          intent(in),  optional :: multiflag
     logical,          intent(in),  optional :: relaxedflag
     character(len=*), intent(in),  optional :: name
     integer,          intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_ArrayBundle object from a list of existing Arrays.

The creation of an ArrayBundle leaves the bundled Arrays unchanged, they remain valid individual objects. An ArrayBundle is a light weight container of Array references. The actual data remains in place, there are no data movements or duplications associated with the creation of an ArrayBundle.

[arrayList]
List of ESMF_Array objects to be bundled.
[multiflag]
A setting of .true. allows multiple items with the same name to be added to arraybundle. For .false. added items must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "add" under multiflag=.false. mode, where it is not an error if arrayList contains items with names that are also found in arraybundle. The arraybundle is left unchanged for these items. For .false. this is treated as an error condition. The default setting is .false..
[name]
Name of the created ESMF_ArrayBundle. A default name is generated if not specified.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.7 ESMF_ArrayBundleDestroy - Release resources associated with an ArrayBundle


INTERFACE:

   subroutine ESMF_ArrayBundleDestroy(arraybundle, noGarbage, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)           :: arraybundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional  :: noGarbage
     integer,                intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

Destroys an ESMF_ArrayBundle object. The member Arrays are not touched by this operation and remain valid objects that need to be destroyed individually if necessary.

By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.

The arguments are:

arraybundle
ESMF_ArrayBundle object to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.8 ESMF_ArrayBundleGet - Get object-wide information from an ArrayBundle


INTERFACE:

     ! Private name; call using ESMF_ArrayBundleGet()   
     subroutine ESMF_ArrayBundleGetListAll(arraybundle, &
       itemorderflag, arrayCount, arrayList, arrayNameList, name, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),    intent(in)            :: arraybundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_ItemOrder_Flag), intent(in),  optional :: itemorderflag
     integer,                   intent(out), optional :: arrayCount
     type(ESMF_Array),          intent(out), optional :: arrayList(:)
     character(len=*),          intent(out), optional :: arrayNameList(:)
     character(len=*),          intent(out), optional :: name
     integer,                   intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get general, i.e. not Array name specific information from the ArrayBundle.

arraybundle
ESMF_ArrayBundle to be queried.
[itemorderflag]
Specifies the order of the returned items in the arrayList and arrayNameList. The default is ESMF_ITEMORDER_ABC. See 54.34 for a full list of options.
[arrayCount]
Upon return holds the number of Arrays bundled in the ArrayBundle.
[arrayList]
Upon return holds a list of Arrays bundled in arraybundle. The argument must be allocated to be at least of size arrayCount.
[arrayNameList]
Upon return holds a list of the names of the Arrays bundled in arraybundle. The argument must be allocated to be at least of size arrayCount.
[name]
Name of the ArrayBundle object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.9 ESMF_ArrayBundleGet - Get information about an Array by name and optionally return an Array


INTERFACE:

     ! Private name; call using ESMF_ArrayBundleGet()   
     subroutine ESMF_ArrayBundleGetItem(arraybundle, arrayName, &
       array, arrayCount, isPresent, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in)            :: arraybundle
     character(len=*),       intent(in)            :: arrayName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Array),       intent(out), optional :: array
     integer,                intent(out), optional :: arrayCount
     logical,                intent(out), optional :: isPresent
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get information about items that match arrayName in ArrayBundle.

arraybundle
ESMF_ArrayBundle to be queried.
arrayName
Specified name.
[array]
Upon return holds the requested Array item. It is an error if this argument was specified and there is not exactly one Array item in arraybundle that matches arrayName.
[arrayCount]
Number of Arrays with arrayName in arraybundle.
[isPresent]
Upon return indicates whether Array(s) with arrayName exist in arraybundle.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.10 ESMF_ArrayBundleGet - Get a list of Arrays by name


INTERFACE:

     ! Private name; call using ESMF_ArrayBundleGet()   
     subroutine ESMF_ArrayBundleGetList(arraybundle, arrayName, arrayList, &
       itemorderflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),    intent(in)            :: arraybundle
     character(len=*),          intent(in)            :: arrayName
     type(ESMF_Array),          intent(out)           :: arrayList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_ItemOrder_Flag), intent(in),  optional :: itemorderflag
     integer,                   intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get the list of Arrays from ArrayBundle that match arrayName.

arraybundle
ESMF_ArrayBundle to be queried.
arrayName
Specified name.
arrayList
List of Arrays in arraybundle that match arrayName. The argument must be allocated to be at least of size arrayCount returned for this arrayName.
[itemorderflag]
Specifies the order of the returned items in the arrayList. The default is ESMF_ITEMORDER_ABC. See 54.34 for a full list of options.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.11 ESMF_ArrayBundleHalo - Execute an ArrayBundle halo operation


INTERFACE:

   subroutine ESMF_ArrayBundleHalo(arraybundle, routehandle, &
     checkflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)          :: arraybundle
     type(ESMF_RouteHandle), intent(inout)          :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional :: checkflag
     integer,                intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed ArrayBundle halo operation for the Arrays in arrayBundle.

See ESMF_ArrayBundleHaloStore() on how to precompute routehandle.

This call is collective across the current VM.

arraybundle
ESMF_ArrayBundle containing data to be haloed.
routehandle
Handle to the precomputed Route.
[checkflag]
If set to .TRUE. the input Array pairs will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.12 ESMF_ArrayBundleHaloRelease - Release resources associated with an ArrayBundle halo operation


INTERFACE:

   subroutine ESMF_ArrayBundleHaloRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)          :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional :: noGarbage
     integer,                intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with an ArrayBundle halo operation. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.13 ESMF_ArrayBundleHaloStore - Precompute an ArrayBundle halo operation


INTERFACE:

     subroutine ESMF_ArrayBundleHaloStore(arraybundle, routehandle, &
       startregion, haloLDepth, haloUDepth, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),     intent(inout)         :: arraybundle
     type(ESMF_RouteHandle),     intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_StartRegion_Flag),intent(in),  optional :: startregion
     integer,                    intent(in),  optional :: haloLDepth(:)
     integer,                    intent(in),  optional :: haloUDepth(:)
     integer,                    intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store an ArrayBundle halo operation over the data in arraybundle. By default, i.e. without specifying startregion, haloLDepth and haloUDepth, all elements in the total Array regions that lie outside the exclusive regions will be considered potential destination elements for the halo operation. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.

Specifying startregion allows to change the shape of the effective halo region from the inside. Setting this flag to ESMF_STARTREGION_COMPUTATIONAL means that only elements outside the computational region for each Array are considered for potential destination elements for the halo operation. The default is ESMF_STARTREGION_EXCLUSIVE.

The haloLDepth and haloUDepth arguments allow to reduce the extent of the effective halo region. Starting at the region specified by startregion, the haloLDepth and haloUDepth define a halo depth in each direction. Note that the maximum halo region is limited by the total region for each Array, independent of the actual haloLDepth and haloUDepth setting. The total Array regions are local DE specific. The haloLDepth and haloUDepth are interpreted as the maximum desired extent, reducing the potentially larger region available for the halo operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleHalo() on any pair of ArrayBundles that matches srcArrayBundle and dstArrayBundle in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

arraybundle
ESMF_ArrayBundle containing data to be haloed. The data in the halo regions may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[startregion]
The start of the effective halo region on every DE. The default setting is ESMF_STARTREGION_EXCLUSIVE, rendering all non-exclusive elements potential halo destination elements. See section 54.56 for a complete list of valid settings.
[haloLDepth]
This vector specifies the lower corner of the effective halo region with respect to the lower corner of startregion. The size of haloLDepth must equal the number of distributed Array dimensions.
[haloUDepth]
This vector specifies the upper corner of the effective halo region with respect to the upper corner of startregion. The size of haloUDepth must equal the number of distributed Array dimensions.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.14 ESMF_ArrayBundleIsCreated - Check whether an ArrayBundle object has been created


INTERFACE:

   function ESMF_ArrayBundleIsCreated(arraybundle, rc)
RETURN VALUE:
     logical :: ESMF_ArrayBundleIsCreated
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in)            :: arraybundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
DESCRIPTION:

Return .true. if the arraybundle has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

arraybundle
ESMF_ArrayBundle queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.15 ESMF_ArrayBundlePrint - Print ArrayBundle information


INTERFACE:

   subroutine ESMF_ArrayBundlePrint(arraybundle, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in)            :: arraybundle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Print internal information of the specified ESMF_ArrayBundle object to stdout.

The arguments are:

arraybundle
ESMF_ArrayBundle object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.16 ESMF_ArrayBundleRead - Read Arrays to an ArrayBundle from file(s)


INTERFACE:

   subroutine ESMF_ArrayBundleRead(arraybundle, fileName, &
     singleFile, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)          :: arraybundle
     character(*),           intent(in)             :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional  :: singleFile
     integer,                intent(in),  optional  :: timeslice
     type(ESMF_IOFmt_Flag),  intent(in),  optional  :: iofmt
     integer,                intent(out), optional  :: rc
DESCRIPTION:

Read Array data to an ArrayBundle object from file(s). For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

Limitations:

The arguments are:

arraybundle
An ESMF_ArrayBundle object.
fileName
The name of the file from which ArrayBundle data is read. If the ArrayBundle contains multi-tile Arrays, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being read from a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be read from "myfile1.nc", tile 2 from "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[singleFile]
A logical flag, the default is .true., i.e., all Arrays in the bundle are stored in one single file. If .false., each Array is stored in separate files; these files are numbered with the name based on the argument "file". That is, a set of files are named: [file_name]001, [file_name]002, [file_name]003,...
[timeslice]
The time-slice number of the variable read from file.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.17 ESMF_ArrayBundleRedist - Execute an ArrayBundle redistribution


INTERFACE:

   subroutine ESMF_ArrayBundleRedist(srcArrayBundle, dstArrayBundle, &
     routehandle, checkflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in),    optional :: srcArrayBundle
     type(ESMF_ArrayBundle), intent(inout), optional :: dstArrayBundle
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),    optional :: checkflag
     integer,                intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed ArrayBundle redistribution from the Arrays in srcArrayBundle to the Arrays in dstArrayBundle.

The srcArrayBundle and dstArrayBundle arguments are optional in support of the situation where srcArrayBundle and/or dstArrayBundle are not defined on all PETs. The srcArrayBundle and dstArrayBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

This call is collective across the current VM.

[srcArrayBundle]
ESMF_ArrayBundle with source data.
[dstArrayBundle]
ESMF_ArrayBundle with destination data.
routehandle
Handle to the precomputed Route.
[checkflag]
If set to .TRUE. the input Array pairs will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.18 ESMF_ArrayBundleRedistRelease - Release resources associated with ArrayBundle redistribution


INTERFACE:

   subroutine ESMF_ArrayBundleRedistRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)          :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional :: noGarbage
     integer,                intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with an ArrayBundle redistribution. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.19 ESMF_ArrayBundleRedistStore - Precompute an ArrayBundle redistribution with local factor argument


INTERFACE:

   ! Private name; call using ESMF_ArrayBundleRedistStore()
   subroutine ESMF_ArrayBundleRedistStore<type><kind>(srcArrayBundle, &
     dstArrayBundle, routehandle, factor, ignoreUnmatchedIndicesFlag, &
     srcToDstTransposeMap, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),  intent(in)            :: srcArrayBundle
     type(ESMF_ArrayBundle),  intent(inout)         :: dstArrayBundle
     type(ESMF_RouteHandle),  intent(inout)         :: routehandle
     <type>(ESMF_KIND_<kind>),intent(in)            :: factor
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                 intent(in),  optional :: ignoreUnmatchedIndicesFlag(:)
     integer,                 intent(in),  optional :: srcToDstTransposeMap(:)
     integer,                 intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store an ArrayBundle redistribution operation from srcArrayBundle to dstArrayBundle. The redistribution between ArrayBundles is defined as the sequence of individual Array redistributions over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.

The effect of this method on ArrayBundles that contain aliased members is undefined.

PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.

See the description of method ESMF_ArrayRedistStore() for the definition of the Array based operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleRedist() on any pair of ArrayBundles that matches srcArrayBundle and dstArrayBundle in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

srcArrayBundle
ESMF_ArrayBundle with source data.
dstArrayBundle
ESMF_ArrayBundle with destination data. The data in these Arrays may be destroyed by this call.
routehandle
Handle to the precomputed Route.
factor
Factor by which to multiply source data.
[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover the identical index space, using precisely matching sequence indices. If set to .true., mismatching sequence indices between source and destination side are silently ignored. The size of this array argument must either be 1 or equal the number of Arrays in the srcArrayBundle and dstArrayBundle arguments. In the latter case, the handling of unmatched indices is specified for each Array pair separately. If only one element is specified, it is used for all Array pairs.
[srcToDstTransposeMap]
List with as many entries as there are dimensions in the Arrays in srcArrayBundle. Each entry maps the corresponding source Array dimension against the specified destination Array dimension. Mixing of distributed and undistributed dimensions is supported.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.20 ESMF_ArrayBundleRedistStore - Precompute an ArrayBundle redistribution without local factor argument


INTERFACE:

   ! Private name; call using ESMF_ArrayBundleRedistStore()
   subroutine ESMF_ArrayBundleRedistStoreNF(srcArrayBundle, dstArrayBundle, &
     routehandle, ignoreUnmatchedIndicesFlag, &
     srcToDstTransposeMap, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(in)            :: srcArrayBundle
     type(ESMF_ArrayBundle), intent(inout)         :: dstArrayBundle
     type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: ignoreUnmatchedIndicesFlag(:)
     integer,                intent(in),  optional :: srcToDstTransposeMap(:)
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Store an ArrayBundle redistribution operation from srcArrayBundle to dstArrayBundle. The redistribution between ArrayBundles is defined as the sequence of individual Array redistributions over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.

The effect of this method on ArrayBundles that contain aliased members is undefined.

PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.

See the description of method ESMF_ArrayRedistStore() for the definition of the Array based operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleRedist() on any pair of ArrayBundles that matches srcArrayBundle and dstArrayBundle in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcArrayBundle
ESMF_ArrayBundle with source data.
dstArrayBundle
ESMF_ArrayBundle with destination data. The data in these Arrays may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover the identical index space, using precisely matching sequence indices. If set to .true., mismatching sequence indices between source and destination side are silently ignored. The size of this array argument must either be 1 or equal the number of Arrays in the srcArrayBundle and dstArrayBundle arguments. In the latter case, the handling of unmatched indices is specified for each Array pair separately. If only one element is specified, it is used for all Array pairs.
[srcToDstTransposeMap]
List with as many entries as there are dimensions in the Arrays in srcArrayBundle. Each entry maps the corresponding source Array dimension against the specified destination Array dimension. Mixing of distributed and undistributed dimensions is supported.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.21 ESMF_ArrayBundleRemove - Remove Arrays from ArrayBundle


INTERFACE:

     subroutine ESMF_ArrayBundleRemove(arraybundle, arrayNameList, &
       multiflag, relaxedflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)         :: arraybundle
     character(len=*),       intent(in)            :: arrayNameList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: multiflag
     logical,                intent(in),  optional :: relaxedflag
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Remove Array(s) by name from ArrayBundle. In the relaxed setting it is not an error if arrayNameList contains names that are not found in arraybundle.

arraybundle
ESMF_ArrayBundle from which to remove items.
arrayNameList
List of items to remove.
[multiflag]
A setting of .true. allows multiple Arrays with the same name to be removed from arraybundle. For .false., items to be removed must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "remove" where it is not an error if arrayNameList contains item names that are not found in arraybundle. For .false. this is treated as an error condition. Further, in multiflag=.false. mode, the relaxed definition of "remove" also covers the case where there are multiple items in arraybundle that match a single entry in arrayNameList. For relaxedflag=.false. this is treated as an error condition. The default setting is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.22 ESMF_ArrayBundleReplace - Replace Arrays in ArrayBundle


INTERFACE:

     subroutine ESMF_ArrayBundleReplace(arraybundle, arrayList, &
       multiflag, relaxedflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle), intent(inout)         :: arraybundle
     type(ESMF_Array),       intent(in)            :: arrayList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: multiflag
     logical,                intent(in),  optional :: relaxedflag
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Replace Array(s) by name in ArrayBundle. In the relaxed setting it is not an error if arrayList contains Arrays that do not match by name any item in arraybundle. These Arrays are simply ignored in this case.

arraybundle
ESMF_ArrayBundle in which to replace items.
arrayList
List of items to replace.
[multiflag]
A setting of .true. allows multiple items with the same name to be replaced in arraybundle. For .false., items to be replaced must have unique names. The default setting is .false..
[relaxedflag]
A setting of .true. indicates a relaxed definition of "replace" where it is not an error if arrayList contains items with names that are not found in arraybundle. These items in arrayList are ignored in the relaxed mode. For .false. this is treated as an error condition. Further, in multiflag=.false. mode, the relaxed definition of "replace" also covers the case where there are multiple items in arraybundle that match a single entry by name in arrayList. For relaxedflag=.false. this is treated as an error condition. The default setting is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.23 ESMF_ArrayBundleSMM - Execute an ArrayBundle sparse matrix multiplication


INTERFACE:

   subroutine ESMF_ArrayBundleSMM(srcArrayBundle, dstArrayBundle, &
     routehandle, &
     zeroregion, & ! DEPRECATED ARGUMENT
     zeroregionflag, termorderflag, checkflag, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),    intent(in),         optional :: srcArrayBundle
     type(ESMF_ArrayBundle),    intent(inout),      optional :: dstArrayBundle
     type(ESMF_RouteHandle),    intent(inout)                :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Region_Flag),    intent(in), optional :: zeroregion ! DEPRECATED ARGUMENT
     type(ESMF_Region_Flag),    intent(in), target, optional :: zeroregionflag(:)
     type(ESMF_TermOrder_Flag), intent(in), target, optional :: termorderflag(:)
     logical,                   intent(in),         optional :: checkflag
     integer,                   intent(out),        optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed ArrayBundle sparse matrix multiplication from the Arrays in srcArrayBundle to the Arrays in dstArrayBundle.

The srcArrayBundle and dstArrayBundle arguments are optional in support of the situation where srcArrayBundle and/or dstArrayBundle are not defined on all PETs. The srcArrayBundle and dstArrayBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

This call is collective across the current VM.

[srcArrayBundle]
ESMF_ArrayBundle with source data.
[dstArrayBundle]
ESMF_ArrayBundle with destination data.
routehandle
Handle to the precomputed Route.
[zeroregion]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in all Arrays in dstArrayBundle will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in the Arrays in dstArrayBundle will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination Arrays that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings.
[zeroregionflag]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in the destination Array will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in the destination Array will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. A setting of ESMF_REGION_SELECT will only zero out those elements in the destination Array that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings. The size of this array argument must either be 1 or equal the number of Arrays in the srcArrayBundle and dstArrayBundle arguments. In the latter case, the zero region for each Array SMM operation is indicated separately. If only one zero region element is specified, it is used for all Array pairs.
[termorderflag]
Specifies the order of the source side terms in all of the destination sums. The termorderflag only affects the order of terms during the execution of the RouteHandle. See the 37.2.1 section for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods. See 54.60 for a full list of options. The size of this array argument must either be 1 or equal the number of Arrays in the srcArrayBundle and dstArrayBundle arguments. In the latter case, the term order for each Array SMM operation is indicated separately. If only one term order element is specified, it is used for all Array pairs. The default is (/ESMF_TERMORDER_FREE/), allowing maximum flexibility in the order of terms for optimum performance.
[checkflag]
If set to .TRUE. the input Array pairs will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.24 ESMF_ArrayBundleSMMRelease - Release resources associated with ArrayBundle sparse matrix multiplication


INTERFACE:

   subroutine ESMF_ArrayBundleSMMRelease(routehandle, &
     noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)          :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional :: noGarbage
     integer,                intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with an ArrayBundle sparse matrix multiplication. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.25 ESMF_ArrayBundleSMMStore - Precompute an ArrayBundle sparse matrix multiplication with local factors


INTERFACE:

   ! Private name; call using ESMF_ArrayBundleSMMStore()
   subroutine ESMF_ArrayBundleSMMStore<type><kind>(srcArrayBundle, &
     dstArrayBundle, routehandle, factorList, factorIndexList, &
     ignoreUnmatchedIndicesFlag, srcTermProcessing, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),           intent(in)    :: srcArrayBundle
     type(ESMF_ArrayBundle),           intent(inout) :: dstArrayBundle
     type(ESMF_RouteHandle),           intent(inout) :: routehandle
     <type>(ESMF_KIND_<kind>), target, intent(in)    :: factorList(:)
     integer,                          intent(in)    :: factorIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),    optional :: ignoreUnmatchedIndicesFlag(:)
     integer,                intent(inout), optional :: srcTermProcessing(:)
     integer,                intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Store an ArrayBundle sparse matrix multiplication operation from srcArrayBundle to dstArrayBundle. The sparse matrix multiplication between ArrayBundles is defined as the sequence of individual Array sparse matrix multiplications over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.

The effect of this method on ArrayBundles that contain aliased members is undefined.

PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

See the description of method ESMF_ArraySMMStore() for the definition of the Array based operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleSMM() on any pair of ArrayBundles that matches srcArrayBundle and dstArrayBundle in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

srcArrayBundle
ESMF_ArrayBundle with source data.
dstArrayBundle
ESMF_ArrayBundle with destination data. The data in these Arrays may be destroyed by this call.
routehandle
Handle to the precomputed Route.
factorList
List of non-zero coefficients.
factorIndexList
Pairs of sequence indices for the factors stored in factorList.

The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.

In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the source Array while factorIndexList(2,:) specifies the sequence index of the destination element in the destination Array. For this format to be a valid option source and destination Arrays must have matching number of tensor elements (the product of the sizes of all Array tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.

The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the source Array. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the destination Array.

See section 28.2.18 for details on the definition of Array sequence indices and tensor sequence indices.

[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover all of the squence indices defined in the sparse matrix. An error will be returned if a sequence index in the sparse matrix does not match on either the source or destination side. If set to .true., mismatching sequence indices are silently ignored. The size of this array argument must either be 1 or equal the number of Arrays in the srcArrayBundle and dstArrayBundle arguments. In the latter case, the handling of unmatched indices is specified for each Array pair separately. If only one element is specified, it is used for all Array pairs.

[srcTermProcessing]
Source term summing options for route handle creation. See ESMF_ArraySMMStore documentation for a full parameter description. Two forms may be provided. If a single element list is provided, this integer value is applied across all bundle members. Otherwise, the list must contain as many elements as there are bundle members. For the special case of accessing the auto-tuned parameter (providing a negative integer value), the list length must equal the bundle member count.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.26 ESMF_ArrayBundleSMMStore - Precompute an ArrayBundle sparse matrix multiplication without local factors


INTERFACE:

   ! Private name; call using ESMF_ArrayBundleSMMStore()
   subroutine ESMF_ArrayBundleSMMStoreNF(srcArrayBundle, dstArrayBundle, &
     routehandle, ignoreUnmatchedIndicesFlag, srcTermProcessing, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),  intent(in)              :: srcArrayBundle
     type(ESMF_ArrayBundle),  intent(inout)           :: dstArrayBundle
     type(ESMF_RouteHandle),  intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                 intent(in),    optional :: ignoreUnmatchedIndicesFlag(:)
     integer,                 intent(inout), optional :: srcTermProcessing(:)
     integer,                 intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Store an ArrayBundle sparse matrix multiplication operation from srcArrayBundle to dstArrayBundle. The sparse matrix multiplication between ArrayBundles is defined as the sequence of individual Array sparse matrix multiplications over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.

The effect of this method on ArrayBundles that contain aliased members is undefined.

PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

See the description of method ESMF_ArraySMMStore() for the definition of the Array based operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleSMM() on any pair of ArrayBundles that matches srcArrayBundle and dstArrayBundle in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcArrayBundle
ESMF_ArrayBundle with source data.
dstArrayBundle
ESMF_ArrayBundle with destination data. The data in these Arrays may be destroyed by this call.
routehandle
Handle to the precomputed Route.

[ignoreUnmatchedIndicesFlag]
If set to .false., the default, source and destination side must cover all of the squence indices defined in the sparse matrix. An error will be returned if a sequence index in the sparse matrix does not match on either the source or destination side. If set to .true., mismatching sequence indices are silently ignored. The size of this array argument must either be 1 or equal the number of Arrays in the srcArrayBundle and dstArrayBundle arguments. In the latter case, the handling of unmatched indices is specified for each Array pair separately. If only one element is specified, it is used for all Array pairs.

[srcTermProcessing]
Source term summing options for route handle creation. See ESMF_ArraySMMStore documentation for a full parameter description. Two forms may be provided. If a single element list is provided, this integer value is applied across all bundle members. Otherwise, the list must contain as many elements as there are bundle members. For the special case of accessing the auto-tuned parameter (providing a negative integer value), the list length must equal the bundle member count.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

27.5.27 ESMF_ArrayBundleWrite - Write the Arrays into a file


INTERFACE:

   subroutine ESMF_ArrayBundleWrite(arraybundle, fileName, &
     convention, purpose, singleFile, overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_ArrayBundle),     intent(in)              :: arraybundle
     character(*),               intent(in)              :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(*),               intent(in),  optional  :: convention
     character(*),               intent(in),  optional  :: purpose
     logical,                    intent(in),  optional  :: singleFile
     logical,                    intent(in),  optional  :: overwrite
     type(ESMF_FileStatus_Flag), intent(in),  optional  :: status
     integer,                    intent(in),  optional  :: timeslice
     type(ESMF_IOFmt_Flag),      intent(in),  optional  :: iofmt
     integer,                    intent(out), optional  :: rc
DESCRIPTION:

Write the Arrays into a file. For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

When convention and purpose arguments are specified, NetCDF dimension labels and variable attributes are written from each Array in the ArrayBundle from the corresponding Attribute package. Additionally, Attributes may be set on the ArrayBundle level under the same Attribute package. This allows the specification of global attributes within the file. As with individual Arrays, the value associated with each name may be either a scalar character string, or a scalar or array of type integer, real, or double precision.

Limitations:

The arguments are:

arraybundle
An ESMF_ArrayBundle object.
fileName
The name of the output file to which array bundle data is written. If the ArrayBundle contains multi-tile Arrays, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being written to a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be written to "myfile1.nc", tile 2 to "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[convention]
Specifies an Attribute package associated with the ArrayBundle, and the contained Arrays, used to create NetCDF dimension labels and attributes in the file. When this argument is present, the purpose argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[purpose]
Specifies an Attribute package associated with the ArrayBundle, and the contained Arrays, used to create NetCDF dimension labels and attributes in the file. When this argument is present, the convention argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[singleFile]
A logical flag, the default is .true., i.e., all arrays in the bundle are written in one single file. If .false., each array will be written in separate files; these files are numbered with the name based on the argument "file". That is, a set of files are named: [file_name]001, [file_name]002, [file_name]003,...
[overwrite]
A logical flag, the default is .false., i.e., existing Array data may not be overwritten. If .true., only the data corresponding to each Array's name will be be overwritten. If the timeslice option is given, only data for the given timeslice may be overwritten. Note that it is always an error to attempt to overwrite a NetCDF variable with data which has a different shape.
[status]
The file status. Please see Section 54.22 for the list of options. If not present, defaults to ESMF_FILESTATUS_UNKNOWN.
[timeslice]
Some I/O formats (e.g. NetCDF) support the output of data in form of time slices. The timeslice argument provides access to this capability. timeslice must be positive. The behavior of this option may depend on the setting of the overwrite flag:
overwrite = .false.:
If the timeslice value is less than the maximum time already in the file, the write will fail.
overwrite = .true.:
Any positive timeslice value is valid.
By default, i.e. by omitting the timeslice argument, no provisions for time slicing are made in the output file, however, if the file already contains a time axis for the variable, a timeslice one greater than the maximum will be written.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28 Array Class

28.1 Description

The Array class is an alternative to the Field class for representing distributed, structured data. Unlike Fields, which are built to carry grid coordinate information, Arrays only carry information about the indices associated with grid cells. Since they do not have coordinate information, Arrays cannot be used to calculate interpolation weights. However, if the user supplies interpolation weights, the Array sparse matrix multiply (SMM) operation can be used to apply the weights and transfer data to the new grid. Arrays carry enough information to perform redistribution, scatter, and gather communication operations.

Like Fields, Arrays can be added to a State and used in inter-Component data communications. Arrays can also be grouped together into ArrayBundles, allowing operations to be performed collectively on the whole group. One motivation for this is convenience; another is the ability to schedule optimized, collective data transfers.

From a technical standpoint, the ESMF Array class is an index space based, distributed data storage class. Its purpose is to hold distributed user data. Each decompositon element (DE) is associated with its own memory allocation. The index space relationship between DEs is described by the ESMF DistGrid class. DEs, and their associated memory allocation, are pinned either to a specific perisistent execution thread (PET), virtual address space (VAS), or a single system image (SSI). This aspect is managed by the ESMF DELayout class. Pinning to PET is the most common mode and is the default.

The Array class offers common communication patterns within the index space formalism. All RouteHandle based communication methods of the Field, FieldBundle, and ArrayBundle layers are implemented via the Array SMM operation.

28.2 Use and Examples

An ESMF_Array is a distributed object that must exist on all PETs of the current context. Each PET-local instance of an Array object contains memory allocations for all PET-local DEs. There may be 0, 1, or more DEs per PET and the number of DEs per PET can differ between PETs for the same Array object. Memory allocations may be provided for each PET by the user during Array creation or can be allocated as part of the Array create call. Many of the concepts of the ESMF_Array class are illustrated by the following examples.


28.2.1 Array from native Fortran array with 1 DE per PET

The create call of the ESMF_Array class has been overloaded extensively to facilitate the need for generality while keeping simple cases simple. The following program demonstrates one of the simpler cases, where existing local Fortran arrays are to be used to provide the PET-local memory allocations for the Array object.

program ESMF_ArrayFarrayEx

  use ESMF
  use ESMF_TestMod
  
  implicit none

The Fortran language provides a variety of ways to define and allocate an array. Actual Fortran array objects must either be explicit-shape or deferred-shape. In the first case the memory allocation and deallocation is automatic from the user's perspective and the details of the allocation (static or dynamic, heap or stack) are left to the compiler. (Compiler flags may be used to control some of the details). In the second case, i.e. for deferred-shape actual objects, the array definition must include the pointer or allocatable attribute and it is the user's responsibility to allocate memory. While it is also the user's responsibility to deallocate memory for arrays with the pointer attribute the compiler will automatically deallocate allocatable arrays under certain circumstances defined by the Fortran standard.

The ESMF_ArrayCreate() interface has been written to accept native Fortran arrays of any flavor as a means to allow user-controlled memory management. The Array create call will check on each PET if sufficient memory has been provided by the specified Fortran arrays and will indicate an error if a problem is detected. However, the Array create call cannot validate the lifetime of the provided memory allocations. If, for instance, an Array object was created in a subroutine from an automatic explicit-shape array or an allocatable array, the memory allocations referenced by the Array object will be automatically deallocated on return from the subroutine unless provisions are made by the application writer to prevent such behavior. The Array object cannot control when memory that has been provided by the user during Array creation becomes deallocated, however, the Array will indicate an error if its memory references have been invalidated.

The easiest, portable way to provide safe native Fortran memory allocations to Array create is to use arrays with the pointer attribute. Memory allocated for an array pointer will not be deallocated automatically. However, in this case the possibility of memory leaks becomes an issue of concern. The deallocation of memory provided to an Array in form of a native Fortran allocation will remain the users responsibility.

None of the concerns discussed above are an issue in this example where the native Fortran array farray is defined in the main program. All different types of array memory allocation are demonstrated in this example. First farrayE is defined as a 2D explicit-shape array on each PET which will automatically provide memory for $10\times 10$ elements.

  ! local variables
  real(ESMF_KIND_R8)       :: farrayE(10,10)  ! explicit shape Fortran array

Then an allocatable array farrayA is declared which will be used to show user-controlled dynamic memory allocation.

  real(ESMF_KIND_R8), allocatable :: farrayA(:,:) ! allocatable Fortran array

Finally an array with pointer attribute farrayP is declared, also used for user-controlled dynamic memory allocation.

  real(ESMF_KIND_R8), pointer :: farrayP(:,:)   ! Fortran array pointer

A matching array pointer must also be available to gain access to the arrays held by an Array object.

  real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching Fortran array ptr 
  type(ESMF_DistGrid)         :: distgrid       ! DistGrid object
  type(ESMF_Array)            :: array          ! Array object
  integer                     :: rc

  call ESMF_Initialize(defaultlogfilename="ArrayFarrayEx.Log", &
                    logkindflag=ESMF_LOGKIND_MULTI, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

On each PET farrayE can be accessed directly to initialize the entire PET-local array.

  farrayE = 12.45d0 ! initialize to some value

In order to create an Array object a DistGrid must first be created that describes the total index space and how it is decomposed and distributed. In the simplest case only the minIndex and maxIndex of the total space must be provided.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)

This example is assumed to run on 4 PETs. The default 2D decomposition will then be into 4 x 1 DEs as to ensure 1 DE per PET.

Now the Array object can be created using the farrayE and the DistGrid just created.

  array = ESMF_ArrayCreate(farray=farrayE, distgrid=distgrid, &
    indexflag=ESMF_INDEX_DELOCAL, rc=rc)

The 40 x 10 index space defined by the minIndex and maxIndex arguments paired with the default decomposition will result in the following distributed Array.

  
         +---------------------------> 2nd dimension
         |   (1,1)-------+
         |     |         |
         |     |   DE 0  |   <--- farray on PET 0
         |     |         |
         |     +------(10,10)
         |  (11,1)-------+
         |     |         |
         |     |   DE 1  |   <--- farray on PET 1
         |     |         |
         |     +------(20,10)
         |  (21,1)-------+
         |     |         |
         |     |   DE 2  |   <--- farray on PET 2
         |     |         |
         |     +------(30,10)
         |  (31,1)-------+
         |     |         |
         |     |   DE 3  |   <--- farray on PET 3
         |     |         |
         |     +------(40,10)
         v
       1st dimension

Providing farrayE during Array creation does not change anything about the actual farrayE object. This means that each PET can use its local farrayE directly to access the memory referenced by the Array object.

  print *, farrayE

Another way of accessing the memory associated with an Array object is to use ArrayGet() to obtain an Fortran pointer that references the PET-local array.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)

  print *, farrayPtr

Finally the Array object must be destroyed. The PET-local memory of the farrayEs will remain in user control and will not be altered by ArrayDestroy().

  call ESMF_ArrayDestroy(array, rc=rc)

Since the memory allocation for each farrayE is automatic there is nothing more to do.

The interaction between farrayE and the Array class is representative also for the two other cases farrayA and farrayP. The only difference is in the handling of memory allocations.

  allocate(farrayA(10,10))    ! user controlled allocation
  farrayA = 23.67d0           ! initialize to some value
  array = ESMF_ArrayCreate(farray=farrayA, distgrid=distgrid, &
    indexflag=ESMF_INDEX_DELOCAL, rc=rc)

  print *, farrayA            ! print PET-local farrayA directly
  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)! obtain array pointer
  print *, farrayPtr          ! print PET-local piece of Array through pointer
  call ESMF_ArrayDestroy(array, rc=rc) ! destroy the Array
  deallocate(farrayA)         ! user controlled de-allocation

The farrayP case is identical.

  allocate(farrayP(10,10))    ! user controlled allocation
  farrayP = 56.81d0           ! initialize to some value
  array = ESMF_ArrayCreate(farray=farrayP, distgrid=distgrid, &
    indexflag=ESMF_INDEX_DELOCAL, rc=rc)

  print *, farrayP            ! print PET-local farrayA directly
  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)! obtain array pointer
  print *, farrayPtr          ! print PET-local piece of Array through pointer
  call ESMF_ArrayDestroy(array, rc=rc) ! destroy the Array
  deallocate(farrayP)         ! user controlled de-allocation

To wrap things up the DistGrid object is destroyed and ESMF can be finalized.

  call ESMF_DistGridDestroy(distgrid, rc=rc) ! destroy the DistGrid

  call ESMF_Finalize(rc=rc)

end program


28.2.2 Array from native Fortran array with extra elements for halo or padding

The example of the previous section showed how easy it is to create an Array object from existing PET-local Fortran arrays. The example did, however, not define any halo elements around the DE-local regions. The following code demonstrates how an Array object with space for a halo can be set up.

program ESMF_ArrayFarrayHaloEx

  use ESMF
  use ESMF_TestMod
  
  implicit none

The allocatable array farrayA will be used to provide the PET-local Fortran array for this example.

  ! local variables
  real(ESMF_KIND_R8), allocatable :: farrayA(:,:) ! allocatable Fortran array
  real(ESMF_KIND_R8), pointer :: farrayPtr(:,:)   ! matching Fortran array ptr
  type(ESMF_DistGrid)         :: distgrid         ! DistGrid object
  type(ESMF_Array)            :: array            ! Array object
  integer                     :: rc, i, j
  real(ESMF_KIND_R8)          :: localSum

  call ESMF_Initialize(defaultlogfilename="ArrayFarrayHaloEx.Log", &
                    logkindflag=ESMF_LOGKIND_MULTI, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

The Array is to cover the exact same index space as in the previous example. Furthermore decomposition and distribution are also kept the same. Hence the same DistGrid object will be created and it is expected to execute this example with 4 PETs.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)

This DistGrid describes a 40 x 10 index space that will be decomposed into 4 DEs when executed on 4 PETs, associating 1 DE per PET. Each DE-local exclusive region contains 10 x 10 elements. The DistGrid also stores and provides information about the relationship between DEs in index space, however, DistGrid does not contain information about halos. Arrays contain halo information and it is possible to create multiple Arrays covering the same index space with identical decomposition and distribution using the same DistGrid object, while defining different, Array-specific halo regions.

The extra memory required to cover the halo in the Array object must be taken into account when allocating the PET-local farrayA arrays. For a halo of 2 elements in each direction the following allocation will suffice.

  allocate(farrayA(14,14))    ! Fortran array with halo: 14 = 10 + 2 * 2

The farrayA can now be used to create an Array object with enough space for a two element halo in each direction. The Array creation method checks for each PET that the local Fortran array can accommodate the requested regions.

The default behavior of ArrayCreate() is to center the exclusive region within the total region. Consequently the following call will provide the 2 extra elements on each side of the exclusive 10 x 10 region without having to specify any additional arguments.

  array = ESMF_ArrayCreate(farray=farrayA, distgrid=distgrid, &
    indexflag=ESMF_INDEX_DELOCAL, rc=rc)

The exclusive Array region on each PET can be accessed through a suitable Fortran array pointer. See section 28.2.6 for more details on Array regions.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)

Following Array bounds convention, which by default puts the beginning of the exclusive region at (1, 1, ...), the following loop will add up the values of the local exclusive region for each DE, regardless of how the bounds were chosen for the original PET-local farrayA arrays.

  localSum = 0.
  do j=1, 10
    do i=1, 10
      localSum = localSum + farrayPtr(i, j)
    enddo
  enddo

Elements with $i$ or $j$ in the [-1,0] or [11,12] ranges are located outside the exclusive region and may be used to define extra computational points or halo operations.

Cleanup and shut down ESMF.

  call ESMF_ArrayDestroy(array, rc=rc)

  deallocate(farrayA)
  call ESMF_DistGridDestroy(distgrid, rc=rc)

  call ESMF_Finalize(rc=rc)

end program


28.2.3 Array from ESMF_LocalArray

Alternative to the direct usage of Fortran arrays during Array creation it is also possible to first create an ESMF_LocalArray and create the Array from it. While this may seem more burdensome for the 1 DE per PET cases discussed in the previous sections it allows a straightforward generalization to the multiple DE per PET case. The following example first recaptures the previous example using an ESMF_LocalArray and then expands to the multiple DE per PET case.

program ESMF_ArrayLarrayEx

  use ESMF
  use ESMF_TestMod
  
  implicit none

The current ESMF_LocalArray interface requires Fortran arrays to be defined with pointer attribute.

  ! local variables
  real(ESMF_KIND_R8), pointer :: farrayP(:,:)   ! Fortran array pointer
  real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching Fortran array ptr 
  type(ESMF_LocalArray)       :: larray         ! ESMF_LocalArray object
  type(ESMF_LocalArray)       :: larrayRef      ! ESMF_LocalArray object
  type(ESMF_DistGrid)         :: distgrid       ! DistGrid object
  type(ESMF_Array)            :: array          ! Array object
  integer                     :: rc, i, j, de
  real(ESMF_KIND_R8)          :: localSum
  type(ESMF_LocalArray), allocatable :: larrayList(:) ! LocalArray object list
  type(ESMF_LocalArray), allocatable :: larrayRefList(:)!LocalArray obj. list
  
  type(ESMF_VM):: vm
  integer:: localPet, petCount

  call ESMF_Initialize(vm=vm, defaultlogfilename="ArrayLarrayEx.Log", &
                    logkindflag=ESMF_LOGKIND_MULTI, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  call ESMF_VMGet(vm, localPet=localPet, petCount=petCount, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  
  if (petCount /= 4) then
    finalrc = ESMF_FAILURE
    goto 10
  endif

DistGrid and array allocation remains unchanged.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)

  allocate(farrayP(14,14))    ! allocate Fortran array on each PET with halo

Now instead of directly creating an Array object using the PET-local farrayPs an ESMF_LocalArray object will be created on each PET.

  larray = ESMF_LocalArrayCreate(farrayP, &
               datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)

The Array object can now be created from larray. The Array creation method checks for each PET that the LocalArray can accommodate the requested regions.

  array = ESMF_ArrayCreate(localarrayList=(/larray/), distgrid=distgrid, rc=rc)

Once created there is no difference in how the Array object can be used. The exclusive Array region on each PET can be accessed through a suitable Fortran array pointer as before.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)

Alternatively it is also possible (independent of how the Array object was created) to obtain the reference to the array allocation held by Array in form of an ESMF_LocalArray object. The farrayPtr can then be extracted using LocalArray methods.

  call ESMF_ArrayGet(array, localarray=larrayRef, rc=rc)

  call ESMF_LocalArrayGet(larrayRef, farrayPtr, rc=rc)

Either way the farrayPtr reference can be used now to add up the values of the local exclusive region for each DE. The following loop works regardless of how the bounds were chosen for the original PET-local farrayP arrays and consequently the PET-local larray objects.

  localSum = 0.
  do j=1, 10
    do i=1, 10
      localSum = localSum + farrayPtr(i, j)
    enddo
  enddo
  print *, "localSum=", localSum

Cleanup.

  call ESMF_ArrayDestroy(array, rc=rc)
  call ESMF_LocalArrayDestroy(larray, rc=rc)
  deallocate(farrayP)   ! use the pointer that was used in allocate statement
  call ESMF_DistGridDestroy(distgrid, rc=rc)

While the usage of LocalArrays is unnecessarily cumbersome for 1 DE per PET Arrays, it provides a straightforward path for extending the interfaces to multiple DEs per PET.

In the following example a 8 x 8 index space will be decomposed into 2 x 4 = 8 DEs. The situation is captured by the following DistGrid object.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/8,8/), &
    regDecomp=(/2,4/), rc=rc)

The distgrid object created in this manner will contain 8 DEs no matter how many PETs are available during execution. Assuming an execution on 4 PETs will result in the following distribution of the decomposition.

   
    +---------------------------------------> 2nd dimension
    |  (1,1)
    |    +-----------+-----------+-----------+-----------+
    |    | DE0, PET0 | DE2, PET1 | DE4, PET2 | DE6, PET3 |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    |           |           |           |           |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    |           |           |           |           |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    |           |           |           |           |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    +-----------+-----------+-----------+-----------+
    |    | DE1, PET0 | DE3, PET1 | DE5, PET2 | DE7, PET3 |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    |           |           |           |           |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    |           |           |           |           |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    |           |           |           |           |
    |    |  *    *   |  *    *   |  *    *   |  *    *   |
    |    +-----------+-----------+-----------+-----------+
    |                                                    (8,8)
    v 
   1st dimension

Obviously each PET is associated with 2 DEs. Each PET must allocate enough space for all its DEs. This is done by allocating as many DE-local arrays as there are DEs on the PET. The reference to these array allocations is passed into ArrayCreate via a LocalArray list argument that holds as many elements as there are DEs on the PET. Here each PET must allocate for two DEs.

  allocate(larrayList(2))   ! 2 DEs per PET
  allocate(farrayP(4, 2))   ! without halo each DE is of size 4 x 2 
  farrayP = 123.456d0
  larrayList(1) = ESMF_LocalArrayCreate(farrayP, &
    datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc) !1st DE
  allocate(farrayP(4, 2))   ! without halo each DE is of size 4 x 2 
  farrayP = 456.789d0
  larrayList(2) = ESMF_LocalArrayCreate(farrayP, &
    datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc) !2nd DE

Notice that it is perfectly fine to re-use farrayP for all allocations of DE-local Fortran arrays. The allocated memory can be deallocated at the end using the array pointer contained in the larrayList.

With this information an Array object can be created. The distgrid object indicates 2 DEs for each PET and ArrayCreate() expects to find two LocalArray elements in larrayList.

  array = ESMF_ArrayCreate(localarrayList=larrayList, distgrid=distgrid, rc=rc)

Usage of a LocalArray list is the only way to provide a list of variable length of Fortran array allocations to ArrayCreate() for each PET. The array object created by the above call is an ESMF distributed object. As such it must follow the ESMF convention that requires that the call to ESMF_ArrayCreate() must be issued in unison by all PETs of the current context. Each PET only calls ArrayCreate() once, even if there are multiple DEs per PET.

The ArrayGet() method provides access to the list of LocalArrays on each PET.

  allocate(larrayRefList(2))
  call ESMF_ArrayGet(array, localarrayList=larrayRefList, rc=rc)

Finally, access to the actual Fortran pointers is done on a per DE basis. Generally each PET will loop over its DEs.

  do de=1, 2
    call ESMF_LocalArrayGet(larrayRefList(de), farrayPtr, rc=rc)
    localSum = 0.
    do j=1, 2
      do i=1, 4
        localSum = localSum + farrayPtr(i, j)
      enddo
    enddo
    print *, "localSum=", localSum
  enddo

Note: If the VM associates multiple PEs with a PET the application writer may decide to use OpenMP loop parallelization on the de loop.

Cleanup requires that the PET-local deallocations are done before the pointers to the actual Fortran arrays are lost. Notice that larrayList is used to obtain the pointers used in the deallocate statement. Pointers obtained from the larrayRefList, while pointing to the same data, cannot be used to deallocate the array allocations!

  do de=1, 2
    call ESMF_LocalArrayGet(larrayList(de), farrayPtr, rc=rc)

    deallocate(farrayPtr)
    call ESMF_LocalArrayDestroy(larrayList(de), rc=rc)

  enddo
  deallocate(larrayList)
  deallocate(larrayRefList)
  call ESMF_ArrayDestroy(array, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  call ESMF_DistGridDestroy(distgrid, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

With that ESMF can be shut down cleanly.

  call ESMF_Finalize(rc=rc)

end program

28.2.4 Create Array with automatic memory allocation

In the examples of the previous sections the user provided memory allocations for each of the DE-local regions for an Array object. The user was able to use any of the Fortran methods to allocate memory, or go through the ESMF_LocalArray interfaces to obtain memory allocations before passing them into ArrayCreate(). Alternatively ESMF offers methods that handle Array memory allocations inside the library.

As before, to create an ESMF_Array object an ESMF_DistGrid must be created. The DistGrid object holds information about the entire index space and how it is decomposed into DE-local exclusive regions. The following line of code creates a DistGrid for a 5x5 global index space that is decomposed into 2 x 3 = 6 DEs.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), rc=rc)

The following is a representation of the index space and its decomposition into DEs. Each asterisk (*) represents a single element.

   
    +---------------------------------------> 2nd dimension
    |  (1,1)
    |    +-----------+-----------+------+
    |    | DE 0      | DE 2      | DE 4 |
    |    |           |           |      |
    |    |  *    *   |  *    *   |  *   |
    |    |           |           |      |
    |    |  *    *   |  *    *   |  *   |
    |    |           |           |      |
    |    |  *    *   |  *    *   |  *   |
    |    +-----------+-----------+------+
    |    |           |           |      |
    |    | DE 1      | DE 3      | DE 5 |
    |    |           |           |      |
    |    |  *    *   |  *    *   |  *   |
    |    |           |           |      |
    |    |  *    *   |  *    *   |  *   |
    |    +-----------+-----------+------+
    |                                 (5,5)
    v 
   1st dimension

Besides the DistGrid it is the type, kind and rank information, "tkr" for short, that is required to create an Array object. It turns out that the rank of the Array object is fully determined by the DistGrid and other (optional) arguments passed into ArrayCreate(), so that explicit specification of the Array rank is redundant.

The simplest way to supply the type and kind information of the Array is directly through the typekind argument. Here a double precision Array is created on the previously created DistGrid. Since no other arguments are specified that could alter the rank of the Array it becomes equal to the dimCount of the DistGrid, i.e a 2D Array is created on top of the DistGrid.

  array = ESMF_ArrayCreate(typekind=ESMF_TYPEKIND_R8, distgrid=distgrid, rc=rc)

The different methods on how an Array object is created have no effect on the use of ESMF_ArrayDestroy().

  call ESMF_ArrayDestroy(array, rc=rc)

Alternatively the same Array can be created specifying the "tkr" information in form of an ArraySpec variable. The ArraySpec explicitly contains the Array rank and thus results in an over specification on the ArrayCreate() interface. ESMF checks all input information for consistency and returns appropriate error codes in case any inconsistencies are found.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)

The Array object created by the above call is an ESMF distributed object. As such it must follow the ESMF convention that requires that the call to ESMF_ArrayCreate() must be issued in unison by all PETs of the current context.


28.2.5 Native language memory access

There are two different methods by which the user can access the data held inside an ESMF Array object. The first method provides direct access to a native language array object. Specifically, the farrayPtr argument returned by ESMF_ArrayGet() is a Fortran array pointer that can be used to access the PET-local data inside the Array object.

Many applications work in the 1 DE per PET mode, with exactly one DE on every PET. Accessing the Array memory on each PET for this situation is especially simple as is shown in section 28.2.1. However, the Array class is not restricted to the special 1 DE per PET case, but supports multiple separate memory allocations on each PET. The number of such PET-local allocations is given by the localDeCount, i.e. there is one memory allocation for every DE that is associated with the local PET.

Access to a specific local memory allocation of an Array object is still accomplished by returning the farrayPtr argument. However, for $localDeCount > 1 $ the formally optional localDe argument to ESMF_ArrayGet() turns into a practically required argument. While in general the localDe in ESMF is simply a local index variable that enumerates the DEs that are associated with the local PET (e.g. see section 50.3.7), the bounds of this index variable are strictly defined as [0,...,localDeCount-1] when it is used as an input argument. The following code demonstrates this.

First query the Array for localDeCount. This number may be different on each PET and indicates how many DEs are mapped against the local PET.

  call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc)

Looping the localDe index variable from 0 to localDeCount-1 allows access to each of the local memory allocations of the Array object:

  do localDe=0, localDeCount-1
    call ESMF_ArrayGet(array, farrayPtr=myFarray, localDe=localDe, rc=rc)

    ! use myFarray to access local DE data
  enddo

The second method to access the memory allocations in an Array object is to go through the ESMF LocalArray object. To this end the Array is queried for a list of PET-local LocalArray objects. The LocalArray objects in the list correspond to the DEs on the local PET. Here the localDe argument is solely a user level index variable, and in principle the lower bound can be chosen freely. However, for better alignment with the previous case (where localDe served as an input argument to an ESMF method) the following example again fixes the lower bound at zero.

  allocate(larrayList(0:localDeCount-1))
  call ESMF_ArrayGet(array, localarrayList=larrayList, rc=rc)

  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
       datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)

    ! use myFarray to access local DE data
  enddo

See section 28.2.3 for more on LocalArray usage in Array. In most cases memory access through a LocalArray list is less convenient than the direct farrayPtr method because it adds an extra object level between the ESMF Array and the native language array.


28.2.6 Regions and default bounds

Each ESMF_Array object is decomposed into DEs as specified by the associated ESMF_DistGrid object. Each piece of this decomposition, i.e. each DE, holds a chunk of the Array data in its own local piece of memory. The details of the Array decomposition are described in the following paragraphs.

At the center of the Array decomposition is the ESMF_DistGrid class. The DistGrid object specified during Array creation contains three essential pieces of information:

Each element of an Array is associated with a single DE. The union of elements associated with a DE, as defined by the DistGrid above, corresponds to a LR chunk of index space, called the exclusive region of the DE.

There is a hierarchy of four regions that can be identified for each DE in an Array object. Their definition and relationship to each other is as follows:

  
     +-totalLBound(:)----------------------------------+
     |\                                                |
     | \ <--- totalLWidth(:)                           |
     |  \                                              |
     |   +-computationalLBound(:)------------------+   |
     |   |\                                        |   |
     |   | \ <--- computationalLWidth(:)           |   |
     |   |  \                                      |   |
     |   |   +-exclusiveLBound(:)-------------+    |   |
     |   |   |                                |    |   |
     |   |   |     +------+      +-----+      |    |   |
     |   |   |     |      |      |     |      |    |   |
     |   |   |     |      +------+     |      |    |   |
     |   |   |     | "Interior Region" |      |    |   |
     |   |   |     +-----+             |      |    |   |
     |   |   |           |             |      |    |   |
     |   |   |           +-------------+      |    |   |
     |   |   |                                |    |   |
     |   |   | "Exclusive Region"             |    |   |
     |   |   +-------------exclusiveUBound(:)-+    |   |
     |   |                                     \   |   |
     |   |           computationalUWidth(:) --> \  |   |
     |   |                                       \ |   |
     |   | "Computational Region"                 \|   |
     |   +------------------computationalUBound(:)-+   |
     |                                              \  | 
     |                             totalUWidth(:) -> \ | 
     | "Total Region"                                 \| 
     +--------------------------------- totalUBound(:)-+

With the following definitions:

  
   computationalLWidth(:) = exclusiveLBound(:) - computationalLBound(:)
   computationalUWidth(:) = computationalUBound(:) - exclusiveUBound(:)
and
  
   totalLWidth(:) = exclusiveLBound(:) - totalLBound(:)
   totalUWidth(:) = totalUBound(:) - exclusiveUBound(:)

The exclusive region is determined during Array creation by the DistGrid argument. Optional arguments may be used to specify the computational region when the Array is created, by default it will be set equal to the exclusive region. The total region, i.e. the actual memory allocation for each DE, is also determined during Array creation. When creating the Array object from existing Fortran arrays the total region is set equal to the memory provided by the Fortran arrays. Otherwise the default is to allocate as much memory as is needed to accommodate the union of the DE-local exclusive and computational region. Finally it is also possible to use optional arguments to the ArrayCreate() call to specify the total region of the object explicitly.

The ESMF_ArrayCreate() call checks that the input parameters are consistent and will result in an Array that fulfills all of the above mentioned requirements for its DE-local regions.

Once an Array object has been created the exclusive and total regions are fixed. The computational region, however, may be adjusted within the limits of the total region using the ArraySet() call.

The interior region is very different from the other regions in that it cannot be specified. The interior region for each DE is a consequence of the choices made for the other regions collectively across all DEs into which an Array object is decomposed. An Array object can be queried for its DE-local interior regions as to offer additional information to the user necessary to write more efficient code.

By default the bounds of each DE-local total region are defined as to put the start of the DE-local exclusive region at the "origin" of the local index space, i.e. at (1, 1, ..., 1). With that definition the following loop will access each element of the DE-local memory segment for each PET-local DE of the Array object used in the previous sections and print its content.

  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
       datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
    do i=1, size(myFarray, 1)
      do j=1, size(myFarray, 2)
        print *, "localPET=", localPet, " localDE=", &
            localDe, ": array(",i,",",j,")=", myFarray(i,j)
      enddo
    enddo
  enddo

28.2.7 Array bounds

The loop over Array elements at the end of the last section only works correctly because of the default definition of the computational and total regions used in the example. In general, without such specific knowledge about an Array object, it is necessary to use a more formal approach to access its regions with DE-local indices.

The DE-local exclusive region takes a central role in the definition of Array bounds. Even as the computational region may adjust during the course of execution the exclusive region remains unchanged. The exclusive region provides a unique reference frame for the index space of all Arrays associated with the same DistGrid.

There is a choice between two indexing options that needs to be made during Array creation. By default each DE-local exclusive region starts at (1, 1, ..., 1). However, for some computational kernels it may be more convenient to choose the index bounds of the DE-local exclusive regions to match the index space coordinates as they are defined in the corresponding DistGrid object. The second option is only available if the DistGrid object does not contain any non-contiguous decompositions (such as cyclically decomposed dimensions).

The following example code demonstrates the safe way of dereferencing the DE-local exclusive regions of the previously created array object.

  allocate(exclusiveUBound(2, 0:localDeCount-1))  ! dimCount=2
  allocate(exclusiveLBound(2, 0:localDeCount-1))  ! dimCount=2
  call ESMF_ArrayGet(array, indexflag=indexflag, &
    exclusiveLBound=exclusiveLBound, exclusiveUBound=exclusiveUBound, rc=rc)
  if (indexflag == ESMF_INDEX_DELOCAL) then
    ! this is the default
!    print *, "DE-local exclusive regions start at (1,1)"
    do localDe=0, localDeCount-1
      call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
          datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
      do i=1, exclusiveUBound(1, localDe)
        do j=1, exclusiveUBound(2, localDe)
!          print *, "DE-local exclusive region for localDE=", localDe, &
!            ": array(",i,",",j,")=", myFarray(i,j)
        enddo
      enddo
    enddo
  else if (indexflag == ESMF_INDEX_GLOBAL) then
    ! only if set during ESMF_ArrayCreate()
!    print *, "DE-local exclusive regions of this Array have global bounds"
    do localDe=0, localDeCount-1
      call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
         datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
      do i=exclusiveLBound(1, localDe), exclusiveUBound(1, localDe)
        do j=exclusiveLBound(2, localDe), exclusiveUBound(2, localDe)
!          print *, "DE-local exclusive region for localDE=", localDe, &
!            ": array(",i,",",j,")=", myFarray(i,j)
        enddo
      enddo
    enddo
  endif
  call ESMF_ArrayDestroy(array, rc=rc) ! destroy the array object

Obviously the second branch of this simple code will work for either case, however, if a complex computational kernel was written assuming ESMF_INDEX_DELOCAL type bounds the second branch would simply be used to indicate the problem and bail out.

The advantage of the ESMF_INDEX_GLOBAL index option is that the Array bounds directly contain information on where the DE-local Array piece is located in a global index space sense. When the ESMF_INDEX_DELOCAL option is used the correspondence between local and global index space must be made by querying the associated DistGrid for the DE-local indexList arguments.


28.2.8 Computational region and extra elements for halo or padding

In the previous examples the computational region of array was chosen by default to be identical to the exclusive region defined by the DistGrid argument during Array creation. In the following the same arrayspec and distgrid objects as before will be used to create an Array but now a larger computational region shall be defined around each DE-local exclusive region. Furthermore, extra space will be defined around the computational region of each DE to accommodate a halo and/or serve as memory padding.

In this example the indexflag argument is set to ESMF_INDEX_GLOBAL indicating that the bounds of the exclusive region correspond to the index space coordinates as they are defined by the DistGrid object.

The same arrayspec and distgrid objects as before are used which also allows the reuse of the already allocated larrayList variable.

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    computationalLWidth=(/0,3/), computationalUWidth=(/1,1/), &
    totalLWidth=(/1,4/), totalUWidth=(/3,1/), &
    indexflag=ESMF_INDEX_GLOBAL, rc=rc)

Obtain the larrayList on every PET.

  allocate(localDeToDeMap(0:localDeCount-1))
  call ESMF_ArrayGet(array, localarrayList=larrayList, &
    localDeToDeMap=localDeToDeMap, rc=rc)

The bounds of DE 1 for array are shown in the following diagram to illustrate the situation. Notice that the totalLWidth and totalUWidth arguments in the ArrayCreate() call define the total region with respect to the exclusive region given for each DE by the distgrid argument.

        +-(3,-3)---------------------------------+ 
        |\                                       | 
        | +-(4,-2)-+-(4,1)--------------------+--+ 
        | |        |                          |  | 
        | |        |                          |  | 
        | |        |          DE 1            |  | 
        | |        |                          |  | 
        | |        |                          |  | 
        | |        | Exclusive Region         |  | 
        | |        +--------------------(5,2)-+  | 
        | | Computational Region                 | 
        | +-------------------------------(6,3)--+ 
        |                                        | 
        | Total Region                           | 
        +---------------------------------(8,3)--+

When working with this array it is possible for the computational kernel to overstep the exclusive region for both read/write access (computational region) and potentially read-only access into the total region outside of the computational region, if a halo operation provides valid entries for these elements.

The Array object can be queried for absolute bounds

  allocate(computationalLBound(2, 0:localDeCount-1))  ! dimCount=2
  allocate(computationalUBound(2, 0:localDeCount-1))  ! dimCount=2
  allocate(totalLBound(2, 0:localDeCount-1))          ! dimCount=2
  allocate(totalUBound(2, 0:localDeCount-1))          ! dimCount=2
  call ESMF_ArrayGet(array, exclusiveLBound=exclusiveLBound, &
    exclusiveUBound=exclusiveUBound, &
    computationalLBound=computationalLBound, &
    computationalUBound=computationalUBound, &
    totalLBound=totalLBound, &
    totalUBound=totalUBound, rc=rc)

or for the relative widths.

  allocate(computationalLWidth(2, 0:localDeCount-1))  ! dimCount=2
  allocate(computationalUWidth(2, 0:localDeCount-1))  ! dimCount=2
  allocate(totalLWidth(2, 0:localDeCount-1))          ! dimCount=2
  allocate(totalUWidth(2, 0:localDeCount-1))          ! dimCount=2
  call ESMF_ArrayGet(array, computationalLWidth=computationalLWidth, &
    computationalUWidth=computationalUWidth, totalLWidth=totalLWidth, &
    totalUWidth=totalUWidth, rc=rc)

Either way the dereferencing of Array data is centered around the DE-local exclusive region:

  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
       datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
    ! initialize the DE-local array
    myFarray = 0.1d0 * localDeToDeMap(localDe)
    ! first time through the total region of array    
!    print *, "myFarray bounds for DE=", localDeToDeMap(localDe), &
!      lbound(myFarray),  ubound(myFarray)
    do j=exclusiveLBound(2, localDe), exclusiveUBound(2, localDe)
      do i=exclusiveLBound(1, localDe), exclusiveUBound(1, localDe)
!        print *, "Excl region DE=", localDeToDeMap(localDe), &
!        ": array(",i,",",j,")=",  myFarray(i,j)
      enddo
    enddo
    do j=computationalLBound(2, localDe), computationalUBound(2, localDe)
      do i=computationalLBound(1, localDe), computationalUBound(1, localDe)
!        print *, "Excl region DE=", localDeToDeMap(localDe), &
!        ": array(",i,",",j,")=", myFarray(i,j)
      enddo
    enddo
    do j=totalLBound(2, localDe), totalUBound(2, localDe)
      do i=totalLBound(1, localDe), totalUBound(1, localDe)
!        print *, "Total region DE=", localDeToDeMap(localDe), &
!        ": array(",i,",",j,")=", myFarray(i,j)
      enddo
    enddo

    ! second time through the total region of array    
    do j=exclusiveLBound(2, localDe)-totalLWidth(2, localDe), &
      exclusiveUBound(2, localDe)+totalUWidth(2, localDe)
      do i=exclusiveLBound(1, localDe)-totalLWidth(1, localDe), &
        exclusiveUBound(1, localDe)+totalUWidth(1, localDe)
!        print *, "Excl region DE=", localDeToDeMap(localDe), &
!        ": array(",i,",",j,")=", myFarray(i,j)
      enddo
    enddo
  enddo

28.2.9 Create 1D and 3D Arrays

All previous examples were written for the 2D case. There is, however, no restriction within the Array or DistGrid class that limits the dimensionality of Array objects beyond the language-specific limitations (7D for Fortran).

In order to create an n-dimensional Array the rank indicated by both the arrayspec and the distgrid arguments specified during Array create must be equal to n. A 1D Array of double precision real data hence requires the following arrayspec.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=1, rc=rc)

The index space covered by the Array and the decomposition description is provided to the Array create method by the distgrid argument. The index space in this example has 16 elements and covers the interval $[-10, 5]$. It is decomposed into as many DEs as there are PETs in the current context.

  distgrid1D = ESMF_DistGridCreate(minIndex=(/-10/), maxIndex=(/5/), &
    regDecomp=(/petCount/), rc=rc)

A 1D Array object with default regions can now be created.

  array1D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid1D, rc=rc)

The creation of a 3D Array proceeds analogous to the 1D case. The rank of the arrayspec must be changed to 3

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)

and an appropriate 3D DistGrid object must be created

  distgrid3D = ESMF_DistGridCreate(minIndex=(/1,1,1/), &
    maxIndex=(/16,16,16/), regDecomp=(/4,4,4/), rc=rc)

before an Array object can be created.

  array3D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid3D, rc=rc)

The distgrid3D object decomposes the 3-dimensional index space into $4\times 4\times 4 = 64$ DEs. These DEs are laid out across the computational resources (PETs) of the current component according to a default DELayout that is created during the DistGrid create call. Notice that in the index space proposal a DELayout does not have a sense of dimensionality. The DELayout function is simply to map DEs to PETs. The DistGrid maps chunks of index space against DEs and thus its rank is equal to the number of index space dimensions.

The previously defined DistGrid and the derived Array object decompose the index space along all three dimension. It is, however, not a requirement that the decomposition be along all dimensions. An Array with the same 3D index space could as well be decomposed along just one or along two of the dimensions. The following example shows how for the same index space only the last two dimensions are decomposed while the first Array dimension has full extent on all DEs.

  call ESMF_ArrayDestroy(array3D, rc=rc)
  call ESMF_DistGridDestroy(distgrid3D, rc=rc)
  distgrid3D = ESMF_DistGridCreate(minIndex=(/1,1,1/), &
    maxIndex=(/16,16,16/), regDecomp=(/1,4,4/), rc=rc)
  array3D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid3D, rc=rc)

28.2.10 Working with Arrays of different rank

Assume a computational kernel that involves the array3D object as it was created at the end of the previous section. Assume further that the kernel also involves a 2D Array on a 16x16 index space where each point (j,k) was interacting with each (i,j,k) column of the 3D Array. An efficient formulation would require that the decomposition of the 2D Array must match that of the 3D Array and further the DELayout be identical. The following code shows how this can be accomplished.

  call ESMF_DistGridGet(distgrid3D, delayout=delayout, rc=rc) ! get DELayout
  distgrid2D = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/16,16/), &
    regDecomp=(/4,4/), delayout=delayout, rc=rc)
  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
  array2D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid2D, rc=rc)

Now the following kernel is sure to work with array3D and array2D.

  call ESMF_DELayoutGet(delayout, localDeCount=localDeCount, rc=rc)
  allocate(larrayList1(0:localDeCount-1))
  call ESMF_ArrayGet(array3D, localarrayList=larrayList1, rc=rc)
  allocate(larrayList2(0:localDeCount-1))
  call ESMF_ArrayGet(array2D, localarrayList=larrayList2, rc=rc)
  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList1(localDe), myFarray3D, &
      datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
    myFarray3D = 0.1d0 * localDe ! initialize
    call ESMF_LocalArrayGet(larrayList2(localDe), myFarray2D, &
      datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
    myFarray2D = 0.5d0 * localDe ! initialize
    do k=1, 4
      do j=1, 4
        dummySum = 0.d0
        do i=1, 16
          dummySum = dummySum + myFarray3D(i,j,k) ! sum up the (j,k) column
        enddo
        dummySum = dummySum * myFarray2D(j,k) ! multiply with local 2D element
!        print *, "dummySum(",j,k,")=",dummySum
      enddo
    enddo
  enddo


28.2.11 Arrays with undistributed dimensions

Except for the special Array create interface that implements a copy from an existing Array object all other Array create interfaces require the specification of at least two arguments: farray and distgrid, larrayList and distgrid, or arrayspec and distgrid. In all these cases both required arguments contain a sense of dimensionality. The relationship between these two arguments deserves extra attention.

The first argument, farray, larrayList or arrayspec, determines the rank of the created Array object, i.e. the dimensionality of the actual data storage. The rank of a native language array, extracted from an Array object, is equal to the rank specified by either of these arguments. So is the rank that is returned by the ESMF_ArrayGet() call.

The rank specification contained in the distgrid argument, which is of type ESMF_DistGrid, on the other hand has no effect on the rank of the Array. The dimCount specified by the DistGrid object, which may be equal, greater or less than the Array rank, determines the dimensionality of the decomposition.

While there is no constraint between DistGrid dimCount and Array rank, there is an important relationship between the two, resulting in the concept of index space dimensionality. Array dimensions can be arbitrarily mapped against DistGrid dimension, rendering them decomposed dimensions. The index space dimensionality is equal to the number of decomposed Array dimensions.

Array dimensions that are not mapped to DistGrid dimensions are the undistributed dimensions of the Array. They are not part of the index space. The mapping is specified during ESMF_ArrayCreate() via the distgridToArrayMap argument. DistGrid dimensions that have not been associated with Array dimensions are replicating dimensions. The Array will be replicated across the DEs that lie along replication DistGrid dimensions.

Undistributed Array dimensions can be used to store multi-dimensional data for each Array index space element. One application of this is to store the components of a vector quantity in a single Array. The same 2D distgrid object as before will be used.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), rc=rc)

The rank in the arrayspec argument, however, must change from 2 to 3 in order to provide for the extra Array dimension.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)

During Array creation with extra dimension(s) it is necessary to specify the bounds of these undistributed dimension(s). This requires two additional arguments, undistLBound and undistUBound, which are vectors in order to accommodate multiple undistributed dimensions. The other arguments remain unchanged and apply across all undistributed components.

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    totalLWidth=(/0,1/), totalUWidth=(/0,1/), &
    undistLBound=(/1/), undistUBound=(/2/), rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)

This will create array with 2+1 dimensions. The 2D DistGrid is used to describe decomposition into DEs with 2 Array dimensions mapped to the DistGrid dimensions resulting in a 2D index space. The extra Array dimension provides storage for multi component user data within the Array object.

By default the distgrid dimensions are associated with the first Array dimensions in sequence. For the example above this means that the first 2 Array dimensions are decomposed according to the provided 2D DistGrid. The 3rd Array dimension does not have an associated DistGrid dimension, rendering it an undistributed Array dimension.

Native language access to an Array with undistributed dimensions is in principle the same as without extra dimensions.

  call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc)
  allocate(larrayList(0:localDeCount-1))
  call ESMF_ArrayGet(array, localarrayList=larrayList, rc=rc)

The following loop shows how a Fortran pointer to the DE-local data chunks can be obtained and used to set data values in the exclusive regions. The myFarray3D variable must be of rank 3 to match the Array rank of array. However, variables such as exclusiveUBound that store the information about the decomposition, remain to be allocated for the 2D index space.

  call ESMF_ArrayGet(array, exclusiveLBound=exclusiveLBound, &
    exclusiveUBound=exclusiveUBound, rc=rc)
  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList(localDe), myFarray3D, &
       datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
    myFarray3D = 0.0 ! initialize
    myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
      exclusiveLBound(2,localDe):exclusiveUBound(2,localDe), &
      1) = 5.1 ! dummy assignment
    myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
      exclusiveLBound(2,localDe):exclusiveUBound(2,localDe), &
      2) = 2.5 ! dummy assignment
  enddo
  deallocate(larrayList)

For some applications the default association rules between DistGrid and Array dimensions may not satisfy the user's needs. The optional distgridToArrayMap argument can be used during Array creation to explicitly specify the mapping between DistGrid and Array dimensions. To demonstrate this the following lines of code reproduce the above example but with rearranged dimensions. Here the distgridToArrayMap argument is a list with two elements corresponding to the DistGrid dimCount of 2. The first element indicates which Array dimension the first DistGrid dimension is mapped against. Here the 1st DistGrid dimension maps against the 3rd Array dimension and the 2nd DistGrid dimension maps against the 1st Array dimension. This leaves the 2nd Array dimension to be the extra and undistributed dimension in the resulting Array object.

  call ESMF_ArrayDestroy(array, rc=rc)
  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    distgridToArrayMap=(/3, 1/), totalLWidth=(/0,1/), totalUWidth=(/0,1/), &
    undistLBound=(/1/), undistUBound=(/2/), rc=rc)

Operations on the Array object as a whole are unchanged by the different mapping of dimensions.

When working with Arrays that contain explicitly mapped Array and DistGrid dimensions it is critical to know the order in which the entries of width and bound arguments that are associated with distributed Array dimensions are specified. The size of these arguments is equal to the DistGrid dimCount, because the maximum number of distributed Array dimensions is given by the dimensionality of the index space.

The order of dimensions in these arguments, however, is not that of the associated DistGrid. Instead each entry corresponds to the distributed Array dimensions in sequence. In the example above the entries in totalLWidth and totalUWidth correspond to Array dimensions 1 and 3 in this sequence.

The distgridToArrrayMap argument optionally provided during Array create indicates how the DistGrid dimensions map to Array dimensions. The inverse mapping, i.e. Array to DistGrid dimensions, is just as important. The ESMF_ArrayGet() call offers both mappings as distgridToArrrayMap and arrayToDistGridMap, respectively. The number of elements in arrayToDistGridMap is equal to the rank of the Array. Each element corresponds to an Array dimension and indicates the associated DistGrid dimension by an integer number. An entry of "0" in arrayToDistGridMap indicates that the corresponding Array dimension is undistributed.

Correct understanding about the association between Array and DistGrid dimensions becomes critical for correct data access into the Array.

  allocate(arrayToDistGridMap(3))  ! arrayRank = 3
  call ESMF_ArrayGet(array, arrayToDistGridMap=arrayToDistGridMap, &
    exclusiveLBound=exclusiveLBound, exclusiveUBound=exclusiveUBound, &
    localDeCount=localDeCount, rc=rc)  
  if (arrayToDistGridMap(2) /= 0) then   ! check if extra dimension at 
    ! expected index indicate problem and bail out
  endif
  ! obtain larrayList for local DEs
  allocate(larrayList(0:localDeCount-1))
  call ESMF_ArrayGet(array, localarrayList=larrayList, rc=rc)
  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList(localDe), myFarray3D, &
       datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
    myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
      1, exclusiveLBound(2,localDe):exclusiveUBound(2, &
      localDe)) = 10.5 !dummy assignment
    myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
      2, exclusiveLBound(2,localDe):exclusiveUBound(2, &
      localDe)) = 23.3 !dummy assignment
  enddo
  deallocate(exclusiveLBound, exclusiveUBound)
  deallocate(arrayToDistGridMap)
  deallocate(larrayList)
  call ESMF_ArrayDestroy(array, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)


28.2.12 Arrays with replicated dimensions

Thus far most examples demonstrated cases where the DistGrid dimCount was equal to the Array rank. The previous section introduced the concept of Array tensor dimensions when dimCount < rank. In this section dimCount and rank are assumed completely unconstrained and the relationship to distgridToArrayMap and arrayToDistGridMap will be discussed.

The Array class allows completely arbitrary mapping between Array and DistGrid dimensions. Most cases considered in the previous sections used the default mapping which assigns the DistGrid dimensions in sequence to the lower Array dimensions. Extra Array dimensions, if present, are considered non-distributed tensor dimensions for which the optional undistLBound and undistUBound arguments must be specified.

The optional distgridToArrayMap argument provides the option to override the default DistGrid to Array dimension mapping. The entries of the distgridToArrayMap array correspond to the DistGrid dimensions in sequence and assign a unique Array dimension to each DistGrid dimension. DistGrid and Array dimensions are indexed starting at 1 for the lowest dimension. A value of "0" in the distgridToArrayMap array indicates that the respective DistGrid dimension is not mapped against any Array dimension. What this means is that the Array will be replicated along this DistGrid dimension.

As a first example consider the case where a 1D Array

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=1, rc=rc)

is created on the 2D DistGrid used during the previous section.

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)

Here the default DistGrid to Array dimension mapping is used which assigns the Array dimensions in sequence to the DistGrid dimensions starting with dimension "1". Extra DistGrid dimensions are considered replicator dimensions because the Array will be replicated along those dimensions. In the above example the 2nd DistGrid dimension will cause 1D Array pieces to be replicated along the DEs of the 2nd DistGrid dimension. Replication in the context of ESMF_ArrayCreate() does not mean that data values are communicated and replicated between different DEs, but it means that different DEs provide memory allocations for identical exclusive elements.

Access to the data storage of an Array that has been replicated along DistGrid dimensions is the same as for Arrays without replication.

  call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc)

  allocate(larrayList(0:localDeCount-1))
  allocate(localDeToDeMap(0:localDeCount-1))
  call ESMF_ArrayGet(array, localarrayList=larrayList, &
    localDeToDeMap=localDeToDeMap, rc=rc)

The array object was created without additional padding which means that the bounds of the Fortran array pointer correspond to the bounds of the exclusive region. The following loop will cycle through all local DEs, print the DE number as well as the Fortran array pointer bounds. The bounds should be:

            lbound       ubound
  
   DE 0:      1            3         --+
   DE 2:      1            3         --|  1st replication set
   DE 4:      1            3         --+
  
   DE 1:      1            2         --+
   DE 3:      1            2         --|  2nd replication set
   DE 5:      1            2         --+

  do localDe=0, localDeCount-1
    call ESMF_LocalArrayGet(larrayList(localDe), myFarray1D, &
      datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)

    print *, "localPet: ", localPet, "DE ",localDeToDeMap(localDe)," [", &
      lbound(myFarray1D), ubound(myFarray1D),"]"
  enddo
  deallocate(larrayList)
  deallocate(localDeToDeMap)
  call ESMF_ArrayDestroy(array, rc=rc)

The Fortran array pointer in the above loop was of rank 1 because the Array object was of rank 1. However, the distgrid object associated with array is 2-dimensional! Consequently DistGrid based information queried from array will be 2D. The distgridToArrayMap and arrayToDistGridMap arrays provide the necessary mapping to correctly associate DistGrid based information with Array dimensions.

The next example creates a 2D Array

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

on the previously used 2D DistGrid. By default, i.e. without the distgridToArrayMap argument, both DistGrid dimensions would be associated with the two Array dimensions. However, the distgridToArrayMap specified in the following call will only associate the second DistGrid dimension with the first Array dimension. This will render the first DistGrid dimension a replicator dimension and the second Array dimension a tensor dimension for which 1D undistLBound and undistUBound arguments must be supplied.

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    distgridToArrayMap=(/0,1/), undistLBound=(/11/), &
    undistUBound=(/14/), rc=rc)

  call ESMF_ArrayDestroy(array, rc=rc)

Finally, the same arrayspec and distgrid arguments are used to create a 2D Array that is fully replicated in both dimensions of the DistGrid. Both Array dimensions are now tensor dimensions and both DistGrid dimensions are replicator dimensions.

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    distgridToArrayMap=(/0,0/), undistLBound=(/11,21/), &
    undistUBound=(/14,22/), rc=rc)

The result will be an Array with local lower bound (/11,21/) and upper bound (/14,22/) on all 6 DEs of the DistGrid.

  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)

Replicated Arrays can also be created from existing local Fortran arrays. The following Fortran array allocation will provide a 3 x 10 array on each PET.

  allocate(myFarray2D(3,10))

Assuming a petCount of 4 the following DistGrid defines a 2D index space that is distributed across the PETs along the first dimension.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)

The following call creates an Array object on the above distgrid using the locally existing myFarray2D Fortran arrays. The difference compared to the case with automatic memory allocation is that instead of arrayspec the Fortran array is provided as argument. Furthermore, the undistLBound and undistUBound arguments can be omitted, defaulting into Array tensor dimension lower bound of 1 and an upper bound equal to the size of the respective Fortran array dimension.

  array = ESMF_ArrayCreate(farray=myFarray2D, distgrid=distgrid, &
    indexflag=ESMF_INDEX_DELOCAL, distgridToArrayMap=(/0,2/), rc=rc)

The array object associates the 2nd DistGrid dimension with the 2nd Array dimension. The first DistGrid dimension is not associated with any Array dimension and will lead to replication of the Array along the DEs of this direction.

  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)


28.2.13 Shared memory features: DE pinning, sharing, and migration

Practically all modern computer systems today utilize multi-core processors, supporting the concurrent execution of multiple hardware threads. A number of these multi-core processors are commonly packaged into the same compute node, having access to the same physical memory. Under ESMF each hardware thread (or core) is identified as a unique Processing Element (PE). The collection of PEs that share the same physical memory (i.e. compute node) is referred to as a Single System Image (SSI). The ESMF Array class implements features that allow the user to leverage the shared memory within each SSI to efficiently exchange data without copies or explicit communication calls.

The software threads executing an ESMF application on the hardware, and that ESMF is aware of, are referred to as Persistent Execution Threads (PETs). In practice a PET can typically be thought of as an MPI rank, i.e. an OS process, defining its own private virtual address space (VAS). The ESMF Virtual Machine (VM) class keeps track of the mapping between PETs and PEs, and their location on the available SSIs.

When an ESMF Array object is created, the specified DistGrid indicates how many Decomposition Elements (DEs) are created. Each DE has its own memory allocation to hold user data. The DELayout, referenced by the DistGrid, determines which PET is considered the owner of each of the DEs. Queried for the local DEs, the Array object returns the list of DEs that are owned by the local PET making the query.

By default DEs are pinned to the PETs under which they were created. The memory allocation associated with a specific DE is only defined in the VAS of the PET to which the DE is pinned. As a consequence, only the PET owning a DE has access to its memory allocation.

On shared memory systems, however, ESMF allows DEs to be pinned to SSIs instead of PETs. In this case the PET under which a DE was created is still consider the owner, but now all PETs under the same SSI have access to the DE. For this the memory allocation associated with the DE is mapped into the VAS of all the PETs under the SSI.

To create an Array with each DE pinned to SSI instead of PET, first query the VM for the available level of support.

  call ESMF_VMGet(vm, ssiSharedMemoryEnabledFlag=ssiSharedMemoryEnabled, rc=rc)

  if (ssiSharedMemoryEnabled) then

Knowing that the SSI shared memory feature is available, it is now possible to create an Array object with DE to SSI pinning.

    distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)

    array = ESMF_ArrayCreate(typekind=ESMF_TYPEKIND_R8, distgrid=distgrid, &
      pinflag=ESMF_PIN_DE_TO_SSI, rc=rc)

Just as in the cases discussed before, where the same DistGrid was used, a default DELayout with as many DEs as PETs in the VM is constructed. Setting the pinflag to ESMF_PIN_DE_TO_SSI does not change the fact that each PET owns exactly one of the DEs. However, assuming that this code is run on a set of PETs that are all located under the same SSI, every PET now has access to all of the DEs. The situation can be observed by querying for both the localDeCount, and the ssiLocalDeCount.

    call ESMF_ArrayGet(array, localDeCount=localDeCount, &
      ssiLocalDeCount=ssiLocalDeCount, rc=rc)

Assuming execution on 4 PETs, all located on the same SSI, the values of the returned variable are localDeCount==1 and ssiLocalDeCount==4 on all of the PETs. The mapping between each PET's local DE, and the global DE index is provided through the localDeToDeMap array argument. The amount of mapping information returned is dependent on how large localDeToDeMap has been sized by the user. For size(localDeToDeMap)==localDeCount, only mapping information for those DEs owned by the local PET is filled in. However for size(localDeToDeMap)==ssiLocalDeCount, mapping information for all locally accessible DEs is returned, including those owned by other PETs on the same SSI.

    allocate(localDeToDeMap(0:ssiLocalDeCount-1))
    call ESMF_ArrayGet(array, localDeToDeMap=localDeToDeMap, rc=rc)

The first localDeCount entries of localDeToDeMap are always the global DE indices of the DEs owned by the local PET. The remaining ssiLocalDeCount-localDeCount entries are the global DE indices of DEs shared by other PETs. The ordering of the shared DEs is from smallest to greatest, excluding the locally owned DEs, which were already listed at the beginning of localDeToDeMap. For the current case, again assuming execution on 4 PETs all located on the same SSI, we expect the following situation:

PET 0: localDeToDeMap==(/0,1,2,3/)
PET 1: localDeToDeMap==(/1,0,2,3/)
PET 2: localDeToDeMap==(/2,0,1,3/)
PET 3: localDeToDeMap==(/3,0,1,2/)

Each PET can access the memory allocations associated with all of the DEs listed in the localDeToDeMap returned by the Array object. Direct access to the Fortran array pointer of a specific memory allocation is available through ESMF_ArrayGet(). Here each PET queries for the farrayPtr of localDe==2, i.e. the 2nd shared DE.

    call ESMF_ArrayGet(array, farrayPtr=myFarray, localDe=2, rc=rc)

Now variable myFarray on PETs 0 and 1 both point to the same memory allocation for global DE 2. Both PETs have access to the same piece of shared memory! The same is true for PETs 2 and 3, pointing to the shared memory allocation of global DE 1.

It is important to note that all of the typical considerations surrounding shared memory programming apply when accessing shared DEs! Proper synchronization between PETs accessing shared DEs is critical to avoid race conditions. Also performance issues like false sharing need to be considered for optimal use.

For a simple demonstration, PETs 0 and 2 fill the entire memory allocation of DE 2 and 1, respectively, to a unique value.

    if (localPet==0) then
      myFarray = 12345.6789d0
    else if (localPet==2) then
      myFarray = 6789.12345d0
    endif

Here synchronization is needed before any PETs that share access to the same DEs can safely access the data without race condition. The Array class provides a simple synchronization method that can be used.

    call ESMF_ArraySync(array, rc=rc) ! prevent race condition

Now it is safe for PETs 1 and 3 to access the shared DEs. We expect to find the data that was set above. For simplicity of the code only the first array element is inspected here.

    if (localPet==1) then
      if (abs(myFarray(1,1)-12345.6789d0)>1.d10) print *, "bad data detected"
    else if (localPet==3) then
      if (abs(myFarray(1,1)-6789.12345d0)>1.d10) print *, "bad data detected"
    endif

Working with shared DEs requires additional bookkeeping on the user code level. In some situations, however, DE sharing is simply used as a mechanism to move DEs between PETs without requiring data copies. One practical application of this case is the transfer of an Array between two components, both of which use the same PEs, but run with different number of PETs. These would typically be sequential components that use OpenMP on the user level with varying threading levels.

DEs that are pinned to SSI can be moved or migrated to any PET within the SSI. This is accomplished by creating a new Array object from an existing Array that was created with pinflag=ESMF_PIN_DE_TO_SSI. The information of how the DEs are to migrate between the old and the new Array is provided through a DELayout object. This object must have the same number of DEs and describes how they map to the PETs on the current VM. If this is in the context of a different component, the number of PETs might differ from the original VM under which the existing Array was created. This situation is explicitly supported, still the number of DEs must match.

Here a simple DELayout is created on the same 4 PETs, but with rotated DE ownerships:

DE 0 -> PET 1 (old PET 0)
DE 1 -> PET 2 (old PET 1)
DE 2 -> PET 3 (old PET 2)
DE 3 -> PET 0 (old PET 3)

    delayout = ESMF_DELayoutCreate(petMap=(/1,2,3,0/), rc=rc) ! DE->PET mapping

The creation of the new Array is done by reference, i.e. datacopyflag=ESMF_DATACOPY_REFERENCE, since the new Array does not create its own memory allocations. Instead the new Array references the shared memory resources held by the incoming Array object.

    arrayMigrated = ESMF_ArrayCreate(array, delayout=delayout, &
      datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)

Querying arrayMigrated for the number of local DEs will return 1 on each PET. Sizing the localDeToDeMap accordingly and querying for it.

    deallocate(localDeToDeMap) ! free previous allocation
    allocate(localDeToDeMap(0:1))
    call ESMF_ArrayGet(arrayMigrated, localDeToDeMap=localDeToDeMap, rc=rc)

This yields the following expected outcome:

PET 0: localDeToDeMap==(/1/)
PET 1: localDeToDeMap==(/2/)
PET 2: localDeToDeMap==(/3/)
PET 3: localDeToDeMap==(/0/)

On each PET the respective Fortran array pointer is returned by the Array.

    call ESMF_ArrayGet(arrayMigrated, farrayPtr=myFarray, rc=rc)

The same situation could have been achieved with the original array. However, it would have required first finding the correct local DE for the target global DE on each PET, and then querying array accordingly. If needed more repeatedly, this bookkeeping would need to be kept in a user code data structure. The DE migration feature on the other hand provides a formal way to create a standard ESMF Array object that can be used directly in any Array level method as usual, letting ESMF handle the extra bookkeeping needed.

Before destroying an Array whose DEs are shared between PETs, it is advisable to issue one more synchronization. This prevents cases where a PET still might be accessing a shared DE, while the owner PET is already destroying the Array, therefore deallocating the shared memory resource.

    call ESMF_ArraySync(array, rc=rc) ! prevent race condition

    call ESMF_ArrayDestroy(array, rc=rc)

Remember that arrayMigrated shares the same memory allocations that were held by array. Array arrayMigrated must therefore not be used beyond the life time of array. Best to destroy it now.

    call ESMF_ArrayDestroy(arrayMigrated, rc=rc)

  endif ! ending the ssiSharedMemoryEnabled conditional


28.2.14 Communication - Scatter and Gather

It is a common situation, particularly in legacy code, that an ESMF Array object must be filled with data originating from a large Fortran array stored on a single PET.

  if (localPet == 0) then
    allocate(farray(10,20,30))
    do k=1, 30
      do j=1, 20
        do i=1, 10
          farray(i, j, k) = k*1000 + j*100 +  i
        enddo
      enddo
    enddo
  else
    allocate(farray(0,0,0))
  endif

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1,1/), maxIndex=(/10,20,30/), &
    rc=rc)

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=3, rc=rc)

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)

The ESMF_ArrayScatter() method provides a convenient way of scattering array data from a single root PET across the DEs of an ESMF Array object.

  call ESMF_ArrayScatter(array, farray=farray, rootPet=0, rc=rc)

  deallocate(farray)

The destination of the ArrayScatter() operation are all the DEs of a single tile. For multi-tile Arrays the destination tile can be specified. The shape of the scattered Fortran array must match the shape of the destination tile in the ESMF Array.

Gathering data decomposed and distributed across the DEs of an ESMF Array object into a single Fortran array on root PET is accomplished by calling ESMF_ArrayGather().

  if (localPet == 3) then
    allocate(farray(10,20,30))
  else
    allocate(farray(0,0,0))
  endif
  
  call ESMF_ArrayGather(array, farray=farray, rootPet=3, rc=rc)

  deallocate(farray)

The source of the ArrayGather() operation are all the DEs of a single tile. For multi-tile Arrays the source tile can be specified. The shape of the gathered Fortran array must match the shape of the source tile in the ESMF Array.

The ESMF_ArrayScatter() operation allows to fill entire replicated Array objects with data coming from a single root PET.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), rc=rc)

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    distgridToArrayMap=(/0,0/), undistLBound=(/11,21/), &
    undistUBound=(/14,22/), rc=rc)

The shape of the Fortran source array used in the Scatter() call must be that of the contracted Array, i.e. contracted DistGrid dimensions do not count. For the array just created this means that the source array on rootPet must be of shape 4 x 2.

  if (localPet == 0) then
    allocate(myFarray2D(4,2))
    do j=1,2
      do i=1,4
        myFarray2D(i,j) = i * 100.d0 + j * 1.2345d0 ! initialize
      enddo
    enddo
  else
    allocate(myFarray2D(0,0))
  endif
  
  call ESMF_ArrayScatter(array, farray=myFarray2D, rootPet=0, rc=rc)

  deallocate(myFarray2D)

This will have filled each local 4 x 2 Array piece with the replicated data of myFarray2D.

  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)

As a second example for the use of Scatter() and Gather() consider the following replicated Array created from existing local Fortran arrays.

  allocate(myFarray2D(3,10))
  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)

  array = ESMF_ArrayCreate(farray=myFarray2D, distgrid=distgrid, &
    indexflag=ESMF_INDEX_DELOCAL, distgridToArrayMap=(/0,2/), rc=rc)

The array object associates the 2nd DistGrid dimension with the 2nd Array dimension. The first DistGrid dimension is not associated with any Array dimension and will lead to replication of the Array along the DEs of this direction. Still, the local arrays that comprise the array object refer to independent pieces of memory and can be initialized independently.

  myFarray2D = localPet ! initialize

However, the notion of replication becomes visible when an array of shape 3 x 10 on root PET 0 is scattered across the Array object.

  if (localPet == 0) then
    allocate(myFarray2D2(5:7,11:20))
  
    do j=11,20
      do i=5,7
        myFarray2D2(i,j) = i * 100.d0 + j * 1.2345d0 ! initialize
      enddo
    enddo
  else
    allocate(myFarray2D2(0,0))
  endif
  
  call ESMF_ArrayScatter(array, farray=myFarray2D2, rootPet=0, rc=rc)

  deallocate(myFarray2D2)

The Array pieces on every DE will receive the same source data, resulting in a replication of data along DistGrid dimension 1.

When the inverse operation, i.e. ESMF_ArrayGather(), is applied to a replicated Array an intrinsic ambiguity needs to be considered. ESMF defines the gathering of data of a replicated Array as the collection of data originating from the numerically higher DEs. This means that data in replicated elements associated with numerically lower DEs will be ignored during ESMF_ArrayGather(). For the current example this means that changing the Array contents on PET 1, which here corresponds to DE 1,

  if (localPet == 1) then
    myFarray2D = real(1.2345, ESMF_KIND_R8)
  endif

will not affect the result of

  allocate(myFarray2D2(3,10))
  myFarray2D2 = 0.d0    ! initialize to a known value
  call ESMF_ArrayGather(array, farray=myFarray2D2, rootPet=0, rc=rc)

The result remains completely defined by the unmodified values of Array in DE 3, the numerically highest DE. However, overriding the DE-local Array piece on DE 3

  if (localPet==3) then
    myFarray2D = real(5.4321, ESMF_KIND_R8)
  endif

will change the outcome of

  call ESMF_ArrayGather(array, farray=myFarray2D2, rootPet=0, rc=rc)

as expected.

  deallocate(myFarray2D2)

  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)


28.2.15 Communication - Halo

One of the most fundamental communication pattern in domain decomposition codes is the halo operation. The ESMF Array class supports halos by allowing memory for extra elements to be allocated on each DE. See sections 28.2.2 and 28.2.8 for examples and details on how to create an Array with extra DE-local elements.

Here we consider an Array object that is created on a DistGrid that defines a 10 x 20 index space, decomposed into 4 DEs using a regular 2 x 2 decomposition.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
    regDecomp=(/2,2/), rc=rc)

The Array holds 2D double precision float data.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

The totalLWidth and totalUWidth arguments are used during Array creation to allocate 2 extra elements along every direction outside the exclusive region defined by the DistGrid for every DE. (The indexflag set to ESMF_INDEX_GLOBAL in this example does not affect the halo behavior of Array. The setting is simply more convenient for the following code.)

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    totalLWidth=(/2,2/), totalUWidth=(/2,2/), indexflag=ESMF_INDEX_GLOBAL, &
    rc=rc)

Without the explicit definition of boundary conditions in the DistGrid the following inner connections are defined.

   
            +-------------------+       +-------------------+
            | \       2       / |       | \       2       / |
            |  +-------------+  |       |  +-------------+  |
            |  |     DE 0    |  |       |  |     DE 2    |  |
            |  |             |  |       |  |             |  |
            |2 |    5 x 10   | 2|  <->  |2 |    5 x 10   | 2|
            |  |             |  |       |  |             |  |
            |  |             |  |       |  |             |  |
            |  +-------------+  |       |  +-------------+  |
            | /       2       \ |       | /       2       \ |
            +-------------------+       +-------------------+
  
                      ^            \/             ^
                      |            /\             |
                      v                           v
  
            +-------------------+       +-------------------+
            | \       2       / |       | \       2       / |
            |  +-------------+  |       |  +-------------+  |
            |  |     DE 1    |  |       |  |     DE 3    |  |
            |  |             |  |       |  |             |  |
            |2 |    5 x 10   | 2|  <->  |2 |    5 x 10   | 2|
            |  |             |  |       |  |             |  |
            |  |             |  |       |  |             |  |
            |  +-------------+  |       |  +-------------+  |
            | /       2       \ |       | /       2       \ |
            +-------------------+       +-------------------+

The exclusive region on each DE is of shape 5 x 10, while the total region on each DE is of shape (5+2+2) x (10+2+2) = 9 x 14. In a typical application the elements in the exclusive region are updated exclusively by the PET that owns the DE. In this example the exclusive elements on every DE are initialized to the value $f(i,j)$ of the geometric function

\begin{displaymath}
f(i,j) = \sin(\alpha i)\cos(\beta j),
\end{displaymath} (1)

where
\begin{displaymath}
\alpha = 2\pi/N_i, i=1,...N_i
\end{displaymath} (2)

and
\begin{displaymath}
\beta = 2\pi/N_j, j=1,...N_j,
\end{displaymath} (3)

with $N_i = 10$ and $N_j = 20$.

  a = 2. * 3.14159 / 10.
  b = 2. * 3.14159 / 20.
  
  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)

  
  call ESMF_ArrayGet(array, exclusiveLBound=eLB, exclusiveUBound=eUB, rc=rc)

  
  do j=eLB(2,1), eUB(2,1)
    do i=eLB(1,1), eUB(1,1)
      farrayPtr(i,j) = sin(a*i) * cos(b*j)  ! test function
    enddo
  enddo

The above loop only initializes the exclusive elements on each DE. The extra elements, outside the exclusive region, are left untouched, holding undefined values. Elements outside the exclusive region that correspond to exclusive elements in neighboring DEs can be filled with the data values in those neighboring elements. This is the definition of the halo operation.

In ESMF the halo communication pattern is first precomputed and stored in a RouteHandle object. This RouteHandle can then be used repeatedly to perform the same halo operation in the most efficient way.

The default halo operation for an Array is precomputed by the following call.

  call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle, rc=rc)

The haloHandle now holds the default halo operation for array, which matches as many elements as possible outside the exclusive region to their corresponding halo source elements in neighboring DEs. Elements that could not be matched, e.g. at the edge of the global domain with open boundary conditions, will not be updated by the halo operation.

The haloHandle is applied through the ESMF_ArrayHalo() method.

  call ESMF_ArrayHalo(array=array, routehandle=haloHandle, rc=rc)

Finally the resources held by haloHandle need to be released.

  call ESMF_ArrayHaloRelease(routehandle=haloHandle, rc=rc)

The array object created above defines a 2 element wide rim around the exclusive region on each DE. Consequently the default halo operation used above will have resulted in updating both elements along the inside edges. For simple numerical kernels often a single halo element is sufficient. One way to achieve this would be to reduce the size of the rim surrounding the exclusive region to 1 element along each direction. However, if the same Array object is also used for higher order kernels during a different phase of the calculation, a larger element rim is required. For this case ESMF_ArrayHaloStore() offers two optional arguments haloLDepth and haloUDepth. Using these arguments a reduced halo depth can be specified.

  call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle, &
    haloLDepth=(/1,1/), haloUDepth=(/1,1/), rc=rc)

This halo operation with a depth of 1 is sufficient to support a simple quadratic differentiation kernel.

  allocate(farrayTemp(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)))

  do step=1, 4
    call ESMF_ArrayHalo(array=array, routehandle=haloHandle, rc=rc)

    do j=eLB(2,1), eUB(2,1)
      do i=eLB(1,1), eUB(1,1)
        if (i==1) then
          ! global edge
          farrayTemp(i,j) = 0.5 * (-farrayPtr(i+2,j) + 4.*farrayPtr(i+1,j) &
            - 3.*farrayPtr(i,j)) / a
        else if (i==10) then
          ! global edge
          farrayTemp(i,j) = 0.5 * (farrayPtr(i-2,j) - 4.*farrayPtr(i-1,j) &
            + 3.*farrayPtr(i,j)) / a
        else
          farrayTemp(i,j) = 0.5 * (farrayPtr(i+1,j) - farrayPtr(i-1,j)) / a
        endif
      enddo
    enddo
    farrayPtr(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)) = farrayTemp
  enddo
  
  deallocate(farrayTemp)

  call ESMF_ArrayHaloRelease(routehandle=haloHandle, rc=rc)

The special treatment of the global edges in the above kernel is due to the fact that the underlying DistGrid object does not define any special boundary conditions. By default open global boundaries are assumed which means that the rim elements on the global edges are untouched during the halo operation, and cannot be used in the symmetric numerical derivative formula. The kernel can be simplified (and the calculation is more precise) with periodic boundary conditions along the first Array dimension.

First destroy the current Array and DistGrid objects.

  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)

Create a DistGrid with periodic boundary condition along the first dimension.

  allocate(connectionList(1))  ! one connection
  call ESMF_DistGridConnectionSet(connection=connectionList(1), &
     tileIndexA=1, tileIndexB=1, positionVector=(/10, 0/), rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
    regDecomp=(/2,2/), connectionList=connectionList, rc=rc)

  deallocate(connectionList)
  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
    totalLWidth=(/2,2/), totalUWidth=(/2,2/), indexflag=ESMF_INDEX_GLOBAL, &
    rc=rc)

Initialize the exclusive elements to the same geometric function as before.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)

  
  call ESMF_ArrayGet(array, exclusiveLBound=eLB, exclusiveUBound=eUB, rc=rc)

  
  do j=eLB(2,1), eUB(2,1)
    do i=eLB(1,1), eUB(1,1)
      farrayPtr(i,j) = sin(a*i) * cos(b*j)  ! test function
    enddo
  enddo

The numerical kernel only operates along the first dimension. An asymmetric halo depth can be used to take this fact into account.

  call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle, &
    haloLDepth=(/1,0/), haloUDepth=(/1,0/), rc=rc)

Now the same numerical kernel can be used without special treatment of global edge elements. The symmetric derivative formula can be used for all exclusive elements.

  allocate(farrayTemp(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)))

  do step=1, 4
    call ESMF_ArrayHalo(array=array, routehandle=haloHandle, rc=rc)

    do j=eLB(2,1), eUB(2,1)
      do i=eLB(1,1), eUB(1,1)
        farrayTemp(i,j) = 0.5 * (farrayPtr(i+1,j) - farrayPtr(i-1,j)) / a
      enddo
    enddo
    farrayPtr(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)) = farrayTemp
  enddo

The precision of the above kernel can be improved by going to a higher order interpolation. Doing so requires that the halo depth must be increased. The following code resets the exclusive Array elements to the test function, precomputes a RouteHandle for a halo operation with depth 2 along the first dimension, and finally uses the deeper halo in the higher order kernel.

  
  do j=eLB(2,1), eUB(2,1)
    do i=eLB(1,1), eUB(1,1)
      farrayPtr(i,j) = sin(a*i) * cos(b*j)  ! test function
    enddo
  enddo

  call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle2, &
    haloLDepth=(/2,0/), haloUDepth=(/2,0/), rc=rc)

  do step=1, 4
    call ESMF_ArrayHalo(array=array, routehandle=haloHandle2, rc=rc)

    do j=eLB(2,1), eUB(2,1)
      do i=eLB(1,1), eUB(1,1)
        farrayTemp(i,j) = (-farrayPtr(i+2,j) + 8.*farrayPtr(i+1,j) &
          - 8.*farrayPtr(i-1,j) + farrayPtr(i-2,j)) / (12.*a)
      enddo
    enddo
    farrayPtr(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)) = farrayTemp
  enddo
  
  deallocate(farrayTemp)

ESMF supports having multiple halo operations defined on the same Array object at the same time. Each operation can be accessed through its unique RouteHandle. The above kernel could have made ESMF_ArrayHalo() calls with a depth of 1 along the first dimension using the previously precomputed haloHandle if it needed to. Both RouteHandles need to release their resources when no longer used.

  

  call ESMF_ArrayHaloRelease(routehandle=haloHandle, rc=rc)

  call ESMF_ArrayHaloRelease(routehandle=haloHandle2, rc=rc)

Finally the Array and DistGrid objects can be destroyed.

  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)


28.2.16 Communication - Halo for arbitrary distribution

In the previous section the Array halo operation was demonstrated for regularly decomposed ESMF Arrays. However, the ESMF halo operation is not restricted to regular decompositions. The same Array halo methods apply unchanged to Arrays that are created on arbitrarily distributed DistGrids. This includes the non-blocking features discussed in section 28.2.20.

All of the examples in this section are based on the same arbitrarily distributed DistGrid. Section 36.3.5 discusses DistGrids with user-supplied, arbitrary sequence indices in detail. Here a global index space range from 1 through 20 is decomposed across 4 DEs. There are 4 PETs in this example with 1 DE per PET. Each PET constructs its local seqIndexList variable.

  do i=1, 5
#ifdef TEST_I8RANGE_on
    seqIndexList(i) = localPet + (i - 1) * petCount + 1 + seqIndexOffset
#else
    seqIndexList(i) = localPet + (i - 1) * petCount + 1
#endif
  enddo

This results in the following cyclic distribution scheme:

   DE 0 on PET 0: seqIndexList = (/1, 5, 9, 13, 17/)
   DE 1 on PET 1: seqIndexList = (/2, 6, 10, 14, 18/)
   DE 2 on PET 2: seqIndexList = (/3, 7, 11, 15, 19/)
   DE 3 on PET 3: seqIndexList = (/4, 8, 12, 16, 20/)

The local seqIndexList variables are then used to create a DistGrid with the indicated arbitrary distribution pattern.

  distgrid = ESMF_DistGridCreate(arbSeqIndexList=seqIndexList, rc=rc)

The resulting DistGrid is one-dimensional, although the user code may interpret the sequence indices as a 1D map into a problem of higher dimensionality.

In this example the local DE on each PET is associated with a 5 element exclusive region. Providing seqIndexList of different size on the different PETs is supported and would result in different number of exclusive elements on each PET.

28.2.16.1 Halo for a 1D Array from existing memory allocation, created on the 1D arbitrary DistGrid.


Creating an ESMF Array on top of a DistGrid with arbitrary sequence indices is in principle no different from creating an Array on a regular DistGrid. However, while an Array that was created on a regular DistGrid automatically inherits the index space topology information that is contained within the DistGrid object, there is no such topology information available for DistGrid objects with arbitrary sequence indices. As a consequence of this, Arrays created on arbitrary DistGrids do not automatically have the information that is required to associated halo elements with the exclusive elements across DEs. Instead the user must supply this information explicitly during Array creation.

Multiple ArrayCreate() interfaces exist that allow the creation of an Array on a DistGrid with arbitrary sequence indices. The sequence indices for the halo region of the local DE are supplied through an additional argument with dummy name haloSeqIndexList. As in the regular case, the ArrayCreate() interfaces differ in the way that the memory allocations for the Array elements are passed into the call. The following code shows how an ESMF Array can be wrapped around existing PET-local memory allocations. The allocations are of different size on each PET as to accommodate the correct number of local Array elements (exclusive region + halo region).

  allocate(farrayPtr1d(5+localPet+1)) !use explicit Fortran allocate statement
  
  if (localPet==0) then
    allocate(haloList(1))
#ifdef TEST_I8RANGE_on
    haloList(:)=(/1099511627782_ESMF_KIND_I8/)
#else    
    haloList(:)=(/6/)
#endif
    array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==1) then
    allocate(haloList(2))
#ifdef TEST_I8RANGE_on
    haloList(:)=(/1099511627777_ESMF_KIND_I8,&
                  1099511627795_ESMF_KIND_I8/)
#else
    haloList(:)=(/1,19/)
#endif
    array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==2) then
    allocate(haloList(3))
#ifdef TEST_I8RANGE_on
    haloList(:)=(/1099511627792_ESMF_KIND_I8,&
                  1099511627782_ESMF_KIND_I8,&
                  1099511627785_ESMF_KIND_I8/)
#else
    haloList(:)=(/16,6,9/)
#endif
    array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==3) then
    allocate(haloList(4))
#ifdef TEST_I8RANGE_on
    haloList(:)=(/1099511627777_ESMF_KIND_I8,&
                  1099511627779_ESMF_KIND_I8,&
                  1099511627777_ESMF_KIND_I8,&
                  1099511627780_ESMF_KIND_I8/)
#else
    haloList(:)=(/1,3,1,4/)
#endif
    array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif

The haloSeqIndexList arguments are 1D arrays of sequence indices. It is through this argument that the user associates the halo elements with exclusive Array elements covered by the DistGrid. In this example there are different number of halo elements on each DE. They are associated with exclusive elements as follows:

   halo on DE 0 on PET 0: <seqIndex=6>  2nd exclusive element on DE 1
   halo on DE 1 on PET 1: <seqIndex=1>  1st exclusive element on DE 0
                          <seqIndex=19> 5th exclusive element on DE 2
   halo on DE 2 on PET 2: <seqIndex=16> 4th exclusive element on DE 3
                          <seqIndex=6>  2nd exclusive element on DE 1
                          <seqIndex=9>  3rd exclusive element on DE 0
   halo on DE 3 on PET 3: <seqIndex=1>  1st exclusive element on DE 0
                          <seqIndex=3>  1st exclusive element on DE 2
                          <seqIndex=1>  1st exclusive element on DE 0
                          <seqIndex=4>  1st exclusive element on DE 3

The above haloSeqIndexList arguments were constructed very artificially in order to show the following general features:

The ArrayCreate() call checks that the provided Fortran memory allocation is correctly sized to hold the exclusive elements, as indicated by the DistGrid object, plus the halo elements as indicated by the local haloSeqIndexList argument. The size of the Fortran allocation must match exactly or a runtime error will be returned.

Analogous to the case of Arrays on regular DistGrids, it is the exclusive region of the local DE that is typically modified by the code running on each PET. All of the ArrayCreate() calls that accept the haloSeqIndexList argument place the exclusive region at the beginning of the memory allocation on each DE and use the remaining space for the halo elements. The following loop demonstrates this by filling the exclusive elements on each DE with initial values. Remember that in this example each DE holds 5 exclusive elements associated with different arbitrary sequence indices.

  farrayPtr1d = 0 ! initialize
  do i=1, 5
    farrayPtr1d(i) = real(seqIndexList(i), ESMF_KIND_R8)
  enddo
  print *, "farrayPtr1d: ", farrayPtr1d

Now the exclusive elements of array are initialized on each DE, however, the halo elements remain unchanged. A RouteHandle can be set up that encodes the required communication pattern for a halo exchange. The halo exchange is precomputed according to the arbitrary sequence indices specified for the exclusive elements by the DistGrid and the sequence indices provided by the user for each halo element on the local DE in form of the haloSeqIndexList argument during ArrayCreate().

  call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)

Executing this halo operation will update the local halo elements according to the associated sequence indices.

  call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)

As always it is good practice to release the RouteHandle when done with it.

  call ESMF_ArrayHaloRelease(haloHandle, rc=rc)

Also the Array object should be destroyed when no longer needed.

  call ESMF_ArrayDestroy(array, rc=rc)

Further, since the memory allocation was done explicitly using the Fortran allocate() statement, it is necessary to explicitly deallocate in order to prevent memory leaks in the user application.

  deallocate(farrayPtr1d)

28.2.16.2 Halo for a 1D Array with ESMF managed memory allocation, created on the 1D arbitrary DistGrid.


Alternatively the exact same Array can be created where ESMF does the memory allocation and deallocation. In this case the typekind of the Array must be specified explicitly.

  if (localPet==0) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==1) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==2) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==3) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif

Use ESMF_ArrayGet() to gain access to the local memory allocation.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr1d, rc=rc)

The returned Fortran pointer can now be used to initialize the exclusive elements on each DE as in the previous case.

  do i=1, 5
    farrayPtr1d(i) = real(seqIndexList(i),ESMF_KIND_R8) / 10.d0
  enddo

Identical halo operations are constructed and used.

  call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)

  call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)

  call ESMF_ArrayHaloRelease(haloHandle, rc=rc)

  call ESMF_ArrayDestroy(array, rc=rc)

28.2.16.3 Halo for an Array with undistributed dimensions, created on the 1D arbitrary DistGrid, with default Array to DistGrid dimension mapping.


A current limitation of the Array implementation restricts DistGrids that contain user-specified, arbitrary sequence indices to be exactly 1D when used to create Arrays. See section 28.3 for a list of current implementation restrictions. However, an Array created on such a 1D arbitrary DistGrid is allowed to have undistributed dimensions. The following example creates an Array on the same arbitrary DistGrid, with the same arbitrary sequence indices for the halo elements as before, but with one undistributed dimension with a size of 3.

  if (localPet==0) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==1) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==2) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==3) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      haloSeqIndexList=haloList, undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif

By default the DistGrid dimension is mapped to the first Array dimension, associating the remaining Array dimensions with the undistributed dimensions in sequence. The dimension order is important when accessing the individual Array elements. Here the same initialization as before is extended to cover the undistributed dimension.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr2d, rc=rc)

  do j=1, 3
    do i=1, 5
      farrayPtr2d(i,j) = real(seqIndexList(i),ESMF_KIND_R8) / 10.d0 + 100.d0*j
    enddo
  enddo

In the context of the Array halo operation additional undistributed dimensions are treated in a simple factorized manner. The same halo association between elements that is encoded in the 1D arbitrary sequence index scheme is applied to each undistributed element separately. This is completely transparent on the user level and the same halo methods are used as before.

  call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)

  call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)

  call ESMF_ArrayHaloRelease(haloHandle, rc=rc)

  call ESMF_ArrayDestroy(array, rc=rc)

28.2.16.4 Halo for an Array with undistributed dimensions, created on the 1D arbitrary DistGrid, mapping the undistributed dimension first.


In some situations it is more convenient to associate some or all of the undistributed dimensions with the first Array dimensions. This can be done easily by explicitly mapping the DistGrid dimension to an Array dimension other than the first one. The distgridToArrayMap argument is used to provide this information. The following code creates essentially the same Array as before, but with swapped dimension order - now the first Array dimension is the undistributed one.

  if (localPet==0) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==1) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==2) then
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==3) then
#ifdef TEST_I8RANGE_on
    haloList(:)=(/1099511627777_ESMF_KIND_I8,&
                  1099511627780_ESMF_KIND_I8,&
                  1099511627779_ESMF_KIND_I8,&
                  1099511627778_ESMF_KIND_I8/)
#else
    haloList(:)=(/1,3,5,4/)
#endif
    array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/3/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif

Notice that the haloList constructed on PET 3 is different from the previous examples. All other PETs reuse the same haloList as before. In the previous examples the list loaded into PET 3's haloSeqIndexList argument contained a duplicate sequence index. However, now that the undistributed dimension is placed first, the ESMF_ArrayHaloStore() call will try to optimize the data exchange by vectorizing it. Duplicate sequence indices are currently not supported during vectorization.

When accessing the Array elements, the swapped dimension order results in a swapping of i and j. This can be seen in the following initialization loop.

  call ESMF_ArrayGet(array, farrayPtr=farrayPtr2d, rc=rc)

  do j=1, 3
    do i=1, 5
      farrayPtr2d(j,i) = real(seqIndexList(i),ESMF_KIND_R8) / 10.d0 + 100.d0*j
    enddo
  enddo

Once set up, there is no difference in how the the halo operations are applied.

  call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)

  call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)

  call ESMF_ArrayDestroy(array, rc=rc)

28.2.16.5 Halo for an Array with undistributed dimensions, created on the 1D arbitrary DistGrid, re-using the RouteHandle.


Arrays can reuse the same RouteHandle, saving the overhead that is caused by the precompute step. In order to demonstrate this the RouteHandle of the previous halo call was not yet released and will be applied to a new Array.

The following code creates an Array that is compatible to the previous Array by using the same input information as before, only that the size of the undistributed dimension is now 6 instead of 3.

  if (localPet==0) then
    array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/6/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==1) then
    array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/6/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==2) then
    array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/6/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif
  if (localPet==3) then
    array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
      distgridToArrayMap=(/2/), haloSeqIndexList=haloList, &
      undistLBound=(/1/), undistUBound=(/6/), rc=rc)
    if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  endif

Again the exclusive Array elements must be initialized.

  call ESMF_ArrayGet(array2, farrayPtr=farrayPtr2d, rc=rc)

  do j=1, 6
    do i=1, 5
      farrayPtr2d(j,i) = real(seqIndexList(i),ESMF_KIND_R8) / 10.d0 + 100.d0*j
    enddo
  enddo

Now the haloHandle that was previously pre-computed for array can be used directly for array2.

  call ESMF_ArrayHalo(array2, routehandle=haloHandle, rc=rc)

Release the RouteHandle after its last use and clean up the remaining Array and DistGrid objects.

  call ESMF_ArrayHaloRelease(haloHandle, rc=rc)

  call ESMF_ArrayDestroy(array2, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)


28.2.17 Communication - Redist

Arrays used in different models often cover the same index space region, however, the distribution of the Arrays may be different, e.g. the models run on exclusive sets of PETs. Even if the Arrays are defined on the same list of PETs the decomposition may be different.

  srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
    regDecomp=(/4,1/), rc=rc)

  dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
    regDecomp=(/1,4/), rc=rc)

The number of elements covered by srcDistgrid is identical to the number of elements covered by dstDistgrid - in fact the index space regions covered by both DistGrid objects are congruent. However, the decomposition defined by regDecomp, and consequently the distribution of source and destination, are different.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

  srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, rc=rc)

  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)


28.2.17.1 Default Mode

By construction srcArray and dstArray are of identical type and kind. Further the number of exclusive elements matches between both Arrays. These are the prerequisites for the application of an Array redistribution in default mode. In order to increase performance of the actual redistribution the communication pattern is precomputed once, and stored in an ESMF_RouteHandle object.

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

The redistHandle can now be used repeatedly to transfer data from srcArray to dstArray.

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

The use of the precomputed redistHandle is not restricted to the (srcArray, dstArray) pair. Instead the redistHandle can be used to redistribute data between any two Arrays that are compatible with the Array pair used during precomputation. I.e. any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The transferability of RouteHandles between Array pairs can greatly reduce the number of communication store calls needed. In a typical application Arrays are often defined on the same decomposition, typically leading to congruent distributed dimensions. For these Arrays, while they may not have the same shape or size in the undistributed dimensions, RouteHandles are reusable.

For the current case, the redistHandle was precomputed for simple 2D Arrays without undistributed dimensions. The RouteHandle transferability rule allows us to use this same RouteHandle to redistribute between two 3D Array that are built on the same 2D DistGrid, but have an undistributed dimension. Note that the undistributed dimension does not have to be in the same position on source and destination. Here the undistributed dimension is in position 2 for srcArray1, and in position 1 for dstArray1.

  call ESMF_ArraySpecSet(arrayspec3d, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)

  srcArray1 = ESMF_ArrayCreate(arrayspec=arrayspec3d, distgrid=srcDistgrid, &
    distgridToArrayMap=(/1,3/), undistLBound=(/1/), undistUBound=(/10/), rc=rc)

  dstArray1 = ESMF_ArrayCreate(arrayspec=arrayspec3d, distgrid=dstDistgrid, &
    distgridToArrayMap=(/2,3/), undistLBound=(/1/), undistUBound=(/10/), rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray1, dstArray=dstArray1, &
    routehandle=redistHandle, rc=rc)

The following variation of the code shows that the same RouteHandle can be applied to an Array pair even when the number of undistributed dimensions does not match between source and destination Array, as long as to the total number of undistributed elements matches.

We prepare a source Array with two undistributed dimensions, in position 1 and 3, of size $2$ and $5$, respectively. Thus there are $2 \times 5=10$ undistributed source elements. The destination array is the same as before with only a single undistributed dimension in position 1 of size $10$.

  call ESMF_ArraySpecSet(arrayspec4d, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)

  srcArray2 = ESMF_ArrayCreate(arrayspec=arrayspec4d, distgrid=srcDistgrid, &
    distgridToArrayMap=(/2,4/), undistLBound=(/1,1/), undistUBound=(/2,5/), &
    rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray2, dstArray=dstArray1, &
    routehandle=redistHandle, rc=rc)

When done, the resources held by redistHandle need to be deallocated by the user code before the RouteHandle becomes inaccessible.

  call ESMF_ArrayRedistRelease(routehandle=redistHandle, rc=rc)


28.2.17.2 Transpose Mode

In default mode, i.e. without providing the optional srcToDstTransposeMap argument, ESMF_ArrayRedistStore() does not require equal number of dimensions in source and destination Array. Only the total number of elements must match. Specifying srcToDstTransposeMap switches ESMF_ArrayRedistStore() into transpose mode. In this mode each dimension of srcArray is uniquely associated with a dimension in dstArray, and the sizes of associated dimensions must match for each pair.

  dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/20,10/), &
      rc=rc)

  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)

This dstArray object covers a 20 x 10 index space while the srcArray, defined further up, covers a 10 x 20 index space. Setting srcToDstTransposeMap = (/2,1/) will associate the first and second dimension of srcArray with the second and first dimension of dstArray, respectively. This corresponds to a transpose of dimensions. Since the decomposition and distribution of dimensions may be different for source and destination redistribution may occur at the same time.

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, srcToDstTransposeMap=(/2,1/), rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

The transpose mode of ESMF_ArrayRedist() is not limited to distributed dimensions of Arrays. The srcToDstTransposeMap argument can be used to transpose undistributed dimensions in the same manner. Furthermore transposing distributed and undistributed dimensions between Arrays is also supported.

The srcArray used in the following examples is of rank 4 with 2 distributed and 2 undistributed dimensions. The distributed dimensions are the two first dimensions of the Array and are distributed according to the srcDistgrid which describes a total index space region of 100 x 200 elements. The last two Array dimensions are undistributed dimensions of size 2 and 3, respectively.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)

  srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/100,200/), &
    rc=rc)

  srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
    undistLBound=(/1,1/), undistUBound=(/2,3/), rc=rc)

The first dstArray to consider is defined on a DistGrid that also describes a 100 x 200 index space region. The distribution indicated by dstDistgrid may be different from the source distribution. Again the first two Array dimensions are associated with the DistGrid dimensions in sequence. Furthermore, the last two Array dimensions are undistributed dimensions, however, the sizes are 3 and 2, respectively.

  dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/100,200/), &
    rc=rc)

  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
    undistLBound=(/1,1/), undistUBound=(/3,2/), rc=rc)

The desired mapping between srcArray and dstArray dimensions is expressed by srcToDstTransposeMap = (/1,2,4,3/), transposing only the two undistributed dimensions.

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, srcToDstTransposeMap=(/1,2,4,3/), rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

Next consider a dstArray that is defined on the same dstDistgrid, but with a different order of Array dimensions. The desired order is specified during Array creation using the argument distgridToArrayMap = (/2,3/). This map associates the first and second DistGrid dimensions with the second and third Array dimensions, respectively, leaving Array dimensions one and four undistributed.

  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
    distgridToArrayMap=(/2,3/), undistLBound=(/1,1/), undistUBound=(/3,2/), &
    rc=rc)

Again the sizes of the undistributed dimensions are chosen in reverse order compared to srcArray. The desired transpose mapping in this case will be srcToDstTransposeMap = (/2,3,4,1/).

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, srcToDstTransposeMap=(/2,3,4,1/), rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

Finally consider the case where dstArray is constructed on a 200 x 3 index space and where the undistributed dimensions are of size 100 and 2.

  dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/200,3/), &
    rc=rc)

  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
    undistLBound=(/1,1/), undistUBound=(/100,2/), rc=rc)

By construction srcArray and dstArray hold the same number of elements, albeit in a very different layout. Nevertheless, with a srcToDstTransposeMap that maps matching dimensions from source to destination, the following Array redistribution becomes a well defined operation between srcArray and dstArray.

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, srcToDstTransposeMap=(/3,1,4,2/), rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

The srcToDstTransposeMap mechanism supports negative map entries. Negative entries indicate that the order of elements is to be reversed when going from source to destination. Using the same srcArray and dstArray objects as in the previous example, the following code maps the first srcArray dimension to the third dstArray dimension, as before. However, the ordering of the elements along this dimension is reversed between source and destination.

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, srcToDstTransposeMap=(/-3,1,4,2/), rc=rc)

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=redistHandle, rc=rc)

Redistribution of multi-tile Arrays is supported, although not shown as an example here. In default mode, the index space defined by both source and destination Arrays must match, regardless of how it is comprised by tiles. In particular, there is no restriction on the number of source and desination tiles, as long as both sides define the same global index space.

The situation is different in transpose mode. Here the number of source and destination tiles must match. In this case, the redistribution is defined tile-by-tile in order. If the provided srcToDstTransposeMap is of size rank, it is used for all of the tiles. The other supported option is where srcToDstTransposeMap is of size $rank \times tileCount$. In that case each source-destination tile-pair has its own transpose map.


28.2.18 Communication - SparseMatMul

Sparse matrix multiplication is a fundamental Array communication method. One frequently used application of this method is the interpolation between pairs of Arrays. The principle is this: the value of each element in the exclusive region of the destination Array is expressed as a linear combination of potentially all the exclusive elements of the source Array. Naturally most of the coefficients of these linear combinations will be zero and it is more efficient to store explicit information about the non-zero elements than to keep track of all the coefficients.

There is a choice to be made with respect to the format in which to store the information about the non-zero elements. One option is to store the value of each coefficient together with the corresponding destination element index and source element index. Destination and source indices could be expressed in terms of the corresponding DistGrid tile index together with the coordinate tuple within the tile. While this format may be the most natural way to express elements in the source and destination Array, it has two major drawbacks. First the coordinate tuple is dimCount specific and second the format is extremely bulky. For 2D source and destination Arrays it would require 6 integers to store the source and destination element information for each non-zero coefficient and matters get worse for higher dimensions.

Both problems can be circumvented by interpreting source and destination Arrays as sequentialized strings or vectors of elements. This is done by assigning a unique sequence index to each exclusive element in both Arrays. With that the operation of updating the elements in the destination Array as linear combinations of source Array elements takes the form of a sparse matrix multiplication.

The default sequence index rule assigns index $1$ to the minIndex corner element of the first tile of the DistGrid on which the Array is defined. It then increments the sequence index by $1$ for each element running through the DistGrid dimensions by order. The index space position of the DistGrid tiles does not affect the sequence labeling of elements. The default sequence indices for

  srcDistgrid = ESMF_DistGridCreate(minIndex=(/-1,0/), maxIndex=(/1,3/), rc=rc)

for each element are:

     -------------------------------------> 2nd dim
     |
     |   +------+------+------+------+
     |   |(-1,0)|      |      |(-1,3)|
     |   |      |      |      |      |
     |   |   1  |   4  |   7  |  10  |
     |   +------+------+------+------+
     |   |      |      |      |      |
     |   |      |      |      |      |
     |   |   2  |   5  |   8  |  11  |
     |   +------+------+------+------+
     |   | (1,0)|      |      | (1,3)|
     |   |      |      |      |      |
     |   |   3  |   6  |   9  |  12  |
     |   +------+------+------+------+
     |
     v
    1st dim

The assigned sequence indices are decomposition and distribution invariant by construction. Furthermore, when an Array is created with extra elements per DE on a DistGrid the sequence indices (which only cover the exclusive elements) remain unchanged.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)

  srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
    totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
    rc=rc)

The extra padding of 1 element in each direction around the exclusive elements on each DE are "invisible" to the Array sparse matrix multiplication method. These extra elements are either updated by the computational kernel or by Array halo operations.

An alternative way to assign sequence indices to all the elements in the tiles covered by a DistGrid object is to use a special ESMF_DistGridCreate() call. This call has been specifically designed for 1D cases with arbitrary, user-supplied sequence indices.

  seqIndexList(1) = localPet*10
  seqIndexList(2) = localPet*10 + 1
  dstDistgrid = ESMF_DistGridCreate(arbSeqIndexList=seqIndexList, rc=rc)

This call to ESMF_DistGridCreate() is collective across the current VM. The arbSeqIndexList argument specifies the PET-local arbitrary sequence indices that need to be covered by the local DE. The resulting DistGrid has one local DE per PET which covers the entire PET-local index range. The user supplied sequence indices must be unique, but the sequence may be interrupted. The four DEs of dstDistgrid have the following local 1D index space coordinates (given between "()") and sequence indices:

    covered by DE 0    covered by DE 1   covered by DE 2   covered by DE 3
    on PET 0           on PET 1          on PET 2          on PET 3
    ----------------------------------------------------------------------
    (1) : 0            (1) : 10          (1) : 20          (1) : 30
    (2) : 1            (2) : 11          (2) : 21          (2) : 31

Again the DistGrid object provides the sequence index labeling for the exclusive elements of an Array created on the DistGrid regardless of extra, non-exclusive elements.

  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)

With the definition of sequence indices, either by the default rule or as user provided arbitrary sequence indices, it is now possible to uniquely identify each exclusive element in the source and destination Array by a single integer number. Specifying a pair of source and destination elements takes two integer number regardless of the number of dimensions.

The information required to carry out a sparse matrix multiplication are the pair of source and destination sequence indices and the associated multiplication factor for each pair. ESMF requires this information in form of two Fortran arrays. The factors are stored in a 1D array of the appropriate type and kind, e.g. real(ESMF_KIND_R8)::factorList(:). Array sparse matrix multiplications are supported between Arrays of different type and kind. The type and kind of the factors can also be chosen freely. The sequence index pairs associated with the factors provided by factorList are stored in a 2D Fortran array of default integer kind of the shape integer::factorIndexList(2,:). The sequence indices of the source Array elements are stored in the first row of factorIndexList while the sequence indices of the destination Array elements are stored in the second row.

Each PET in the current VM must call into ESMF_ArraySMMStore() to precompute and store the communication pattern for the sparse matrix multiplication. The multiplication factors may be provided in parallel, i.e. multiple PETs may specify factorList and factorIndexList arguments when calling into ESMF_ArraySMMStore(). PETs that do not provide factors either call with factorList and factorIndexList arguments containing zero elements or issue the call omitting both arguments.

  if (localPet == 0) then
    allocate(factorList(1))               ! PET 0 specifies 1 factor
    allocate(factorIndexList(2,1))
    factorList = (/0.2/)                  ! factors
    factorIndexList(1,:) = (/5/)          ! seq indices into srcArray
    factorIndexList(2,:) = (/30/)         ! seq indices into dstArray
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, factorList=factorList, &
      factorIndexList=factorIndexList, rc=rc)

      
    deallocate(factorList)
    deallocate(factorIndexList)
  else if (localPet == 1) then
    allocate(factorList(3))               ! PET 1 specifies 3 factor
    allocate(factorIndexList(2,3))
    factorList = (/0.5, 0.5, 0.8/)        ! factors
    factorIndexList(1,:) = (/8, 2, 12/)   ! seq indices into srcArray
    factorIndexList(2,:) = (/11, 11, 30/) ! seq indices into dstArray
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, factorList=factorList, &
      factorIndexList=factorIndexList, rc=rc)

      
    deallocate(factorList)
    deallocate(factorIndexList)
  else
    ! PETs 2 and 3 do not provide factors
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, rc=rc)

      
  endif

The RouteHandle object sparseMatMulHandle produced by ESMF_ArraySMMStore() can now be used to call ESMF_ArraySMM() collectively across all PETs of the current VM to perform

     dstArray = 0.0
     do n=1, size(combinedFactorList)
         dstArray(combinedFactorIndexList(2, n)) += 
           combinedFactorList(n) * srcArray(combinedFactorIndexList(1, n))
     enddo
in parallel. Here combinedFactorList and combinedFactorIndexList are the combined lists defined by the respective local lists provided by PETs 0 and 1 in parallel. For this example
  call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
    routehandle=sparseMatMulHandle, rc=rc)

will initialize the entire dstArray to 0.0 and then update two elements:

   on DE 1:
   dstArray(2) = 0.5 * srcArray(0,0)  +  0.5 * srcArray(0,2)

and

   on DE 3:
   dstArray(1) = 0.2 * srcArray(0,1)  +  0.8 * srcArray(1,3).

The call to ESMF_ArraySMM() does provide the option to turn the default dstArray initialization off. If argument zeroregion is set to ESMF_REGION_EMPTY

  call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
    routehandle=sparseMatMulHandle, zeroregion=ESMF_REGION_EMPTY, rc=rc)

skips the initialization and elements in dstArray are updated according to:

     do n=1, size(combinedFactorList)
         dstArray(combinedFactorIndexList(2, n)) += 
           combinedFactorList(n) * srcArray(combinedFactorIndexList(1, n)).
     enddo

The ESMF_RouteHandle object returned by ESMF_ArraySMMStore() can be applied to any src/dst Array pairs that is compatible with the Array pair used during precomputation, i.e. any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The resources held by sparseMatMulHandle need to be deallocated by the user code before the handle becomes inaccessible.

  call ESMF_ArraySMMRelease(routehandle=sparseMatMulHandle, rc=rc)

The Array sparse matrix multiplication also applies to Arrays with undistributed dimensions. The undistributed dimensions are interpreted in a sequentialized manner, much like the distributed dimensions, introducing a second sequence index for source and destination elements. Sequence index 1 is assigned to the first element in the first (i.e. fastest varying in memory) undistributed dimension. The following undistributed elements are labeled in consecutive order as they are stored in memory.

In the simplest case the Array sparse matrix multiplication will apply an identity matrix to the vector of sequentialized undistributed Array elements for every non-zero element in the sparse matrix. The requirement in this case is that the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, be the same for source and destination Array.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
  srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
    totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
    distgridToArrayMap=(/1,2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
    distgridToArrayMap=(/2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)

Setting up factorList and factorIndexList is identical to the case for Arrays without undistributed dimensions. Also the call to ESMF_ArraySMMStore() remains unchanged. Internally, however, the source and destination Arrays are checked to make sure the total undistributed element count matches.

  if (localPet == 0) then
    allocate(factorList(1))               ! PET 0 specifies 1 factor
    allocate(factorIndexList(2,1))
    factorList = (/0.2/)                  ! factors
    factorIndexList(1,:) = (/5/)          ! seq indices into srcArray
    factorIndexList(2,:) = (/30/)         ! seq indices into dstArray
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, factorList=factorList, &
      factorIndexList=factorIndexList, rc=rc)

      
    deallocate(factorList)
    deallocate(factorIndexList)
  else if (localPet == 1) then
    allocate(factorList(3))               ! PET 1 specifies 3 factor
    allocate(factorIndexList(2,3))
    factorList = (/0.5, 0.5, 0.8/)        ! factors
    factorIndexList(1,:) = (/8, 2, 12/)   ! seq indices into srcArray
    factorIndexList(2,:) = (/11, 11, 30/) ! seq indices into dstArray
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, factorList=factorList, &
      factorIndexList=factorIndexList, rc=rc)

      
    deallocate(factorList)
    deallocate(factorIndexList)
  else
    ! PETs 2 and 3 do not provide factors
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, rc=rc)

  endif

The call into the ESMF_ArraySMM() operation is completely transparent with respect to whether source and/or destination Arrays contain undistributed dimensions.

  call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
    routehandle=sparseMatMulHandle, rc=rc)

This operation will initialize the entire dstArray to 0.0 and then update four elements:

   on DE 1:
   dstArray[1](2) = 0.5 * srcArray(0,0)[1]  +  0.5 * srcArray(0,2)[1],
   dstArray[2](2) = 0.5 * srcArray(0,0)[2]  +  0.5 * srcArray(0,2)[2]

and

   on DE 3:
   dstArray[1](1) = 0.2 * srcArray(0,1)[1]  +  0.8 * srcArray(1,3)[1],
   dstArray[2](1) = 0.2 * srcArray(0,1)[2]  +  0.8 * srcArray(1,3)[2].

Here indices between "()" refer to distributed dimensions while indices between "[]" correspond to undistributed dimensions.

In a more general version of the Array sparse matrix multiplication the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, need not be the same for source and destination Array. In this formulation each non-zero element of the sparse matrix is identified with a unique element in the source and destination Array. This requires a generalization of the factorIndexList argument which now must contain four integer numbers for each element. These numbers in sequence are the sequence index of the distributed dimensions and the sequence index of the undistributed dimensions of the element in the source Array, followed by the sequence index of the distributed dimensions and the sequence index of the undistributed dimensions of the element in the destination Array.

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
  srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
    totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
    distgridToArrayMap=(/1,2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)

  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
  dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
    distgridToArrayMap=(/2/), undistLBound=(/1/), undistUBound=(/4/), rc=rc)

Setting up factorList is identical to the previous cases since there is still only one value associated with each non-zero matrix element. However, each entry in factorIndexList now has 4 instead of just 2 components.

  if (localPet == 0) then
    allocate(factorList(1))               ! PET 0 specifies 1 factor
    allocate(factorIndexList(4,1))
    factorList = (/0.2/)                  ! factors
    factorIndexList(1,:) = (/5/)          ! seq indices into srcArray
    factorIndexList(2,:) = (/1/)          ! undistr. seq indices into srcArray
    factorIndexList(3,:) = (/30/)         ! seq indices into dstArray
    factorIndexList(4,:) = (/2/)          ! undistr. seq indices into dstArray
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, factorList=factorList, &
      factorIndexList=factorIndexList, rc=rc)

      
    deallocate(factorList)
    deallocate(factorIndexList)
  else if (localPet == 1) then
    allocate(factorList(3))               ! PET 1 specifies 3 factor
    allocate(factorIndexList(4,3))
    factorList = (/0.5, 0.5, 0.8/)        ! factors
    factorIndexList(1,:) = (/8, 2, 12/)   ! seq indices into srcArray
    factorIndexList(2,:) = (/2, 1, 1/)    ! undistr. seq indices into srcArray
    factorIndexList(3,:) = (/11, 11, 30/) ! seq indices into dstArray
    factorIndexList(4,:) = (/4, 4, 2/)    ! undistr. seq indices into dstArray
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, factorList=factorList, &
      factorIndexList=factorIndexList, rc=rc)

      
    deallocate(factorList)
    deallocate(factorIndexList)
  else
    ! PETs 2 and 3 do not provide factors
    
    call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
      routehandle=sparseMatMulHandle, rc=rc)

  endif

The call into the ESMF_ArraySMM() operation remains unchanged.

  call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
    routehandle=sparseMatMulHandle, rc=rc)

This operation will initialize the entire dstArray to 0.0 and then update two elements:

   on DE 1:
   dstArray[4](2) = 0.5 * srcArray(0,0)[1]  +  0.5 * srcArray(0,2)[2],

and

   on DE 3:
   dstArray[2](1) = 0.2 * srcArray(0,1)[1]  +  0.8 * srcArray(1,3)[1],

Here indices in $()$ refer to distributed dimensions while indices in $[]$ correspond to undistributed dimensions.


28.2.19 Communication - Scatter and Gather, revisited

The ESMF_ArrayScatter() and ESMF_ArrayGather() calls, introduced in section 28.2.14, provide a convenient way of communicating data between a Fortran array and all of the DEs of a single Array tile. A key requirement of ESMF_ArrayScatter() and ESMF_ArrayGather() is that the shape of the Fortran array and the Array tile must match. This means that the dimCount must be equal, and that the size of each dimension must match. Element reordering during scatter and gather is only supported on a per dimension level, based on the decompflag option available during DistGrid creation.

While the ESMF_ArrayScatter() and ESMF_ArrayGather() methods cover a broad, and important spectrum of cases, there are situations that require a different set of rules to scatter and gather data between a Fortran array and an ESMF Array object. For instance, it is often convenient to create an Array on a DistGrid that was created with arbitrary, user-supplied sequence indices. See section 36.3.5 for more background on DistGrids with arbitrary sequence indices.

  allocate(arbSeqIndexList(10))   ! each PET will have 10 elements
  
  do i=1, 10
    arbSeqIndexList(i) = (i-1)*petCount + localPet+1 ! initialize unique 
                                                     ! seq. indices
  enddo
  
  distgrid = ESMF_DistGridCreate(arbSeqIndexList=arbSeqIndexList, rc=rc)

  deallocate(arbSeqIndexList)
  
  call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=1, rc=rc)

  array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)

This array object holds 10 elements on each DE, and there is one DE per PET, for a total element count of 10 x petCount. The arbSeqIndexList, used during DistGrid creation, was constructed cyclic across all DEs. DE 0, for example, on a 4 PET run, would hold sequence indices 1, 5, 9, ... . DE 1 would hold 2, 6, 10, ..., and so on.

The usefulness of the user-specified arbitrary sequence indices becomes clear when they are interpreted as global element ids. The ArrayRedist() and ArraySMM() communication methods are based on sequence index mapping between source and destination Arrays. Other than providing a canonical sequence index order via the default sequence scheme, outlined in 28.2.18, ESMF does not place any restrictions on the sequence indices. Objects that were not created with user supplied sequence indices default to the ESMF sequence index order.

A common, and useful interpretation of the arbitrary sequence indices, specified during DistGrid creation, is that of relating them to the canonical ESMF sequence index order of another data object. Within this interpretation the array object created above could be viewed as an arbitrary distribution of a (petCount x 10) 2D array.

  if (localPet == 0) then
    allocate(farray(petCount,10)) ! allocate 2D Fortran array petCount x 10
    do j=1, 10
      do i=1, petCount
        farray(i,j) = 100 + (j-1)*petCount + i    ! initialize to something
      enddo
    enddo
  else
    allocate(farray(0,0)) ! must allocate an array of size 0 on all other PETs
  endif

For a 4 PET run, farray on PET 0 now holds the following data.

     -----1----2----3------------10-----> j
     |
     1   101, 105, 109, ....  , 137
     |
     2   102, 106, 110, ....  , 138
     |
     3   103, 107, 111, ....  , 139
     |
     4   104, 108, 112, ....  , 140
     |
     |
     v
    i

On all other PETs farray has a zero size allocation.

Following the sequence index interpretation from above, scattering the data contained in farray on PET 0 across the array object created further up, seems like a well defined operation. Looking at it a bit closer, it becomes clear that it is in fact more of a redistribution than a simple scatter operation. The general rule for such a "redist-scatter" operation, of a Fortran array, located on a single PET, into an ESMF Array, is to use the canonical ESMF sequence index scheme to label the elements of the Fortran array, and to send the data to the Array element with the same sequence index.

The just described "redist-scatter" operation is much more general than the standard ESMF_ArrayScatter() method. It does not require shape matching, and supports full element reordering based on the sequence indices. Before farray can be scattered across array in the described way, it must be wrapped into an ESMF Array object itself, essentially labeling the array elements according to the canonical sequence index scheme.

  distgridAux = ESMF_DistGridCreate(minIndex=(/1,1/), &
    maxIndex=(/petCount,10/), &
    regDecomp=(/1,1/), rc=rc) ! DistGrid with only 1 DE

The first step is to create a DistGrid object with only a single DE. This DE must be located on the PET on which the Fortran data array resides. In this example farray holds data on PET 0, which is where the default DELayout will place the single DE defined in the DistGrid. If the farray was setup on a different PET, an explicit DELayout would need to be created first, mapping the only DE to the PET on which the data is defined.

Next the Array wrapper object can be created from the farray and the just created DistGrid object.

  arrayAux = ESMF_ArrayCreate(farray=farray, distgrid=distgridAux, &
    indexflag=ESMF_INDEX_DELOCAL, rc=rc)

At this point all of the pieces are in place to use ESMF_ArrayRedist() to do the "redist-scatter" operation. The typical store/execute/release pattern must be followed.

  call ESMF_ArrayRedistStore(srcArray=arrayAux, dstArray=array, &
    routehandle=scatterHandle, rc=rc)

  call ESMF_ArrayRedist(srcArray=arrayAux, dstArray=array, &
    routehandle=scatterHandle, rc=rc)

In this example, after ESMF_ArrayRedist() was called, the content of array on a 4 PET run would look like this:

    PET 0:   101, 105, 109, ....  , 137
    PET 1:   102, 106, 110, ....  , 138
    PET 2:   103, 107, 111, ....  , 139
    PET 3:   104, 108, 112, ....  , 140

Once set up, scatterHandle can be used repeatedly to scatter data from farray on PET 0 to all the DEs of array. All of the resources should be released once scatterHandle is no longer needed.

  call ESMF_ArrayRedistRelease(routehandle=scatterHandle, rc=rc)

The opposite operation, i.e. gathering of the array data into farray on PET 0, follows a very similar setup. In fact, the arrayAux object already constructed for the scatter direction, can directly be re-used. The only thing that is different for the "redist-gather", are the srcArray and dstArray argument assignments, reflecting the opposite direction of data movement.

  call ESMF_ArrayRedistStore(srcArray=array, dstArray=arrayAux, &
    routehandle=gatherHandle, rc=rc)

  call ESMF_ArrayRedist(srcArray=array, dstArray=arrayAux, &
    routehandle=gatherHandle, rc=rc)

Just as for the scatter case, the gatherHandle can be used repeatedly to gather data from array into farray on PET 0. All of the resources should be released once gatherHandle is no longer needed.

  call ESMF_ArrayRedistRelease(routehandle=gatherHandle, rc=rc)

Finally the wrapper Array arrayAux and the associated DistGrid object can also be destroyed.

  call ESMF_ArrayDestroy(arrayAux, rc=rc)

  call ESMF_DistGridDestroy(distgridAux, rc=rc)

Further, the primary data objects of this example must be deallocated and destroyed.

  deallocate(farray)
  
  call ESMF_ArrayDestroy(array, rc=rc)

  call ESMF_DistGridDestroy(distgrid, rc=rc)


28.2.20 Non-blocking Communications

All ESMF_RouteHandle based communication methods, like ESMF_ArrayRedist(), ESMF_ArrayHalo() and ESMF_ArraySMM(), can be executed in blocking or non-blocking mode. The non-blocking feature is useful, for example, to overlap computation with communication, or to implement a more loosely synchronized inter-Component interaction scheme than is possible with the blocking communication mode.

Access to the non-blocking execution mode is provided uniformly across all RouteHandle based communication calls. Every such call contains the optional routesyncflag argument of type ESMF_RouteSync_Flag. Section 54.53 lists all of the valid settings for this flag.

It is an execution time decision to select whether to invoke a precomputed communication pattern, stored in a RouteHandle, in the blocking or non-blocking mode. Neither requires specifically precomputed RouteHandles - i.e. a RouteHandle is neither specifically blocking nor specifically non-blocking.

  call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
    routehandle=routehandle, rc=rc)

The returned RouteHandle routehandle can be used in blocking or non-blocking execution calls. The application is free to switch between both modes for the same RouteHandle.

By default routesyncflag is set to ESMF_ROUTESYNC_BLOCKING in all of the RouteHandle execution methods, and the behavior is that of the VM-wide collective communication calls described in the previous sections. In the blocking mode the user must assume that the communication call will not return until all PETs have exchanged the precomputed information. On the other hand, the user has no guarantee about the exact synchronization behavior, and it is unsafe to make specific assumptions. What is guaranteed in the blocking communication mode is that when the call returns on the local PET, all data exchanges associated with all local DEs have finished. This means that all in-bound data elements are valid and that all out-bound data elements can safely be overwritten by the user.

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_BLOCKING, rc=rc)

The same exchange pattern, that is encoded in routehandle, can be executed in non-blocking mode, simply by setting the appropriate routesyncflag when calling into ESMF_ArrayRedist().

At first sight there are obvious similarities between the non-blocking RouteHandle based execution paradigm and the non-blocking message passing calls provided by MPI. However, there are significant differences in the behavior of the non-blocking point-to-point calls that MPI defines and the non-blocking mode of the collective exchange patterns described by ESMF RouteHandles.

Setting routesyncflag to ESMF_ROUTESYNC_NBSTART in any RouteHandle execution call returns immediately after all out-bound data has been moved into ESMF internal transfer buffers and the exchange has been initiated.

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_NBSTART, rc=rc)

Once a call with routesyncflag = ESMF_ROUTESYNC_NBSTART returns, it is safe to modify the out-bound data elements in the srcArray object. However, no guarantees are made for the in-bound data elements in dstArray at this phase of the non-blocking execution. It is unsafe to access these elements until the exchange has finished locally.

One way to ensure that the exchange has finished locally is to call with routesyncflag set to ESMF_ROUTESYNC_NBWAITFINISH.

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_NBWAITFINISH, rc=rc)

Calling with routesyncflag = ESMF_ROUTESYNC_NBWAITFINISH instructs the communication method to wait and block until the previously started exchange has finished, and has been processed locally according to the RouteHandle. Once the call returns, it is safe to access both in-bound and out-bound data elements in dstArray and srcArray, respectively.

Some situations require more flexibility than is provided by the ESMF_ROUTESYNC_NBSTART - ESMF_ROUTESYNC_NBWAITFINISH pair. For instance, a Component that needs to interact with several other Components, virtually simultaneously, would initiated several different exchanges with ESMF_ROUTESYNC_NBSTART. Calling with ESMF_ROUTESYNC_NBWAITFINISH for any of the outstanding exchanges may potentially block for a long time, lowering the throughput. In the worst case a dead lock situation may arise. Calling with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH addresses this problem.

  call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
    routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_NBTESTFINISH, &
    finishedflag=finishflag, rc=rc)

This call tests the locally outstanding data transfer operation in routehandle, and finishes the exchange as much as currently possible. It does not block until the entire exchange has finished locally, instead it returns immediately after one round of testing has been completed. The optional return argument finishedflag is set to .true. if the exchange is completely finished locally, and set to .false. otherwise.

The user code must decide, depending on the value of the returned finishedflag, whether additional calls are required to finish an outstanding non-blocking exchange. If so, it can be done by calling ESMF_ArrayRedist() repeatedly with ESMF_ROUTESYNC_NBTESTFINISH until finishedflag comes back with a value of .true.. Such a loop allows other pieces of user code to be executed between the calls. A call with ESMF_ROUTESYNC_NBWAITFINISH can alternatively be used to block until the exchange has locally finished.

Noteworthy property. It is allowable to invoke a RouteHandle based communication call with routesyncflag set to ESMF_ROUTESYNC_NBTESTFINISH or ESMF_ROUTESYNC_NBWAITFINISH on a specific RouteHandle without there being an outstanding non-blocking exchange. As a matter of fact, it is not required that there was ever a call made with ESMF_ROUTESYNC_NBSTART for the RouteHandle. In these cases the calls made with ESMF_ROUTESYNC_NBTESTFINISH or ESMF_ROUTESYNC_NBWAITFINISH will simply return immediately (with finishedflag set to .true.).

Noteworthy property. It is fine to mix blocking and non-blocking invocations of the same RouteHandle based communication call across the PETs. This means that it is fine for some PETs to issue the call with ESMF_ROUTESYNC_BLOCKING (or using the default), while other PETs call the same communication call with ESMF_ROUTESYNC_NBSTART.

Noteworthy restriction. A RouteHandle that is currently involved in an outstanding non-blocking exchange may not be used to start any further exchanges, neither blocking nor non-blocking. This restriction is independent of whether the newly started RouteHandle based exchange is made for the same or for different data objects.

28.3 Restrictions and Future Work

28.4 Design and Implementation Notes

The Array class is part of the ESMF index space layer and is built on top of the DistGrid and DELayout classes. The DELayout class introduces the notion of decomposition elements (DEs) and their layout across the available PETs. The DistGrid describes how index space is decomposed by assigning logically rectangular index space pieces or DE-local tiles to the DEs. The Array finally associates a local memory allocation with each local DE.

The following is a list of implementation specific details about the current ESMF Array.

28.5 Class API

28.5.1 ESMF_ArrayAssignment(=) - Array assignment


INTERFACE:

   interface assignment(=)
   array1 = array2
ARGUMENTS:
   type(ESMF_Array) :: array1
   type(ESMF_Array) :: array2
STATUS:

DESCRIPTION:

Assign array1 as an alias to the same ESMF Array object in memory as array2. If array2 is invalid, then array1 will be equally invalid after the assignment.

The arguments are:

array1
The ESMF_Array object on the left hand side of the assignment.
array2
The ESMF_Array object on the right hand side of the assignment.

28.5.2 ESMF_ArrayOperator(==) - Array equality operator


INTERFACE:

   interface operator(==)
   if (array1 == array2) then ... endif
   OR
   result = (array1 == array2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_Array), intent(in) :: array1
   type(ESMF_Array), intent(in) :: array2
STATUS:

DESCRIPTION:

Test whether array1 and array2 are valid aliases to the same ESMF Array object in memory. For a more general comparison of two ESMF Arrays, going beyond the simple alias test, the ESMF_ArrayMatch() function (not yet implemented) must be used.

The arguments are:

array1
The ESMF_Array object on the left hand side of the equality operation.
array2
The ESMF_Array object on the right hand side of the equality operation.

28.5.3 ESMF_ArrayOperator(/=) - Array not equal operator


INTERFACE:

   interface operator(/=)
   if (array1 /= array2) then ... endif
   OR
   result = (array1 /= array2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_Array), intent(in) :: array1
   type(ESMF_Array), intent(in) :: array2
STATUS:

DESCRIPTION:

Test whether array1 and array2 are not valid aliases to the same ESMF Array object in memory. For a more general comparison of two ESMF Arrays, going beyond the simple alias test, the ESMF_ArrayMatch() function (not yet implemented) must be used.

The arguments are:

array1
The ESMF_Array object on the left hand side of the non-equality operation.
array2
The ESMF_Array object on the right hand side of the non-equality operation.

28.5.4 ESMF_ArrayCopy - Copy data from one Array object to another


INTERFACE:

   subroutine ESMF_ArrayCopy(arrayOut, arrayIn, rc)
ARGUMENTS:
        type(ESMF_Array), intent(inout) :: arrayOut
        type(ESMF_Array), intent(in) :: arrayIn
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer, intent(out), optional :: rc
DESCRIPTION:

Copy data from one ESMF_Array object to another.

The arguments are:

arrayOut
ESMF_Array object into which to copy the data. The incoming arrayOut must already references a matching memory allocation.
arrayIn
ESMF_Array object that holds the data to be copied.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.5 ESMF_ArrayCreate - Create Array object from Fortran array pointer


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate() 
   function ESMF_ArrayCreateFrmPtr<rank><type><kind>(distgrid, farrayPtr, & 
   datacopyflag, distgridToArrayMap, computationalEdgeLWidth, & 
   computationalEdgeUWidth, computationalLWidth, & 
   computationalUWidth, totalLWidth, & 
   totalUWidth, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateDataPtr<rank><type><kind>
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: distgridToArrayMap(:) 
   integer, intent(in), optional :: computationalEdgeLWidth(:) 
   integer, intent(in), optional :: computationalEdgeUWidth(:) 
   integer, intent(in), optional :: computationalLWidth(:) 
   integer, intent(in), optional :: computationalUWidth(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from existing local native Fortran arrays with pointer attribute. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the farrayPtr argument which will be different on each PET. The bounds of the local arrays are preserved by this call and determine the bounds of the total region of the resulting Array object. Bounds of the DE-local exclusive regions are set to be consistent with the total regions and the specified distgrid argument. Bounds for Array dimensions that are not distributed are automatically set to the bounds provided by farrayPtr.

This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.

The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank of farrayPtr.
farrayPtr
Valid native Fortran array with pointer attribute. Memory must be associated with the actual argument. The type/kind/rank information of farrayPtr will be used to set Array's properties accordingly. The shape of farrayPtr will be checked against the information contained in the distgrid. The bounds of farrayPtr will be preserved by this call and the bounds of the resulting Array object are set accordingly.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation provided by farrayPtr directly or will copy the data from farrayPtr into a new memory allocation. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE. Note that the ESMF_DATACOPY_REFERENCE option may not be safe when providing an array slice in farrayPtr.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in farrayPtr by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the farrayPtr argument in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped farrayPtr dimensions are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile. The default is a zero vector.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile. The default is a zero vector.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region exactly.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is a vector that contains the remaining number of elements in each direction as to fit the union of exclusive and computational region into the memory region provided by the farrayPtr argument.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.6 ESMF_ArrayCreate - Create Array object from Fortran array pointer w/ arbitrary seqIndices for halo


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate() 
   function ESMF_ArrayCreateFrmPtrArb<indexkind><rank><type><kind>(distgrid, & 
   farrayPtr, haloSeqIndexList, datacopyflag, & 
   distgridToArrayMap, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateDataPtrArb<rank><type><kind>
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
   integer(ESMF_KIND_<indexkind>), intent(in) :: haloSeqIndexList(:) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: distgridToArrayMap(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from existing local native Fortran arrays with pointer attribute, according to distgrid. Besides farrayPtr each PET must issue this call with identical arguments in order to create a consistent Array object. The bounds of the local arrays are preserved by this call and determine the bounds of the total region of the resulting Array object. Bounds of the DE-local exclusive regions are set to be consistent with the total regions and the specified distgrid argument. Bounds for Array dimensions that are not distributed are automatically set to the bounds provided by farrayPtr.

This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.

The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank of farrayPtr.
farrayPtr
Valid native Fortran array with pointer attribute. Memory must be associated with the actual argument. The type/kind/rank information of farrayPtr will be used to set Array's properties accordingly. The shape of farrayPtr will be checked against the information contained in the distgrid. The bounds of farrayPtr will be preserved by this call and the bounds of the resulting Array object are set accordingly.
haloSeqIndexList
One dimensional array containing sequence indices of local halo region. The size (and content) of haloSeqIndexList can (and typically will) be different on each PET. The haloSeqIndexList argument is of integer type, but can be of different kind in order to support both 32-bit (ESMF_KIND_I4) and 64-bit (ESMF_KIND_I8) indexing.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation provided by farrayPtr directly or will copy the data from farrayPtr into a new memory allocation. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE. Note that the ESMF_DATACOPY_REFERENCE option may not be safe when providing an array slice in farrayPtr.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in farrayPtr by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the farrayPtr argument in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped farrayPtr dimensions are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.7 ESMF_ArrayCreate - Create Array object from Fortran array


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate() 
   function ESMF_ArrayCreateAsmdSp<rank><type><kind>(distgrid, farray, & 
   indexflag, datacopyflag, distgridToArrayMap, & 
   computationalEdgeLWidth, computationalEdgeUWidth, computationalLWidth, & 
   computationalUWidth, totalLWidth, & 
   totalUWidth, undistLBound, undistUBound, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateDataAssmdShape<rank><type><kind>
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: distgridToArrayMap(:) 
   integer, intent(in), optional :: computationalEdgeLWidth(:) 
   integer, intent(in), optional :: computationalEdgeUWidth(:) 
   integer, intent(in), optional :: computationalLWidth(:) 
   integer, intent(in), optional :: computationalUWidth(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(in), optional :: undistLBound(:) 
   integer, intent(in), optional :: undistUBound(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from an existing local native Fortran array. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the farray argument which will be different on each PET. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for Array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.

This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.

The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank of farray.
farray
Valid native Fortran array, i.e. memory must be associated with the actual argument. The type/kind/rank information of farray will be used to set Array's properties accordingly. The shape of farray will be checked against the information contained in the distgrid.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation provided by farray directly or will copy the data from farray into a new memory allocation. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE. Note that the ESMF_DATACOPY_REFERENCE option may not be safe when providing an array slice in farray.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in farray by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the farray argument in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped farray dimensions are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile. The default is a zero vector.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile. The default is a zero vector.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region exactly.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is a vector that contains the remaining number of elements in each direction as to fit the union of exclusive and computational region into the memory region provided by the farray argument.
[undistLBound]
Lower bounds for the array dimensions that are not distributed. By default lbound is 1.
[undistUBound]
Upper bounds for the array dimensions that are not distributed. By default ubound is equal to the extent of the corresponding dimension in farray.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.8 ESMF_ArrayCreate - Create Array object from Fortran array w/ arbitrary seqIndices for halo


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate() 
   function ESMF_ArrayCreateAsmdSpArb<indexkind><rank><type><kind>(distgrid, & 
   farray, indexflag, haloSeqIndexList, datacopyflag, & 
   distgridToArrayMap, computationalEdgeLWidth, computationalEdgeUWidth, & 
   computationalLWidth, computationalUWidth, totalLWidth, totalUWidth, & 
   undistLBound, undistUBound, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateDataAssmdShapeArb<rank><type><kind>
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   type(ESMF_Index_Flag), intent(in) :: indexflag 
   integer(ESMF_KIND_<indexkind>), intent(in) :: haloSeqIndexList(:) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: distgridToArrayMap(:) 
   integer, intent(in), optional :: computationalEdgeLWidth(:) 
   integer, intent(in), optional :: computationalEdgeUWidth(:) 
   integer, intent(in), optional :: computationalLWidth(:) 
   integer, intent(in), optional :: computationalUWidth(:) 
   integer, intent(in), optional :: totalLWidth(:) 
   integer, intent(in), optional :: totalUWidth(:) 
   integer, intent(in), optional :: undistLBound(:) 
   integer, intent(in), optional :: undistUBound(:) 
   character (len=*), intent(in), optional :: name 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from an existing local native Fortran array. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the farray argument which will be different on each PET. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for Array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.

This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.

The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank of farray.
farray
Valid native Fortran array, i.e. memory must be associated with the actual argument. The type/kind/rank information of farray will be used to set Array's properties accordingly. The shape of farray will be checked against the information contained in the distgrid.
indexflag
Indicate how DE-local indices are defined. See section 54.29 for a list of valid indexflag options.
haloSeqIndexList
One dimensional array containing sequence indices of local halo region. The size (and content) of haloSeqIndexList can (and typically will) be different on each PET. The haloSeqIndexList argument is of integer type, but can be of different kind in order to support both 32-bit (ESMF_KIND_I4) and 64-bit (ESMF_KIND_I8) indexing.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation provided by farray directly or will copy the data from farray into a new memory allocation. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE. Note that the ESMF_DATACOPY_REFERENCE option may not be safe when providing an array slice in farray.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in farray by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the farray argument in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped farray dimensions are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile. The default is a zero vector.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile. The default is a zero vector.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region exactly.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is a vector that contains the remaining number of elements in each direction as to fit the union of exclusive and computational region into the memory region provided by the farray argument.
[undistLBound]
Lower bounds for the array dimensions that are not distributed. By default lbound is 1.
[undistUBound]
Upper bounds for the array dimensions that are not distributed. By default ubound is equal to the extent of the corresponding dimension in farray.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.9 ESMF_ArrayCreate - Create Array object from a list of LocalArray objects


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateLocalArray(distgrid, localarrayList, &
     indexflag, datacopyflag, distgridToArrayMap, computationalEdgeLWidth, &
     computationalEdgeUWidth, computationalLWidth, computationalUWidth, &
     totalLWidth, totalUWidth, undistLBound, undistUBound, name, rc)
RETURN VALUE:
     type(ESMF_Array) :: ESMF_ArrayCreateLocalArray
ARGUMENTS:
        type(ESMF_DistGrid), intent(in) :: distgrid
        type(ESMF_LocalArray), intent(in) :: localarrayList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_Index_Flag), intent(in), optional :: indexflag
        type(ESMF_DataCopy_Flag),intent(in), optional :: datacopyflag
        integer, intent(in), optional :: distgridToArrayMap(:)
        integer, intent(in), optional :: computationalEdgeLWidth(:)
        integer, intent(in), optional :: computationalEdgeUWidth(:)
        integer, intent(in), optional :: computationalLWidth(:)
        integer, intent(in), optional :: computationalUWidth(:)
        integer, intent(in), optional :: totalLWidth(:)
        integer, intent(in), optional :: totalUWidth(:)
        integer, intent(in), optional :: undistLBound(:)
        integer, intent(in), optional :: undistUBound(:)
        character (len=*), intent(in), optional :: name
        integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from existing ESMF_LocalArray objects. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the localarrayList argument which will be different on each PET. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for Array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.

This interface is able to handle multiple DEs per PET.

The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank specified in arrayspec, otherwise a runtime ESMF error will be raised.
localarrayList
List of valid ESMF_LocalArray objects, i.e. memory must be associated with the actual arguments. The type/kind/rank information of all localarrayList elements must be identical and will be used to set Array's properties accordingly. The shape of each localarrayList element will be checked against the information contained in the distgrid.
[indexflag]
Indicate how DE-local indices are defined. By default, the exclusive region of each DE is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated DistGrid. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation of the arrays provided in localarrayList directly, or will copy the actual data into new memory allocations. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in the localarrayList elements by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the localarrayList elements in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped dimensions in the localarrayList elements are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region exactly.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is a vector that contains the remaining number of elements in each direction as to fit the union of exclusive and computational region into the memory region provided by the localarrayList argument.
[undistLBound]
Lower bounds for the array dimensions that are not distributed. By default lbound is 1.
[undistUBound]
Upper bounds for the array dimensions that are not distributed. By default ubound is equal to the extent of the corresponding dimension in localarrayList.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.10 ESMF_ArrayCreate - Create Array object from a list of LocalArray objects w/ arbitrary seqIndices for halo


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateLocalArrayArb<indexkind>(distgrid, localarrayList, &
   haloSeqIndexList, indexflag, datacopyflag, &
   distgridToArrayMap, computationalEdgeLWidth, computationalEdgeUWidth, &
   computationalLWidth, computationalUWidth, &
   totalLWidth, totalUWidth, undistLBound, undistUBound, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateLocalArrayArb
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid
   type(ESMF_LocalArray), intent(in) :: localarrayList(:)
   integer(ESMF_KIND_<indexkind>), intent(in) :: haloSeqIndexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_Index_Flag), intent(in), optional :: indexflag
   type(ESMF_DataCopy_Flag),intent(in), optional :: datacopyflag
   integer, intent(in), optional :: distgridToArrayMap(:)
   integer, intent(in), optional :: computationalEdgeLWidth(:)
   integer, intent(in), optional :: computationalEdgeUWidth(:)
   integer, intent(in), optional :: computationalLWidth(:)
   integer, intent(in), optional :: computationalUWidth(:)
   integer, intent(in), optional :: totalLWidth(:)
   integer, intent(in), optional :: totalUWidth(:)
   integer, intent(in), optional :: undistLBound(:)
   integer, intent(in), optional :: undistUBound(:)
   character (len=*), intent(in), optional :: name
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from existing ESMF_LocalArray objects according to distgrid. Each PET must issue this call in unison in order to create a consistent Array object. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank specified in arrayspec, otherwise a runtime ESMF error will be raised.
localarrayList
List of valid ESMF_LocalArray objects, i.e. memory must be associated with the actual arguments. The type/kind/rank information of all localarrayList elements must be identical and will be used to set Array's properties accordingly. The shape of each localarrayList element will be checked against the information contained in the distgrid.
haloSeqIndexList
One dimensional array containing sequence indices of local halo region. The size (and content) of haloSeqIndexList can (and typically will) be different on each PET. The haloSeqIndexList argument is of integer type, but can be of different kind in order to support both 32-bit (ESMF_KIND_I4) and 64-bit (ESMF_KIND_I8) indexing.
[indexflag]
Indicate how DE-local indices are defined. By default, the exclusive region of each DE is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated DistGrid. See section 54.29 for a list of valid indexflag options.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation of the arrays provided in localarrayList directly, or will copy the actual data into new memory allocations. For valid values see 54.13. The default is ESMF_DATACOPY_REFERENCE.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in the localarrayList elements by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the localarrayList elements in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped dimensions in the localarrayList elements are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region exactly.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is a vector that contains the remaining number of elements in each direction as to fit the union of exclusive and computational region into the memory region provided by the localarrayList argument.
[undistLBound]
Lower bounds for the array dimensions that are not distributed. By default lbound is 1.
[undistUBound]
Upper bounds for the array dimensions that are not distributed. By default ubound is equal to the extent of the corresponding dimension in localarrayList.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.11 ESMF_ArrayCreate - Create Array object from typekind (allocate memory)


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateAllocate(distgrid, typekind, &
     indexflag, pinflag, distgridToArrayMap, computationalEdgeLWidth, &
     computationalEdgeUWidth, computationalLWidth, computationalUWidth, &
     totalLWidth, totalUWidth, undistLBound, undistUBound, name, vm, rc)
RETURN VALUE:
     type(ESMF_Array) :: ESMF_ArrayCreateAllocate
ARGUMENTS:
        type(ESMF_DistGrid), intent(in) :: distgrid
        type(ESMF_TypeKind_Flag), intent(in) :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_Index_Flag), intent(in), optional :: indexflag
        type(ESMF_Pin_Flag), intent(in), optional :: pinflag
        integer, intent(in), optional :: distgridToArrayMap(:)
        integer, intent(in), optional :: computationalEdgeLWidth(:)
        integer, intent(in), optional :: computationalEdgeUWidth(:)
        integer, intent(in), optional :: computationalLWidth(:)
        integer, intent(in), optional :: computationalUWidth(:)
        integer, intent(in), optional :: totalLWidth(:)
        integer, intent(in), optional :: totalUWidth(:)
        integer, intent(in), optional :: undistLBound(:)
        integer, intent(in), optional :: undistUBound(:)
        character (len=*), intent(in), optional :: name
        type(ESMF_VM), intent(in), optional :: vm
        integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object and allocate uninitialized data space according to typekind and distgrid. The Array rank is indirectly determined by the incoming information. Each PET must issue this call in unison in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the distgrid and the optional Width arguments.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank specified in arrayspec, otherwise a runtime ESMF error will be raised.
typekind
The typekind of the Array. See section 54.61 for a list of valid typekind options.
[indexflag]
Indicate how DE-local indices are defined. By default, the exclusive region of each DE is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated DistGrid. See section 54.29 for a list of valid indexflag options.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in the newly allocated Array object by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the Array object in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped dimensions in the Array object are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is to accommodate the union of exclusive and computational region.
[undistLBound]
Lower bounds for the array dimensions that are not distributed.
[undistUBound]
Upper bounds for the array dimensions that are not distributed.
[name]
Name of the Array object.
[vm]
If present, the Array object is created on the specified ESMF_VM object. The default is to create on the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.12 ESMF_ArrayCreate - Create Array object from typekind (allocate memory) w/ arbitrary seqIndices for halo


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateAllocateArb<indexkind>(distgrid, typekind, &
   haloSeqIndexList, pinflag, distgridToArrayMap, &
   undistLBound, undistUBound, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateAllocateArb
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid
   type(ESMF_TypeKind_Flag), intent(in) :: typekind
   integer(ESMF_KIND_<indexkind>), intent(in) :: haloSeqIndexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_Pin_Flag), intent(in), optional :: pinflag
   integer, intent(in), optional :: distgridToArrayMap(:)
   integer, intent(in), optional :: undistLBound(:)
   integer, intent(in), optional :: undistUBound(:)
   character (len=*), intent(in), optional :: name
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object and allocate uninitialized data space according to typekind and distgrid. The Array rank is indirectly determined by the incoming information. Each PET must issue this call in unison in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the distgrid and haloSeqIndexList arguments.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank specified in arrayspec, otherwise a runtime ESMF error will be raised.
typekind
The typekind of the Array. See section 54.61 for a list of valid typekind options.
haloSeqIndexList
One dimensional array containing sequence indices of local halo region. The size (and content) of haloSeqIndexList can (and typically will) be different on each PET. The haloSeqIndexList argument is of integer type, but can be of different kind in order to support both 32-bit (ESMF_KIND_I4) and 64-bit (ESMF_KIND_I8) indexing.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in the newly allocated Array object by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the Array object in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped dimensions in the Array object are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[undistLBound]
Lower bounds for the array dimensions that are not distributed.
[undistUBound]
Upper bounds for the array dimensions that are not distributed.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.13 ESMF_ArrayCreate - Create Array object from ArraySpec (allocate memory)


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateAllocateAS(distgrid, arrayspec, &
     indexflag, pinflag, distgridToArrayMap, computationalEdgeLWidth, &
     computationalEdgeUWidth, computationalLWidth, computationalUWidth, &
     totalLWidth, totalUWidth, undistLBound, undistUBound, name, vm, rc)
RETURN VALUE:
     type(ESMF_Array) :: ESMF_ArrayCreateAllocateAS
ARGUMENTS:
        type(ESMF_DistGrid), intent(in) :: distgrid
        type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_Index_Flag), intent(in), optional :: indexflag
        type(ESMF_Pin_Flag), intent(in), optional :: pinflag
        integer, intent(in), optional :: distgridToArrayMap(:)
        integer, intent(in), optional :: computationalEdgeLWidth(:)
        integer, intent(in), optional :: computationalEdgeUWidth(:)
        integer, intent(in), optional :: computationalLWidth(:)
        integer, intent(in), optional :: computationalUWidth(:)
        integer, intent(in), optional :: totalLWidth(:)
        integer, intent(in), optional :: totalUWidth(:)
        integer, intent(in), optional :: undistLBound(:)
        integer, intent(in), optional :: undistUBound(:)
        character (len=*), intent(in), optional :: name
        type(ESMF_VM), intent(in), optional :: vm
        integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object and allocate uninitialized data space according to arrayspec and distgrid. Each PET must issue this call with identical arguments in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the arguments to this call: distgrid and the optional Width arguments.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank specified in arrayspec, otherwise a runtime ESMF error will be raised.
arrayspec
ESMF_ArraySpec object containing the type/kind/rank information.
[indexflag]
Indicate how DE-local indices are defined. By default, the exclusive region of each DE is placed to start at the local index space origin, i.e. (1, 1, ..., 1). Alternatively the DE-local index space can be aligned with the global index space, if a global index space is well defined by the associated DistGrid. See section 54.29 for a list of valid indexflag options.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in the newly allocated Array object by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the Array object in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped dimensions in the Array object are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[computationalEdgeLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalEdgeUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for DEs that are located on the edge of a tile.
[computationalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the computational region with respect to the lower corner of the exclusive region. The default is a zero vector.
[computationalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the computational region with respect to the upper corner of the exclusive region. The default is a zero vector.
[totalLWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the lower corner of the total memory region with respect to the lower corner of the exclusive region. The default is to accommodate the union of exclusive and computational region.
[totalUWidth]
This vector argument must have dimCount elements, where dimCount is specified in distgrid. It specifies the upper corner of the total memory region with respect to the upper corner of the exclusive region. The default is to accommodate the union of exclusive and computational region.
[undistLBound]
Lower bounds for the array dimensions that are not distributed.
[undistUBound]
Upper bounds for the array dimensions that are not distributed.
[name]
Name of the Array object.
[vm]
If present, the Array object is created on the specified ESMF_VM object. The default is to create on the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.14 ESMF_ArrayCreate - Create Array object from ArraySpec (allocate memory) w/ arbitrary seqIndices for halo


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateAllocateASArb<indexkind>(distgrid, arrayspec, &
   haloSeqIndexList, pinflag, distgridToArrayMap, &
   undistLBound, undistUBound, name, rc)
RETURN VALUE:
   type(ESMF_Array) :: ESMF_ArrayCreateAllocateASArb
ARGUMENTS:
   type(ESMF_DistGrid), intent(in) :: distgrid
   type(ESMF_ArraySpec), intent(in) :: arrayspec
   integer(ESMF_KIND_<indexkind>), intent(in) :: haloSeqIndexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_Pin_Flag), intent(in), optional :: pinflag
   integer, intent(in), optional :: distgridToArrayMap(:)
   integer, intent(in), optional :: undistLBound(:)
   integer, intent(in), optional :: undistUBound(:)
   character (len=*), intent(in), optional :: name
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object and allocate uninitialized data space according to arrayspec and distgrid. Each PET must issue this call in unison in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the arguments to this call: distgrid and haloSeqIndexList arguments.

The return value is the newly created ESMF_Array object.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs. The dimCount of distgrid must be smaller or equal to the rank specified in arrayspec, otherwise a runtime ESMF error will be raised.
arrayspec
ESMF_ArraySpec object containing the type/kind/rank information.
haloSeqIndexList
One dimensional array containing sequence indices of local halo region. The size (and content) of haloSeqIndexList can (and typically will) be different on each PET. The haloSeqIndexList argument is of integer type, but can be of different kind in order to support both 32-bit (ESMF_KIND_I4) and 64-bit (ESMF_KIND_I8) indexing.
[pinflag]
Specify which type of resource DEs are pinned to. See section 50.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.
[distgridToArrayMap]
List that contains as many elements as is indicated by distgrids's dimCount. The list elements map each dimension of the DistGrid object to a dimension in the newly allocated Array object by specifying the appropriate Array dimension index. The default is to map all of distgrid's dimensions against the lower dimensions of the Array object in sequence, i.e. distgridToArrayMap = (/1, 2, .../). Unmapped dimensions in the Array object are not decomposed dimensions and form a tensor of rank = Array.rank - DistGrid.dimCount. All distgridToArrayMap entries must be greater than or equal to zero and smaller than or equal to the Array rank. It is erroneous to specify the same entry multiple times unless it is zero. If the Array rank is less than the DistGrid dimCount then the default distgridToArrayMap will contain zeros for the dimCount - rank rightmost entries. A zero entry in the distgridToArrayMap indicates that the particular DistGrid dimension will be replicating the Array across the DEs along this direction.
[undistLBound]
Lower bounds for the array dimensions that are not distributed.
[undistUBound]
Upper bounds for the array dimensions that are not distributed.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.15 ESMF_ArrayCreate - Create Array object from an existing Array object


INTERFACE:

   ! Private name; call using ESMF_ArrayCreate()
   function ESMF_ArrayCreateFromArray(array, datacopyflag, delayout, &
     trailingUndistSlice, name, rc)
RETURN VALUE:
     type(ESMF_Array) :: ESMF_ArrayCreateFromArray
ARGUMENTS:
        type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
        type(ESMF_DELayout), intent(in), optional :: delayout
        integer, intent(in), optional :: trailingUndistSlice(:)
        character (len=*), intent(in), optional :: name
        integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_Array object from an existing Array. Supports array slicing.

The return value is the newly created ESMF_Array object.

The arguments are:

array
ESMF_Array object to be used as originator.
[datacopyflag]
Specifies whether the created Array object references the memory allocation provided by array directly or copies the data from array into a new memory allocation. For valid values see 54.13. The default is ESMF_DATACOPY_VALUE.
[delayout]
If present, override the DELayout of the incoming distgrid. By default use the DELayout defined in distgrid.
[trailingUndistSlice]
If present, the returned Array refers to a slice of array created by removing some or all of the trailing (i.e. rightmost) undistributed dimensions from the incoming Array object. The number of trailing dimensions removed is equal to the size $n$ of trailingUndistSlice, which must be less than or equal to the total number of trailing undistributed dimensions in array. The trailingUndistSlice argument is interpreted as an index tuple for the $n$ trailing undistributed dimensions in array in order, and the Array slice at that location is returned.
[name]
Name of the Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.16 ESMF_ArrayDestroy - Release resources associated with an Array object


INTERFACE:

   subroutine ESMF_ArrayDestroy(array, noGarbage, rc)
ARGUMENTS:
     type(ESMF_Array), intent(inout) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical, intent(in), optional :: noGarbage
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Destroy an ESMF_Array, releasing the resources associated with the object.

By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.

The arguments are:

array
ESMF_Array object to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.17 ESMF_ArrayGather - Gather a Fortran array from an ESMF_Array


INTERFACE:

   subroutine ESMF_ArrayGather(array, farray, rootPet, tile, vm, rc)
ARGUMENTS:
   type(ESMF_Array), intent(in) :: array 
   <type>(ESMF_KIND_<kind>), intent(out), target :: farray(<rank>) 
   integer, intent(in) :: rootPet 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(in), optional :: tile 
   type(ESMF_VM), intent(in), optional :: vm 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Gather the data of an ESMF_Array object into the farray located on rootPET. A single DistGrid tile of array must be gathered into farray. The optional tile argument allows selection of the tile. For Arrays defined on a single tile DistGrid the default selection (tile 1) will be correct. The shape of farray must match the shape of the tile in Array.

If the Array contains replicating DistGrid dimensions data will be gathered from the numerically higher DEs. Replicated data elements in numerically lower DEs will be ignored.

This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.

The arguments are:

array
The ESMF_Array object from which data will be gathered.
{farray}
The Fortran array into which to gather data. Only root must provide a valid farray, the other PETs may treat farray as an optional argument.
rootPet
PET that holds the valid destination array, i.e. farray.
[tile]
The DistGrid tile in array from which to gather farray. By default farray will be gathered from tile 1.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.18 ESMF_ArrayGet - Get object-wide Array information


INTERFACE:

   ! Private name; call using ESMF_ArrayGet()
 subroutine ESMF_ArrayGetDefault(array, arrayspec, typekind, &
     rank, localarrayList, indexflag, distgridToArrayMap, &
     distgridToPackedArrayMap, arrayToDistGridMap, undistLBound, &
     undistUBound, exclusiveLBound, exclusiveUBound, computationalLBound, &
     computationalUBound, totalLBound, totalUBound, computationalLWidth, &
     computationalUWidth, totalLWidth, totalUWidth, distgrid, &
     dimCount, undistDimCount, replicatedDimCount, &
     tileCount, minIndexPTile, maxIndexPTile, deToTileMap, indexCountPDe, &
     delayout, deCount, localDeCount, ssiLocalDeCount, localDeToDeMap, &
     localDeList, & ! DEPRECATED ARGUMENT
     isESMFAllocated, name, vm, rc)
ARGUMENTS:
     type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_ArraySpec), intent(out), optional :: arrayspec
     type(ESMF_TypeKind_Flag), intent(out), optional :: typekind
     integer, intent(out), optional :: rank
     type(ESMF_LocalArray), target, intent(out), optional :: localarrayList(:)
     type(ESMF_Index_Flag), intent(out), optional :: indexflag
     integer, target, intent(out), optional :: distgridToArrayMap(:)
     integer, target, intent(out), optional :: distgridToPackedArrayMap(:)
     integer, target, intent(out), optional :: arrayToDistGridMap(:)
     integer, target, intent(out), optional :: undistLBound(:)
     integer, target, intent(out), optional :: undistUBound(:)
     integer, target, intent(out), optional :: exclusiveLBound(:,:)
     integer, target, intent(out), optional :: exclusiveUBound(:,:)
     integer, target, intent(out), optional :: computationalLBound(:,:)
     integer, target, intent(out), optional :: computationalUBound(:,:)
     integer, target, intent(out), optional :: totalLBound(:,:)
     integer, target, intent(out), optional :: totalUBound(:,:)
     integer, target, intent(out), optional :: computationalLWidth(:,:)
     integer, target, intent(out), optional :: computationalUWidth(:,:)
     integer, target, intent(out), optional :: totalLWidth(:,:)
     integer, target, intent(out), optional :: totalUWidth(:,:)
     type(ESMF_DistGrid), intent(out), optional :: distgrid
     integer, intent(out), optional :: dimCount
     integer, intent(out), optional :: undistDimCount
     integer, intent(out), optional :: replicatedDimCount
     integer, intent(out), optional :: tileCount
     integer, intent(out), optional :: minIndexPTile(:,:)
     integer, intent(out), optional :: maxIndexPTile(:,:)
     integer, intent(out), optional :: deToTileMap(:)
     integer, intent(out), optional :: indexCountPDe(:,:)
     type(ESMF_DELayout), intent(out), optional :: delayout
     integer, intent(out), optional :: deCount
     integer, intent(out), optional :: localDeCount
     integer, intent(out), optional :: ssiLocalDeCount
     integer, intent(out), optional :: localDeToDeMap(:)
     integer, intent(out), optional :: localDeList(:) ! DEPRECATED ARGUMENT
     logical, intent(out), optional :: isESMFAllocated
     character(len=*), intent(out), optional :: name
     type(ESMF_VM), intent(out), optional :: vm
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get internal information.

This interface works for any number of DEs per PET.

The arguments are:

array
Queried ESMF_Array object.
[arrayspec]
ESMF_ArraySpec object containing the type/kind/rank information of the Array object.
[typekind]
TypeKind of the Array object.
[rank]
Rank of the Array object.
[localarrayList]
Upon return this holds a list of the associated ESMC_LocalArray objects. localarrayList must be allocated to be of size localDeCount or ssiLocalDeCount.
[indexflag]
Upon return this flag indicates how the DE-local indices are defined. See section 54.29 for a list of possible return values.
[distgridToArrayMap]
Upon return this list holds the Array dimensions against which the DistGrid dimensions are mapped. distgridToArrayMap must be allocated to be of size dimCount. An entry of zero indicates that the respective DistGrid dimension is replicating the Array across the DEs along this direction.
[distgridToPackedArrayMap]
Upon return this list holds the indices of the Array dimensions in packed format against which the DistGrid dimensions are mapped. distgridToPackedArrayMap must be allocated to be of size dimCount. An entry of zero indicates that the respective DistGrid dimension is replicating the Array across the DEs along this direction.
[arrayToDistGridMap]
Upon return this list holds the DistGrid dimensions against which the Array dimensions are mapped. arrayToDistGridMap must be allocated to be of size rank. An entry of zero indicates that the respective Array dimension is not decomposed, rendering it a tensor dimension.
[undistLBound]
Upon return this array holds the lower bounds of the undistributed dimensions of the Array. UndistLBound must be allocated to be of size rank-dimCount.
[undistUBound]
Upon return this array holds the upper bounds of the undistributed dimensions of the Array. UndistUBound must be allocated to be of size rank-dimCount.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive regions for all PET-local DEs. exclusiveLBound must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive regions for all PET-local DEs. exclusiveUBound must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[computationalLBound]
Upon return this holds the lower bounds of the computational regions for all PET-local DEs. computationalLBound must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[computationalUBound]
Upon return this holds the upper bounds of the computational regions for all PET-local DEs. computationalUBound must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[totalLBound]
Upon return this holds the lower bounds of the total regions for all PET-local DEs. totalLBound must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[totalUBound]
Upon return this holds the upper bounds of the total regions for all PET-local DEs. totalUBound must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[computationalLWidth]
Upon return this holds the lower width of the computational regions for all PET-local DEs. computationalLWidth must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[computationalUWidth]
Upon return this holds the upper width of the computational regions for all PET-local DEs. computationalUWidth must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[totalLWidth]
Upon return this holds the lower width of the total memory regions for all PET-local DEs. totalLWidth must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[totalUWidth]
Upon return this holds the upper width of the total memory regions for all PET-local DEs. totalUWidth must be allocated to be of size (dimCount, localDeCount) or (dimCount, ssiLocalDeCount).
[distgrid]
Upon return this holds the associated ESMF_DistGrid object.
[dimCount]
Number of dimensions (rank) of distgrid.
[undistDimCount]
Number of undistributed dimensions in the Array. (See Section 28.2.11 for an explanation of undistributed Array dimensions.)
[replicatedDimCount]
Number of replicated dimensions in the Array. (See Section 28.2.12 for an explanation of replicated Array dimensions.)
[tileCount]
Number of tiles in distgrid.
[minIndexPTile]
Lower index space corner per dim, per tile, with size(minIndexPTile) == (/dimCount, tileCount/).
[maxIndexPTile]
Upper index space corner per dim, per tile, with size(maxIndexPTile) == (/dimCount, tileCount/).
[deToTileMap]
List of tile id numbers, one for each DE, with size(deToTileMap) == (/deCount/)
[indexCountPDe]
Array of extents per dim, per de, with size(indexCountPDe) == (/dimCount, deCount/).
[delayout]
The associated ESMF_DELayout object.
[deCount]
The total number of DEs in the Array.
[localDeCount]
The number of DEs in the Array associated with the local PET.
[ssiLocalDeCount]
The number of DEs in the Array available to the local PET. This includes DEs that are local to other PETs on the same SSI, that are accessible via shared memory.
[localDeToDeMap]
Mapping between localDe indices and the (global) DEs associated with the local PET. The localDe index variables are discussed in sections 50.3.7 and 28.2.5. The provided actual argument must be of size localDeCount, or ssiLocalDeCount, and will be filled accordingly.
[localDeList]
DEPRECATED ARGUMENT! Please use the argument localDeToDeMap instead.
[isESMFAllocated]
Set to .true. for data allocations held by ESMF. Set to .false. otherwise.
[name]
Name of the Array object.
[vm
The VM on which the Array object was created.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.19 ESMF_ArrayGet - Get DE-local Array information for a specific dimension


INTERFACE:

   ! Private name; call using ESMF_ArrayGet()
   subroutine ESMF_ArrayGetPLocalDePDim(array, dim, localDe, &
     indexCount, indexList, rc)
ARGUMENTS:
     type(ESMF_Array), intent(in) :: array
     integer, intent(in) :: dim
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: localDe
     integer, intent(out), optional :: indexCount
     integer, intent(out), optional :: indexList(:)
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Get internal information per local DE, per dim.

This interface works for any number of DEs per PET.

The arguments are:

array
Queried ESMF_Array object.
dim
Dimension for which information is requested. [1,..,dimCount]
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
[indexCount]
DistGrid indexCount associated with localDe, dim.
[indexList]
List of DistGrid tile-local indices for localDe along dimension dim.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.20 ESMF_ArrayGet - Get a DE-local Fortran array pointer from an Array


INTERFACE:

   ! Private name; call using ESMF_ArrayGet() 
   subroutine ESMF_ArrayGetFPtr<rank><type><kind>(array, localDe, & 
   farrayPtr, rc)
ARGUMENTS:
   type(ESMF_Array), intent(in) :: array 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(in), optional :: localDe 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Access Fortran array pointer to the specified DE-local memory allocation of the Array object.

The arguments are:

array
Queried ESMF_Array object.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
farrayPtr
Upon return, farrayPtr points to the DE-local data allocation of localDe in array. It depends on the specific entry point of ESMF_ArrayCreate() used during array creation, which Fortran operations are supported on the returned farrayPtr. See 28.3 for more details.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.21 ESMF_ArrayGet - Get a DE-local LocalArray object from an Array


INTERFACE:

   ! Private name; call using ESMF_ArrayGet()
   subroutine ESMF_ArrayGetLocalArray(array, localDe, localarray, rc)
ARGUMENTS:
     type(ESMF_Array), intent(in) :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: localDe
     type(ESMF_LocalArray), intent(inout) :: localarray
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Provide access to ESMF_LocalArray object that holds data for the specified local DE.

The arguments are:

array
Queried ESMF_Array object.
[localDe]
Local DE for which information is requested. [0,..,localDeCount-1]. For localDeCount==1 the localDe argument may be omitted, in which case it will default to localDe=0.
localarray
Upon return localarray refers to the DE-local data allocation of array.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.22 ESMF_ArrayHalo - Execute an Array halo operation


INTERFACE:

   subroutine ESMF_ArrayHalo(array, routehandle, &
     routesyncflag, finishedflag, cancelledflag, checkflag, rc)
ARGUMENTS:
     type(ESMF_Array),          intent(inout)         :: array
     type(ESMF_RouteHandle),    intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_RouteSync_Flag), intent(in),  optional :: routesyncflag
     logical,                   intent(out), optional :: finishedflag
     logical,                   intent(out), optional :: cancelledflag
     logical,                   intent(in),  optional :: checkflag
     integer,                   intent(out), optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed Array halo operation for array. The array argument must match the respective Array used during ESMF_ArrayHaloStore() in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

See ESMF_ArrayHaloStore() on how to precompute routehandle.

This call is collective across the current VM.

array
ESMF_Array containing data to be haloed.
routehandle
Handle to the precomputed Route.
[routesyncflag]
Indicate communication option. Default is ESMF_ROUTESYNC_BLOCKING, resulting in a blocking operation. See section 54.53 for a complete list of valid settings.
[finishedflag]
Used in combination with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH. Returned finishedflag equal to .true. indicates that all operations have finished. A value of .false. indicates that there are still unfinished operations that require additional calls with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH, or a final call with routesyncflag = ESMF_ROUTESYNC_NBWAITFINISH. For all other routesyncflag settings the returned value in finishedflag is always .true..
[cancelledflag]
A value of .true. indicates that were cancelled communication operations. In this case the data in the dstArray must be considered invalid. It may have been partially modified by the call. A value of .false. indicates that none of the communication operations was cancelled. The data in dstArray is valid if finishedflag returns equal .true..
[checkflag]
If set to .TRUE. the input Array pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.23 ESMF_ArrayHaloRelease - Release resources associated with Array halo operation


INTERFACE:

   subroutine ESMF_ArrayHaloRelease(routehandle, noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: noGarbage
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with an Array halo operation. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.24 ESMF_ArrayHaloStore - Precompute an Array halo operation


INTERFACE:

     subroutine ESMF_ArrayHaloStore(array, routehandle, &
       startregion, haloLDepth, haloUDepth, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),            intent(inout)           :: array
     type(ESMF_RouteHandle),      intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_StartRegion_Flag), intent(in),    optional :: startregion
     integer,                     intent(in),    optional :: haloLDepth(:)
     integer,                     intent(in),    optional :: haloUDepth(:)
     integer,                     intent(inout), optional :: pipelineDepth
     integer,                     intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Store an Array halo operation over the data in array. By default, i.e. without specifying startregion, haloLDepth and haloUDepth, all elements in the total Array region that lie outside the exclusive region will be considered potential destination elements for halo. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.

Specifying startregion allows the shape of the effective halo region to be changed from the inside. Setting this flag to ESMF_STARTREGION_COMPUTATIONAL means that only elements outside the computational region of the Array are considered for potential destination elements for the halo operation. The default is ESMF_STARTREGION_EXCLUSIVE.

The haloLDepth and haloUDepth arguments allow to reduce the extent of the effective halo region. Starting at the region specified by startregion, the haloLDepth and haloUDepth define a halo depth in each direction. Note that the maximum halo region is limited by the total Array region, independent of the actual haloLDepth and haloUDepth setting. The total Array region is local DE specific. The haloLDepth and haloUDepth are interpreted as the maximum desired extent, reducing the potentially larger region available for the halo operation.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayHalo() on any Array that matches array in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

array
ESMF_Array containing data to be haloed. The data in the halo region may be destroyed by this call.
routehandle
Handle to the precomputed Route.
[startregion]
The start of the effective halo region on every DE. The default setting is ESMF_STARTREGION_EXCLUSIVE, rendering all non-exclusive elements potential halo destination elements. See section 54.56 for a complete list of valid settings.
[haloLDepth]
This vector specifies the lower corner of the effective halo region with respect to the lower corner of startregion. The size of haloLDepth must equal the number of distributed Array dimensions.
[haloUDepth]
This vector specifies the upper corner of the effective halo region with respect to the upper corner of startregion. The size of haloUDepth must equal the number of distributed Array dimensions.
[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a halo exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.25 ESMF_ArrayIsCreated - Check whether an Array object has been created


INTERFACE:

   function ESMF_ArrayIsCreated(array, rc)
RETURN VALUE:
     logical :: ESMF_ArrayIsCreated
ARGUMENTS:
     type(ESMF_Array), intent(in)            :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,          intent(out), optional :: rc
DESCRIPTION:

Return .true. if the array has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

array
ESMF_Array queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.26 ESMF_ArrayPrint - Print Array information


INTERFACE:

   subroutine ESMF_ArrayPrint(array, rc)
ARGUMENTS:
     type(ESMF_Array), intent(in)            :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,          intent(out), optional :: rc
STATUS:

DESCRIPTION:

Print internal information of the specified ESMF_Array object.

The arguments are:

array
ESMF_Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.27 ESMF_ArrayRead - Read Array data from a file


INTERFACE:

   subroutine ESMF_ArrayRead(array, fileName, variableName, &
     timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_Array),      intent(inout)         :: array
     character(*),          intent(in)            :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(*),          intent(in),  optional :: variableName
     integer,               intent(in),  optional :: timeslice
     type(ESMF_IOFmt_Flag), intent(in),  optional :: iofmt
     integer,               intent(out), optional :: rc
DESCRIPTION:

Read Array data from file and put it into an ESMF_Array object. For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

Limitations:

The arguments are:

array
The ESMF_Array object in which the read data is returned.
fileName
The name of the file from which Array data is read. If this is a multi-tile Array, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being read from a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be read from "myfile1.nc", tile 2 from "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[variableName]
Variable name in the file; default is the "name" of Array. Use this argument only in the I/O format (such as NetCDF) that supports variable name. If the I/O format does not support this (such as binary format), ESMF will return an error code.
[timeslice]
The time-slice number of the variable read from file.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.28 ESMF_ArrayRedist - Execute an Array redistribution


INTERFACE:

   subroutine ESMF_ArrayRedist(srcArray, dstArray, routehandle, &
     routesyncflag, finishedflag, cancelledflag, zeroregion, checkflag, rc)
ARGUMENTS:
     type(ESMF_Array),          intent(in),    optional :: srcArray
     type(ESMF_Array),          intent(inout), optional :: dstArray
     type(ESMF_RouteHandle),    intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_RouteSync_Flag), intent(in),    optional :: routesyncflag
     logical,                   intent(out),   optional :: finishedflag
     logical,                   intent(out),   optional :: cancelledflag
     type(ESMF_Region_Flag),    intent(in),    optional :: zeroregion
     logical,                   intent(in),    optional :: checkflag
     integer,                   intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed Array redistribution from srcArray to dstArray. Both srcArray and dstArray must match the respective Arrays used during ESMF_ArrayRedisttore() in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcArray and dstArray arguments are optional in support of the situation where srcArray and/or dstArray are not defined on all PETs. The srcArray and dstArray must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

See ESMF_ArrayRedistStore() on how to precompute routehandle.

This call is collective across the current VM.

[srcArray]
ESMF_Array with source data.
[dstArray]
ESMF_Array with destination data.
routehandle
Handle to the precomputed Route.
[routesyncflag]
Indicate communication option. Default is ESMF_ROUTESYNC_BLOCKING, resulting in a blocking operation. See section 54.53 for a complete list of valid settings.
[finishedflag]
Used in combination with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH. Returned finishedflag equal to .true. indicates that all operations have finished. A value of .false. indicates that there are still unfinished operations that require additional calls with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH, or a final call with routesyncflag = ESMF_ROUTESYNC_NBWAITFINISH. For all other routesyncflag settings the returned value in finishedflag is always .true..
[cancelledflag]
A value of .true. indicates that were cancelled communication operations. In this case the data in the dstArray must be considered invalid. It may have been partially modified by the call. A value of .false. indicates that none of the communication operations was cancelled. The data in dstArray is valid if finishedflag returns equal .true..
[zeroregion]
If set to ESMF_REGION_TOTAL the total regions of all DEs in dstArray will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in dstArray will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination Array that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings. The default is ESMF_REGION_SELECT.
[checkflag]
If set to .TRUE. the input Array pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.29 ESMF_ArrayRedistRelease - Release resources associated with Array redistribution


INTERFACE:

   subroutine ESMF_ArrayRedistRelease(routehandle, noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: noGarbage
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with an Array redistribution. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.30 ESMF_ArrayRedistStore - Precompute Array redistribution with local factor argument


INTERFACE:

   ! Private name; call using ESMF_ArrayRedistStore()
   subroutine ESMF_ArrayRedistStore<type><kind>(srcArray, dstArray, &
     routehandle, factor, srcToDstTransposeMap, &
     ignoreUnmatchedIndices, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),       intent(in)              :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
     <type>(ESMF_KIND_<kind>),intent(in)             :: factor
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),    optional :: srcToDstTransposeMap(:)
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: pipelineDepth
     integer,                intent(out),   optional :: rc
STATUS:

DESCRIPTION:

ESMF_ArrayRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArrayRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArrayRedistStore() method, as provided through the separate entry points shown in 28.5.30 and 28.5.32, is described in the following paragraphs as a whole.

Store an Array redistribution operation from srcArray to dstArray. Interface 28.5.30 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 28.5.32. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source Array, destination Array, and the factor may be of different <type><kind>. Further, source and destination Arrays may differ in shape, however, the number of elements must match.

The default redistribution operation, when srcToDstTransposeMap is not specified, corresponds to the identity mapping: each element of the sequentialized source Array is copied to the sequentialized destination Array element in order.

If the srcToDstTransposeMap argument is provided it must be identical across all PETs. The srcToDstTransposeMap allows source and destination Array dimensions to be transposed during the redistribution. To support this option, the number of source and destination Array dimensions must be equal and the size of the associated dimensions must match. See section 28.2.17 for more details about the use of the srcToDstTransposeMap argument.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayRedist() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

srcArray
ESMF_Array with source data.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

factor
Factor by which to multiply source data.

[srcToDstTransposeMap]
A list with as many entries as there are dimensions in srcArray, or tileCount times this many entries. Each entry maps the corresponding srcArray dimension against the specified dstArray dimension. Mixing distributed and undistributed dimensions is supported. Negative entries reverse the order of elements along the specified dimension when going from source to destination. When providing $rank \times tileCount$ elements in srcToDstTransposeMap, each block of size rank is associated with the corresponding tile (in order), and interpreted as the tile-specific transpose map.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when not all elements match between the srcArray and dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores unmatched indices.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a redist exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.31 ESMF_ArrayRedistStore - Precompute Array redistribution and transpose with local factor argument


INTERFACE:

   ! Private name; call using ESMF_ArrayRedistStore()
   subroutine ESMF_ArrayRedistStore<type><kind>TP(srcArray, dstArray, &
     routehandle, transposeRoutehandle, factor, &
     srcToDstTransposeMap, ignoreUnmatchedIndices, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),       intent(inout)           :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
     type(ESMF_RouteHandle), intent(inout)           :: transposeRoutehandle
     <type>(ESMF_KIND_<kind>),intent(in)             :: factor
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),    optional :: srcToDstTransposeMap(:)
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: pipelineDepth
     integer,                intent(out),   optional :: rc
DESCRIPTION:

ESMF_ArrayRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArrayRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArrayRedistStore() method, as provided through the separate entry points shown in 28.5.31 and 28.5.33, is described in the following paragraphs as a whole.

Store an Array redistribution operation from srcArray to dstArray. Interface 28.5.31 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 28.5.33. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source Array, destination Array, and the factor may be of different <type><kind>. Further, source and destination Arrays may differ in shape, however, the number of elements must match.

The default redistribution operation, when srcToDstTransposeMap is not specified, corresponds to the identity mapping: each element of the sequentialized source Array is copied to the sequentialized destination Array element in order.

If the srcToDstTransposeMap argument is provided it must be identical across all PETs. The srcToDstTransposeMap allows source and destination Array dimensions to be transposed during the redistribution. To support this option, the number of source and destination Array dimensions must be equal and the size of the associated dimensions must match. See section 28.2.17 for more details about the use of the srcToDstTransposeMap argument.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayRedist() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

srcArray
ESMF_Array with source data. The data in this Array may be destroyed by this call.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

transposeRoutehandle
Handle to the transposed matrix operation. The transposed operation goes from dstArray to srcArray.

factor
Factor by which to multiply source data.

[srcToDstTransposeMap]
A list with as many entries as there are dimensions in srcArray, or tileCount times this many entries. Each entry maps the corresponding srcArray dimension against the specified dstArray dimension. Mixing distributed and undistributed dimensions is supported. Negative entries reverse the order of elements along the specified dimension when going from source to destination. When providing $rank \times tileCount$ elements in srcToDstTransposeMap, each block of size rank is associated with the corresponding tile (in order), and interpreted as the tile-specific transpose map.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when not all elements match between the srcArray and dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores unmatched indices.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a redist exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.32 ESMF_ArrayRedistStore - Precompute Array redistribution without local factor argument


INTERFACE:

   ! Private name; call using ESMF_ArrayRedistStore()
   subroutine ESMF_ArrayRedistStoreNF(srcArray, dstArray, routehandle, &
     srcToDstTransposeMap, ignoreUnmatchedIndices, &
     pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),       intent(in)              :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),    optional :: srcToDstTransposeMap(:)
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: pipelineDepth
     integer,                intent(out),   optional :: rc
STATUS:

DESCRIPTION:

ESMF_ArrayRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArrayRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArrayRedistStore() method, as provided through the separate entry points shown in 28.5.30 and 28.5.32, is described in the following paragraphs as a whole.

Store an Array redistribution operation from srcArray to dstArray. Interface 28.5.30 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 28.5.32. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source Array, destination Array, and the factor may be of different <type><kind>. Further, source and destination Arrays may differ in shape, however, the number of elements must match.

The default redistribution operation, when srcToDstTransposeMap is not specified, corresponds to the identity mapping: each element of the sequentialized source Array is copied to the sequentialized destination Array element in order.

If the srcToDstTransposeMap argument is provided it must be identical across all PETs. The srcToDstTransposeMap allows source and destination Array dimensions to be transposed during the redistribution. To support this option, the number of source and destination Array dimensions must be equal and the size of the associated dimensions must match. See section 28.2.17 for more details about the use of the srcToDstTransposeMap argument.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayRedist() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcArray
ESMF_Array with source data.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

[srcToDstTransposeMap]
A list with as many entries as there are dimensions in srcArray, or tileCount times this many entries. Each entry maps the corresponding srcArray dimension against the specified dstArray dimension. Mixing distributed and undistributed dimensions is supported. Negative entries reverse the order of elements along the specified dimension when going from source to destination. When providing $rank \times tileCount$ elements in srcToDstTransposeMap, each block of size rank is associated with the corresponding tile (in order), and interpreted as the tile-specific transpose map.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when not all elements match between the srcArray and dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores unmatched indices.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a redist exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.33 ESMF_ArrayRedistStore - Precompute Array redistribution and transpose without local factor argument


INTERFACE:

   ! Private name; call using ESMF_ArrayRedistStore()
   subroutine ESMF_ArrayRedistStoreNFTP(srcArray, dstArray, routehandle, &
     transposeRoutehandle, srcToDstTransposeMap, &
     ignoreUnmatchedIndices, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),       intent(inout)           :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
     type(ESMF_RouteHandle), intent(inout)           :: transposeRoutehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),    optional :: srcToDstTransposeMap(:)
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: pipelineDepth
     integer,                intent(out),   optional :: rc
DESCRIPTION:

ESMF_ArrayRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArrayRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArrayRedistStore() method, as provided through the separate entry points shown in 28.5.31 and 28.5.33, is described in the following paragraphs as a whole.

Store an Array redistribution operation from srcArray to dstArray. Interface 28.5.31 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 28.5.33. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source Array, destination Array, and the factor may be of different <type><kind>. Further, source and destination Arrays may differ in shape, however, the number of elements must match.

The default redistribution operation, when srcToDstTransposeMap is not specified, corresponds to the identity mapping: each element of the sequentialized source Array is copied to the sequentialized destination Array element in order.

If the srcToDstTransposeMap argument is provided it must be identical across all PETs. The srcToDstTransposeMap allows source and destination Array dimensions to be transposed during the redistribution. To support this option, the number of source and destination Array dimensions must be equal and the size of the associated dimensions must match. See section 28.2.17 for more details about the use of the srcToDstTransposeMap argument.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayRedist() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcArray
ESMF_Array with source data. The data in this Array may be destroyed by this call.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

transposeRoutehandle
Handle to the transposed matrix operation. The transposed operation goes from dstArray to srcArray.

[srcToDstTransposeMap]
A list with as many entries as there are dimensions in srcArray, or tileCount times this many entries. Each entry maps the corresponding srcArray dimension against the specified dstArray dimension. Mixing distributed and undistributed dimensions is supported. Negative entries reverse the order of elements along the specified dimension when going from source to destination. When providing $rank \times tileCount$ elements in srcToDstTransposeMap, each block of size rank is associated with the corresponding tile (in order), and interpreted as the tile-specific transpose map.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when not all elements match between the srcArray and dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores unmatched indices.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a redist exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.34 ESMF_ArrayScatter - Scatter a Fortran array across the ESMF_Array


INTERFACE:

   subroutine ESMF_ArrayScatter(array, farray, rootPet, tile, vm, rc)
ARGUMENTS:
   type(ESMF_Array), intent(inout) :: array 
   <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) 
   integer, intent(in) :: rootPet 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   integer, intent(in), optional :: tile 
   type(ESMF_VM), intent(in), optional :: vm 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Scatter the data of farray located on rootPET across an ESMF_Array object. A single farray must be scattered across a single DistGrid tile in Array. The optional tile argument allows selection of the tile. For Arrays defined on a single tile DistGrid the default selection (tile 1) will be correct. The shape of farray must match the shape of the tile in Array.

If the Array contains replicating DistGrid dimensions data will be scattered across all of the replicated pieces.

This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.

The arguments are:

array
The ESMF_Array object across which data will be scattered.
{farray}
The Fortran array that is to be scattered. Only root must provide a valid farray, the other PETs may treat farray as an optional argument.
rootPet
PET that holds the valid data in farray.
[tile]
The DistGrid tile in array into which to scatter farray. By default farray will be scattered into tile 1.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.35 ESMF_ArraySet - Set object-wide Array information


INTERFACE:

   ! Private name; call using ESMF_ArraySet()
   subroutine ESMF_ArraySetDefault(array, computationalLWidth, &
     computationalUWidth, name, rc)
ARGUMENTS:
     type(ESMF_Array),   intent(inout)         :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,            intent(in),  optional :: computationalLWidth(:,:)
     integer,            intent(in),  optional :: computationalUWidth(:,:)
     character(len = *), intent(in),  optional :: name
     integer,            intent(out), optional :: rc
STATUS:

DESCRIPTION:

Sets adjustable settings in an ESMF_Array object. Arrays with tensor dimensions will set values for all tensor components.

The arguments are:

array
ESMF_Array object for which to set properties.
[computationalLWidth]
This argument must have of size (dimCount, localDeCount). computationalLWidth specifies the lower corner of the computational region with respect to the lower corner of the exclusive region for all local DEs.
[computationalUWidth]
This argument must have of size (dimCount, localDeCount). computationalUWidth specifies the upper corner of the computational region with respect to the upper corner of the exclusive region for all local DEs.
[name]
The Array name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.36 ESMF_ArraySet - Set DE-local Array information


INTERFACE:

   ! Private name; call using ESMF_ArraySet()
   subroutine ESMF_ArraySetPLocalDe(array, localDe, rimSeqIndex, rc)
ARGUMENTS:
     type(ESMF_Array),   intent(inout)         :: array
     integer,            intent(in)            :: localDe
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,            intent(in),  optional :: rimSeqIndex(:)
     integer,            intent(out), optional :: rc
STATUS:

DESCRIPTION:

Sets adjustable settings in an ESMF_Array object for a specific localDe.

The arguments are:

array
ESMF_Array object for which to set properties.
localDe
Local DE for which to set values.
[rimSeqIndex]
Sequence indices in the halo rim of localDe.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.37 ESMF_ArraySMM - Execute an Array sparse matrix multiplication


INTERFACE:

   subroutine ESMF_ArraySMM(srcArray, dstArray, routehandle, &
     routesyncflag, finishedflag, cancelledflag, zeroregion, termorderflag, &
     checkflag, dynamicMask, rc)
ARGUMENTS:
     type(ESMF_Array),               intent(in),    optional :: srcArray
     type(ESMF_Array),               intent(inout), optional :: dstArray
     type(ESMF_RouteHandle),         intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_RouteSync_Flag),      intent(in),    optional :: routesyncflag
     logical,                        intent(out),   optional :: finishedflag
     logical,                        intent(out),   optional :: cancelledflag
     type(ESMF_Region_Flag),         intent(in),    optional :: zeroregion
     type(ESMF_TermOrder_Flag),      intent(in),    optional :: termorderflag
     logical,                        intent(in),    optional :: checkflag
     type(ESMF_DynamicMask), target, intent(in),    optional :: dynamicMask
     integer,                        intent(out),   optional :: rc
STATUS:

DESCRIPTION:

Execute a precomputed Array sparse matrix multiplication from srcArray to dstArray. Both srcArray and dstArray must match the respective Arrays used during ESMF_ArraySMMStore() in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

The srcArray and dstArray arguments are optional in support of the situation where srcArray and/or dstArray are not defined on all PETs. The srcArray and dstArray must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

See ESMF_ArraySMMStore() on how to precompute routehandle. See section 28.2.18 for details on the operation ESMF_ArraySMM() performs.

This call is collective across the current VM.

[srcArray]
ESMF_Array with source data.
[dstArray]
ESMF_Array with destination data.
routehandle
Handle to the precomputed Route.
[routesyncflag]
Indicate communication option. Default is ESMF_ROUTESYNC_BLOCKING, resulting in a blocking operation. See section 54.53 for a complete list of valid settings.
[finishedflag]
Used in combination with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH. Returned finishedflag equal to .true. indicates that all operations have finished. A value of .false. indicates that there are still unfinished operations that require additional calls with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH, or a final call with routesyncflag = ESMF_ROUTESYNC_NBWAITFINISH. For all other routesyncflag settings the returned value in finishedflag is always .true..
[cancelledflag]
A value of .true. indicates that were cancelled communication operations. In this case the data in the dstArray must be considered invalid. It may have been partially modified by the call. A value of .false. indicates that none of the communication operations was cancelled. The data in dstArray is valid if finishedflag returns equal .true..
[zeroregion]
If set to ESMF_REGION_TOTAL (default) the total regions of all DEs in dstArray will be initialized to zero before updating the elements with the results of the sparse matrix multiplication. If set to ESMF_REGION_EMPTY the elements in dstArray will not be modified prior to the sparse matrix multiplication and results will be added to the incoming element values. Setting zeroregion to ESMF_REGION_SELECT will only zero out those elements in the destination Array that will be updated by the sparse matrix multiplication. See section 54.50 for a complete list of valid settings.
[termorderflag]
Specifies the order of the source side terms in all of the destination sums. The termorderflag only affects the order of terms during the execution of the RouteHandle. See the 37.2.1 section for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods. See 54.60 for a full list of options. The default setting depends on whether the dynamicMask argument is present or not. With dynamicMask argument present, the default of termorderflag is ESMF_TERMORDER_SRCSEQ. This ensures that all source terms are present on the destination side, and the interpolation can be calculated as a single sum. When dynamicMask is absent, the default of termorderflag is ESMF_TERMORDER_FREE, allowing maximum flexibility and partial sums for optimum performance.
[checkflag]
If set to .TRUE. the input Array pair will be checked for consistency with the precomputed operation provided by routehandle. If set to .FALSE. (default) only a very basic input check will be performed, leaving many inconsistencies undetected. Set checkflag to .FALSE. to achieve highest performance.
[dynamicMask]
Object holding dynamic masking information. See section 37.2.6 for a discussion of dynamic masking.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.38 ESMF_ArraySMMRelease - Release resources associated with Array sparse matrix multiplication


INTERFACE:

   subroutine ESMF_ArraySMMRelease(routehandle, noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: noGarbage
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Release resources associated with an Array sparse matrix multiplication. After this call routehandle becomes invalid.

routehandle
Handle to the precomputed Route.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.39 ESMF_ArraySMMStore - Precompute Array sparse matrix multiplication with local factors


INTERFACE:

   ! Private name; call using ESMF_ArraySMMStore()
   subroutine ESMF_ArraySMMStore<type><kind>(srcArray, dstArray, &
     routehandle, factorList, factorIndexList, &
     ignoreUnmatchedIndices, srcTermProcessing, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),          intent(in)              :: srcArray
     type(ESMF_Array),          intent(inout)           :: dstArray
     type(ESMF_RouteHandle),    intent(inout)           :: routehandle
     <type>(ESMF_KIND_<kind>), target, intent(in)       :: factorList(:)
     integer(ESMF_KIND_<kind>), intent(in)              :: factorIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                   intent(in),    optional :: ignoreUnmatchedIndices
     integer,                   intent(inout), optional :: srcTermProcessing
     integer,                   intent(inout), optional :: pipelineDepth
     integer,                   intent(out),   optional :: rc
STATUS:

DESCRIPTION:

ESMF_ArraySMMStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArraySMMStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArraySMMStore() method, as provided through the separate entry points shown in 28.5.39 and 28.5.41, is described in the following paragraphs as a whole.

Store an Array sparse matrix multiplication operation from srcArray to dstArray. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source and destination Arrays, as well as the supplied factorList argument, may be of different <type><kind>. Further source and destination Arrays may differ in shape and number of elements.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArraySMM() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

srcArray
ESMF_Array with source data.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

factorList
List of non-zero coefficients.

factorIndexList
Pairs of sequence indices for the factors stored in factorList.

The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.

In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcArray while factorIndexList(2,:) specifies the sequence index of the destination element in dstArray. For this format to be a valid option source and destination Arrays must have matching number of tensor elements (the product of the sizes of all Array tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.

The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcArray. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstArray.

See section 28.2.18 for details on the definition of Array sequence indices and tensor sequence indices.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcArray or dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.40 ESMF_ArraySMMStore - Precompute Array sparse matrix multiplication and transpose with local factors


INTERFACE:

   ! Private name; call using ESMF_ArraySMMStore()
   subroutine ESMF_ArraySMMStore<type><kind>TP(srcArray, dstArray, &
     routehandle, transposeRoutehandle, factorList, factorIndexList, &
     ignoreUnmatchedIndices, srcTermProcessing, pipelineDepth, &
     rc)
ARGUMENTS:
     type(ESMF_Array),          intent(inout)           :: srcArray
     type(ESMF_Array),          intent(inout)           :: dstArray
     type(ESMF_RouteHandle),    intent(inout)           :: routehandle
     type(ESMF_RouteHandle),    intent(inout)           :: transposeRoutehandle
     <type>(ESMF_KIND_<kind>), target, intent(in)       :: factorList(:)
     integer(ESMF_KIND_<kind>), intent(in)              :: factorIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                   intent(in),    optional :: ignoreUnmatchedIndices
     integer,                   intent(inout), optional :: srcTermProcessing
     integer,                   intent(inout), optional :: pipelineDepth
     integer,                   intent(out),   optional :: rc
DESCRIPTION:

ESMF_ArraySMMStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArraySMMStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArraySMMStore() method, as provided through the separate entry points shown in 28.5.40 and 28.5.42, is described in the following paragraphs as a whole.

Store an Array sparse matrix multiplication operation from srcArray to dstArray. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source and destination Arrays, as well as the supplied factorList argument, may be of different <type><kind>. Further source and destination Arrays may differ in shape and number of elements.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArraySMM() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.

This call is collective across the current VM.

srcArray
ESMF_Array with source data. The data in this Array may be destroyed by this call.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

[transposeRoutehandle]
Handle to the transposed matrix operation. The transposed operation goes from dstArray to srcArray.

factorList
List of non-zero coefficients.

factorIndexList
Pairs of sequence indices for the factors stored in factorList.

The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.

In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcArray while factorIndexList(2,:) specifies the sequence index of the destination element in dstArray. For this format to be a valid option source and destination Arrays must have matching number of tensor elements (the product of the sizes of all Array tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.

The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcArray. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstArray.

See section 28.2.18 for details on the definition of Array sequence indices and tensor sequence indices.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcArray or dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.41 ESMF_ArraySMMStore - Precompute Array sparse matrix multiplication without local factors


INTERFACE:

   ! Private name; call using ESMF_ArraySMMStore()
   subroutine ESMF_ArraySMMStoreNF(srcArray, dstArray, routehandle, &
     ignoreUnmatchedIndices, srcTermProcessing, pipelineDepth, &
     rc)
ARGUMENTS:
     type(ESMF_Array),       intent(in)              :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: srcTermProcessing
     integer,                intent(inout), optional :: pipelineDepth
     integer,                intent(out),   optional :: rc
STATUS:

DESCRIPTION:

ESMF_ArraySMMStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArraySMMStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArraySMMStore() method, as provided through the separate entry points shown in 28.5.39 and 28.5.41, is described in the following paragraphs as a whole.

Store an Array sparse matrix multiplication operation from srcArray to dstArray. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source and destination Arrays, as well as the supplied factorList argument, may be of different <type><kind>. Further source and destination Arrays may differ in shape and number of elements.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArraySMM() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcArray
ESMF_Array with source data.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcArray or dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.42 ESMF_ArraySMMStore - Precompute Array sparse matrix multiplication and transpose without local factors


INTERFACE:

   ! Private name; call using ESMF_ArraySMMStore()
   subroutine ESMF_ArraySMMStoreNFTP(srcArray, dstArray, routehandle, &
     transposeRoutehandle, ignoreUnmatchedIndices, &
     srcTermProcessing, pipelineDepth, rc)
ARGUMENTS:
     type(ESMF_Array),       intent(inout)           :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
     type(ESMF_RouteHandle), intent(inout)           :: transposeRoutehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: srcTermProcessing
     integer,                intent(inout), optional :: pipelineDepth
     integer,                intent(out),   optional :: rc
DESCRIPTION:

ESMF_ArraySMMStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_ArraySMMStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_ArraySMMStore() method, as provided through the separate entry points shown in 28.5.40 and 28.5.42, is described in the following paragraphs as a whole.

Store an Array sparse matrix multiplication operation from srcArray to dstArray. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.

Both srcArray and dstArray are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.18 for details on the definition of sequence indices.

Source and destination Arrays, as well as the supplied factorList argument, may be of different <type><kind>. Further source and destination Arrays may differ in shape and number of elements.

It is erroneous to specify the identical Array object for srcArray and dstArray arguments.

The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArraySMM() on any pair of Arrays that matches srcArray and dstArray in type, kind, and memory layout of the distributed dimensions. However, the size, number, and index order of undistributed dimensions may be different. See section 37.2.5 for a more detailed discussion of RouteHandle reusability.

This call is collective across the current VM.

srcArray
ESMF_Array with source data. The data in this Array may be destroyed by this call.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

routehandle
Handle to the precomputed Route.

[transposeRoutehandle]
Handle to the transposed matrix operation. The transposed operation goes from dstArray to srcArray.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcArray or dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.43 ESMF_ArraySMMStore - Precompute sparse matrix multiplication using factors read from file.


INTERFACE:

   ! Private name; call using ESMF_ArraySMMStore()
   subroutine ESMF_ArraySMMStoreFromFile(srcArray, dstArray, filename, &
     routehandle, ignoreUnmatchedIndices, &
     srcTermProcessing, pipelineDepth, rc)
 
   ! ARGUMENTS:
     type(ESMF_Array),       intent(in)              :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     character(len=*),       intent(in)              :: filename
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: srcTermProcessing
     integer,                intent(inout), optional :: pipeLineDepth
     integer,                intent(out),   optional :: rc
DESCRIPTION:

Compute an ESMF_RouteHandle using factors read from file.

The arguments are:

srcArray
ESMF_Array with source data.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

filename
Path to the file containing weights for creating an ESMF_RouteHandle. See (12.9) for a description of the SCRIP weight file format. Only "row", "col", and "S" variables are required. They must be one-dimensionsal with dimension "n_s".

routehandle
Handle to the ESMF_RouteHandle.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcArray or dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange. The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.44 ESMF_ArraySMMStore - Precompute sparse matrix multiplication and transpose using factors read from file.


INTERFACE:

   ! Private name; call using ESMF_ArraySMMStore()
   subroutine ESMF_ArraySMMStoreFromFileTP(srcArray, dstArray, filename, &
     routehandle, transposeRoutehandle, ignoreUnmatchedIndices,&
     srcTermProcessing, pipelineDepth, rc)
 
   ! ARGUMENTS:
     type(ESMF_Array),       intent(inout)           :: srcArray
     type(ESMF_Array),       intent(inout)           :: dstArray
     character(len=*),       intent(in)              :: filename
     type(ESMF_RouteHandle), intent(inout)           :: routehandle
     type(ESMF_RouteHandle), intent(inout)           :: transposeRoutehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),    optional :: ignoreUnmatchedIndices
     integer,                intent(inout), optional :: srcTermProcessing
     integer,                intent(inout), optional :: pipeLineDepth
     integer,                intent(out),   optional :: rc
DESCRIPTION:

Compute an ESMF_RouteHandle using factors read from file.

The arguments are:

srcArray
ESMF_Array with source data. The data in this Array may be destroyed by this call.

dstArray
ESMF_Array with destination data. The data in this Array may be destroyed by this call.

filename
Path to the file containing weights for creating an ESMF_RouteHandle. See (12.9) for a description of the SCRIP weight file format. Only "row", "col", and "S" variables are required. They must be one-dimensionsal with dimension "n_s".

routehandle
Handle to the ESMF_RouteHandle.

[transposeRoutehandle]
Handle to the transposed matrix operation. The transposed operation goes from dstArray to srcArray.

[ignoreUnmatchedIndices]
A logical flag that affects the behavior for when sequence indices in the sparse matrix are encountered that do not have a match on the srcArray or dstArray side. The default setting is .false., indicating that it is an error when such a situation is encountered. Setting ignoreUnmatchedIndices to .true. ignores entries with unmatched indices.

[srcTermProcessing]
The srcTermProcessing parameter controls how many source terms, located on the same PET and summing into the same destination element, are summed into partial sums on the source PET before being transferred to the destination PET. A value of 0 indicates that the entire arithmetic is done on the destination PET; source elements are neither multiplied by their factors nor added into partial sums before being sent off by the source PET. A value of 1 indicates that source elements are multiplied by their factors on the source side before being sent to the destination PET. Larger values of srcTermProcessing indicate the maximum number of terms in the partial sums on the source side.

Note that partial sums may lead to bit-for-bit differences in the results. See section 37.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.

The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the srcTermProcessing parameter. The intent on the srcTermProcessing argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the srcTermProcessing parameter, and the auto-tuning phase is skipped. In this case the srcTermProcessing argument is not modified on return. If the provided argument is $< 0$, the srcTermProcessing parameter is determined internally using the auto-tuning scheme. In this case the srcTermProcessing argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional srcTermProcessing argument is omitted.

[pipelineDepth]
The pipelineDepth parameter controls how many messages a PET may have outstanding during a sparse matrix exchange. Larger values of pipelineDepth typically lead to better performance. However, on some systems too large a value may lead to performance degradation, or runtime errors.

Note that the pipeline depth has no effect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange. The ESMF_ArraySMMStore() method implements an auto-tuning scheme for the pipelineDepth parameter. The intent on the pipelineDepth argument is "inout" in order to support both overriding and accessing the auto-tuning parameter. If an argument $>= 0$ is specified, it is used for the pipelineDepth parameter, and the auto-tuning phase is skipped. In this case the pipelineDepth argument is not modified on return. If the provided argument is $< 0$, the pipelineDepth parameter is determined internally using the auto-tuning scheme. In this case the pipelineDepth argument is re-set to the internally determined value on return. Auto-tuning is also used if the optional pipelineDepth argument is omitted.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.45 ESMF_ArraySync - Synchronize DEs across the Array in case of sharing


INTERFACE:

   subroutine ESMF_ArraySync(array, rc)
ARGUMENTS:
     type(ESMF_Array), intent(in)            :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,          intent(out), optional :: rc
DESCRIPTION:

Synchronizes access to DEs across array to make sure PETs correctly access the data for read and write when DEs are shared.

The arguments are:

array
Specified ESMF_Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.46 ESMF_ArrayValidate - Validate object-wide Array information


INTERFACE:

   subroutine ESMF_ArrayValidate(array, rc)
ARGUMENTS:
     type(ESMF_Array), intent(in)            :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,          intent(out), optional :: rc
STATUS:

DESCRIPTION:

Validates that the Array is internally consistent. The method returns an error code if problems are found.

The arguments are:

array
Specified ESMF_Array object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.47 ESMF_ArrayWrite - Write Array data into a file


INTERFACE:

   subroutine ESMF_ArrayWrite(array, fileName, &
       variableName, convention, purpose,  &
       overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
     type(ESMF_Array),           intent(in)            :: array
     character(*),               intent(in)            :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(*),               intent(in),  optional :: variableName
     character(*),               intent(in),  optional :: convention
     character(*),               intent(in),  optional :: purpose
     logical,                    intent(in),  optional :: overwrite
     type(ESMF_FileStatus_Flag), intent(in),  optional :: status
     integer,                    intent(in),  optional :: timeslice
     type(ESMF_IOFmt_Flag),      intent(in),  optional :: iofmt
     integer,                    intent(out), optional :: rc
DESCRIPTION:

Write Array data into a file. For this API to be functional, the environment variable ESMF_PIO should be set to either "internal" or "external" when the ESMF library is built. Please see the section on Data I/O, 38.2.

When convention and purpose arguments are specified, a NetCDF variable can be created with user-specified dimension labels and attributes. Dimension labels may be defined for both gridded and ungridded dimensions. Dimension labels for gridded dimensions are specified at the DistGrid level by attaching an ESMF Attribute package to it. The Attribute package must contain an attribute named by the pre-defined ESMF parameter ESMF_ATT_GRIDDED_DIM_LABELS. The corresponding value is an array of character strings specifying the desired names of the dimensions. Likewise, for ungridded dimensions, an Attribute package is attached at the Array level. The name of the name must be ESMF_ATT_UNGRIDDED_DIM_LABELS.

NetCDF attributes for the variable can also be specified. As with dimension labels, an Attribute package is added to the Array with the desired names and values. A value may be either a scalar character string, or a scalar or array of type integer, real, or double precision. Dimension label attributes can co-exist with variable attributes within a common Attribute package.

Limitations:

The arguments are:

array
The ESMF_Array object that contains data to be written.
fileName
The name of the output file to which Array data is written. If this is a multi-tile Array, then fileName must contain exactly one instance of "*"; this is a placeholder that will be replaced by the tile number, with each tile being written to a separate file. (For example, for a fileName of "myfile*.nc", tile 1 will be written to "myfile1.nc", tile 2 to "myfile2.nc", etc.) (This handling of the fileName for multi-tile I/O is subject to change.)
[variableName]
Variable name in the output file; default is the "name" of Array. Use this argument only in the I/O format (such as NetCDF) that supports variable name. If the I/O format does not support this (such as binary format), ESMF will return an error code.
[convention]
Specifies an Attribute package associated with the Array, used to create NetCDF dimension labels and attributes for the variable in the file. When this argument is present, the purpose argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[purpose]
Specifies an Attribute package associated with the Array, used to create NetCDF dimension labels and attributes for the variable in the file. When this argument is present, the convention argument must also be present. Use this argument only with a NetCDF I/O format. If binary format is used, ESMF will return an error code.
[overwrite]
A logical flag, the default is .false., i.e., existing Array data may not be overwritten. If .true., only the data corresponding to the Array's name will be be overwritten. If the timeslice option is given, only data for the given timeslice may be overwritten. Note that it is always an error to attempt to overwrite a NetCDF variable with data which has a different shape.
[status]
The file status. Please see Section 54.22 for the list of options. If not present, defaults to ESMF_FILESTATUS_UNKNOWN.
[timeslice]
Some I/O formats (e.g. NetCDF) support the output of data in form of time slices. An unlimited dimension called time is defined in the file variable for this capability. The timeslice argument provides access to the time dimension, and must have a positive value. The behavior of this option may depend on the setting of the overwrite flag:
overwrite = .false.:
If the timeslice value is less than the maximum time already in the file, the write will fail.
overwrite = .true.:
Any positive timeslice value is valid.
By default, i.e. by omitting the timeslice argument, no provisions for time slicing are made in the output file, however, if the file already contains a time axis for the variable, a timeslice one greater than the maximum will be written.
[iofmt]
The I/O format. Please see Section 54.30 for the list of options. If not present, defaults to ESMF_IOFMT_NETCDF.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.5.48 ESMF_SparseMatrixWrite - Write a sparse matrix to file


INTERFACE:

   subroutine ESMF_SparseMatrixWrite(factorList, factorIndexList, fileName, &
     rc)
ARGUMENTS:
     real(ESMF_KIND_R8),    intent(in)            :: factorList(:)
     integer(ESMF_KIND_I4), intent(in)            :: factorIndexList(:,:)
     character(*),          intent(in)            :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,               intent(out), optional :: rc
DESCRIPTION:

Write the factorList and factorIndexList into a NetCDF file. The data is stored in SCRIP format documented under section  (12.9).

Limitations:

The arguments are:

factorList
The sparse matrix factors to be written.
factorIndexList
The sparse matrix sequence indices to be written.
fileName
The name of the output file to be written.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.


28.6 Class API: DynamicMask Methods

28.6.1 ESMF_DynamicMaskSetR8R8R8 - Set DynamicMask for R8R8R8


INTERFACE:

   subroutine ESMF_DynamicMaskSetR8R8R8(dynamicMask, dynamicMaskRoutine, &
     handleAllElements, dynamicSrcMaskValue, &
     dynamicDstMaskValue, rc)
ARGUMENTS:
     type(ESMF_DynamicMask), intent(out)           :: dynamicMask
     procedure(ESMF_DynamicMaskRoutineR8R8R8)      :: dynamicMaskRoutine
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: handleAllElements
     real(ESMF_KIND_R8),     intent(in),  optional :: dynamicSrcMaskValue
     real(ESMF_KIND_R8),     intent(in),  optional :: dynamicDstMaskValue
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_DynamicMask object suitable for destination element typekind ESMF_TYPEKIND_R8, factor typekind ESMF_TYPEKIND_R8, and source element typekind ESMF_TYPEKIND_R8.

All values in dynamicMask will be reset by this call.

See section 37.2.6 for a general discussion of dynamic masking.

The arguments are:

dynamicMask
DynamicMask object.
dynamicMaskRoutine
The routine responsible for handling dynamically masked source and destination elements. See section 37.2.6 for the precise definition of the dynamicMaskRoutine procedure interface. The routine is only called on PETs where at least one interpolation element is identified for special handling.
[handleAllElements]
If set to .true., all local elements, regardless of their dynamic masking status, are made available to dynamicMaskRoutine for handling. This option can be used to implement fully customized interpolations based on the information provided by ESMF. The default is .false., meaning that only elements affected by dynamic masking will be handed to dynamicMaskRoutine.
[dynamicSrcMaskValue]
The value for which a source element is considered dynamically masked. The default is to not consider any source elements as dynamically masked.
[dynamicDstMaskValue]
The value for which a destination element is considered dynamically masked. The default is to not consider any destination elements as dynamically masked.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.6.2 ESMF_DynamicMaskSetR8R8R8V - Set DynamicMask for R8R8R8 with vectorization


INTERFACE:

   subroutine ESMF_DynamicMaskSetR8R8R8V(dynamicMask, dynamicMaskRoutine, &
     handleAllElements, dynamicSrcMaskValue, &
     dynamicDstMaskValue, rc)
ARGUMENTS:
     type(ESMF_DynamicMask), intent(out)           :: dynamicMask
     procedure(ESMF_DynamicMaskRoutineR8R8R8V)     :: dynamicMaskRoutine
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: handleAllElements
     real(ESMF_KIND_R8),     intent(in),  optional :: dynamicSrcMaskValue
     real(ESMF_KIND_R8),     intent(in),  optional :: dynamicDstMaskValue
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_DynamicMask object suitable for destination element typekind ESMF_TYPEKIND_R8, factor typekind ESMF_TYPEKIND_R8, and source element typekind ESMF_TYPEKIND_R8.

All values in dynamicMask will be reset by this call.

See section 37.2.6 for a general discussion of dynamic masking.

The arguments are:

dynamicMask
DynamicMask object.
dynamicMaskRoutine
The routine responsible for handling dynamically masked source and destination elements. See section 37.2.6 for the precise definition of the dynamicMaskRoutine procedure interface. The routine is only called on PETs where at least one interpolation element is identified for special handling.
[handleAllElements]
If set to .true., all local elements, regardless of their dynamic masking status, are made available to dynamicMaskRoutine for handling. This option can be used to implement fully customized interpolations based on the information provided by ESMF. The default is .false., meaning that only elements affected by dynamic masking will be handed to dynamicMaskRoutine.
[dynamicSrcMaskValue]
The value for which a source element is considered dynamically masked. The default is to not consider any source elements as dynamically masked.
[dynamicDstMaskValue]
The value for which a destination element is considered dynamically masked. The default is to not consider any destination elements as dynamically masked.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.6.3 ESMF_DynamicMaskSetR4R8R4 - Set DynamicMask for R4R8R4


INTERFACE:

   subroutine ESMF_DynamicMaskSetR4R8R4(dynamicMask, dynamicMaskRoutine, &
     handleAllElements, dynamicSrcMaskValue, &
     dynamicDstMaskValue, rc)
ARGUMENTS:
     type(ESMF_DynamicMask), intent(out)           :: dynamicMask
     procedure(ESMF_DynamicMaskRoutineR4R8R4)      :: dynamicMaskRoutine
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: handleAllElements
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicSrcMaskValue
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicDstMaskValue
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_DynamicMask object suitable for destination element typekind ESMF_TYPEKIND_R4, factor typekind ESMF_TYPEKIND_R8, and source element typekind ESMF_TYPEKIND_R4.

All values in dynamicMask will be reset by this call.

See section 37.2.6 for a general discussion of dynamic masking.

The arguments are:

dynamicMask
DynamicMask object.
dynamicMaskRoutine
The routine responsible for handling dynamically masked source and destination elements. See section 37.2.6 for the precise definition of the dynamicMaskRoutine procedure interface. The routine is only called on PETs where at least one interpolation element is identified for special handling.
[handleAllElements]
If set to .true., all local elements, regardless of their dynamic masking status, are made available to dynamicMaskRoutine for handling. This option can be used to implement fully customized interpolations based on the information provided by ESMF. The default is .false., meaning that only elements affected by dynamic masking will be handed to dynamicMaskRoutine.
[dynamicSrcMaskValue]
The value for which a source element is considered dynamically masked. The default is to not consider any source elements as dynamically masked.
[dynamicDstMaskValue]
The value for which a destination element is considered dynamically masked. The default is to not consider any destination elements as dynamically masked.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.6.4 ESMF_DynamicMaskSetR4R8R4V - Set DynamicMask for R4R8R4 with vectorization


INTERFACE:

   subroutine ESMF_DynamicMaskSetR4R8R4V(dynamicMask, dynamicMaskRoutine, &
     handleAllElements, dynamicSrcMaskValue, &
     dynamicDstMaskValue, rc)
ARGUMENTS:
     type(ESMF_DynamicMask), intent(out)           :: dynamicMask
     procedure(ESMF_DynamicMaskRoutineR4R8R4V)     :: dynamicMaskRoutine
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: handleAllElements
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicSrcMaskValue
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicDstMaskValue
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_DynamicMask object suitable for destination element typekind ESMF_TYPEKIND_R4, factor typekind ESMF_TYPEKIND_R8, and source element typekind ESMF_TYPEKIND_R4.

All values in dynamicMask will be reset by this call.

See section 37.2.6 for a general discussion of dynamic masking.

The arguments are:

dynamicMask
DynamicMask object.
dynamicMaskRoutine
The routine responsible for handling dynamically masked source and destination elements. See section 37.2.6 for the precise definition of the dynamicMaskRoutine procedure interface. The routine is only called on PETs where at least one interpolation element is identified for special handling.
[handleAllElements]
If set to .true., all local elements, regardless of their dynamic masking status, are made available to dynamicMaskRoutine for handling. This option can be used to implement fully customized interpolations based on the information provided by ESMF. The default is .false., meaning that only elements affected by dynamic masking will be handed to dynamicMaskRoutine.
[dynamicSrcMaskValue]
The value for which a source element is considered dynamically masked. The default is to not consider any source elements as dynamically masked.
[dynamicDstMaskValue]
The value for which a destination element is considered dynamically masked. The default is to not consider any destination elements as dynamically masked.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.6.5 ESMF_DynamicMaskSetR4R4R4 - Set DynamicMask for R4R4R4


INTERFACE:

   subroutine ESMF_DynamicMaskSetR4R4R4(dynamicMask, dynamicMaskRoutine, &
     handleAllElements, dynamicSrcMaskValue, &
     dynamicDstMaskValue, rc)
ARGUMENTS:
     type(ESMF_DynamicMask), intent(out)           :: dynamicMask
     procedure(ESMF_DynamicMaskRoutineR4R4R4)      :: dynamicMaskRoutine
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: handleAllElements
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicSrcMaskValue
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicDstMaskValue
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_DynamicMask object suitable for destination element typekind ESMF_TYPEKIND_R4, factor typekind ESMF_TYPEKIND_R4, and source element typekind ESMF_TYPEKIND_R4.

All values in dynamicMask will be reset by this call.

See section 37.2.6 for a general discussion of dynamic masking.

The arguments are:

dynamicMask
DynamicMask object.
dynamicMaskRoutine
The routine responsible for handling dynamically masked source and destination elements. See section 37.2.6 for the precise definition of the dynamicMaskRoutine procedure interface. The routine is only called on PETs where at least one interpolation element is identified for special handling.
[handleAllElements]
If set to .true., all local elements, regardless of their dynamic masking status, are made available to dynamicMaskRoutine for handling. This option can be used to implement fully customized interpolations based on the information provided by ESMF. The default is .false., meaning that only elements affected by dynamic masking will be handed to dynamicMaskRoutine.
[dynamicSrcMaskValue]
The value for which a source element is considered dynamically masked. The default is to not consider any source elements as dynamically masked.
[dynamicDstMaskValue]
The value for which a destination element is considered dynamically masked. The default is to not consider any destination elements as dynamically masked.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

28.6.6 ESMF_DynamicMaskSetR4R4R4V - Set DynamicMask for R4R4R4 with vectorization


INTERFACE:

   subroutine ESMF_DynamicMaskSetR4R4R4V(dynamicMask, dynamicMaskRoutine, &
     handleAllElements, dynamicSrcMaskValue, &
     dynamicDstMaskValue, rc)
ARGUMENTS:
     type(ESMF_DynamicMask), intent(out)           :: dynamicMask
     procedure(ESMF_DynamicMaskRoutineR4R4R4V)     :: dynamicMaskRoutine
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional :: handleAllElements
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicSrcMaskValue
     real(ESMF_KIND_R4),     intent(in),  optional :: dynamicDstMaskValue
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_DynamicMask object suitable for destination element typekind ESMF_TYPEKIND_R4, factor typekind ESMF_TYPEKIND_R4, and source element typekind ESMF_TYPEKIND_R4.

All values in dynamicMask will be reset by this call.

See section 37.2.6 for a general discussion of dynamic masking.

The arguments are:

dynamicMask
DynamicMask object.
dynamicMaskRoutine
The routine responsible for handling dynamically masked source and destination elements. See section 37.2.6 for the precise definition of the dynamicMaskRoutine procedure interface. The routine is only called on PETs where at least one interpolation element is identified for special handling.
[handleAllElements]
If set to .true., all local elements, regardless of their dynamic masking status, are made available to dynamicMaskRoutine for handling. This option can be used to implement fully customized interpolations based on the information provided by ESMF. The default is .false., meaning that only elements affected by dynamic masking will be handed to dynamicMaskRoutine.
[dynamicSrcMaskValue]
The value for which a source element is considered dynamically masked. The default is to not consider any source elements as dynamically masked.
[dynamicDstMaskValue]
The value for which a destination element is considered dynamically masked. The default is to not consider any destination elements as dynamically masked.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29 LocalArray Class

29.1 Description

The ESMF_LocalArray class provides a language independent representation of data in array format. One of the major functions of the LocalArray class is to bridge the Fortran/C/C++ language difference that exists with respect to array representation. All ESMF Field and Array data is internally stored in ESMF LocalArray objects allowing transparent access from Fortran and C/C++.

In the ESMF Fortran API the LocalArray becomes visible in those cases where a local PET may be associated with multiple pieces of an Array, e.g. if there are multiple DEs associated with a single PET. The Fortran language standard does not provide an array of arrays construct, however arrays of derived types holding arrays are possible. ESMF calls use arguments that are of type ESMF_LocalArray with dimension attributes where necessary.

29.2 Restrictions and Future Work

29.3 Class API

29.3.1 ESMF_LocalArrayAssignment(=) - LocalArray assignment


INTERFACE:

   interface assignment(=)
   localarray1 = localarray2
ARGUMENTS:
   type(ESMF_LocalArray) :: localarray1
   type(ESMF_LocalArray) :: localarray2
STATUS:

DESCRIPTION:

Assign localarray1 as an alias to the same ESMF LocalArray object in memory as localarray2. If localarray2 is invalid, then localarray1 will be equally invalid after the assignment.

The arguments are:

localarray1
The ESMF_LocalArray object on the left hand side of the assignment.
localarray2
The ESMF_LocalArray object on the right hand side of the assignment.

29.3.2 ESMF_LocalArrayOperator(==) - LocalArray equality operator


INTERFACE:

   interface operator(==)
   if (localarray1 == localarray2) then ... endif
   OR
   result = (localarray1 == localarray2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_LocalArray), intent(in) :: localarray1
   type(ESMF_LocalArray), intent(in) :: localarray2
STATUS:

DESCRIPTION:

Test whether localarray1 and localarray2 are valid aliases to the same ESMF LocalArray object in memory. For a more general comparison of two ESMF LocalArrays, going beyond the simple alias test, the ESMF_LocalArrayMatch() function (not yet implemented) must be used.

The arguments are:

localarray1
The ESMF_LocalArray object on the left hand side of the equality operation.
localarray2
The ESMF_LocalArray object on the right hand side of the equality operation.

29.3.3 ESMF_LocalArrayOperator(/=) - LocalArray not equal operator


INTERFACE:

   interface operator(/=)
   if (localarray1 /= localarray2) then ... endif
   OR
   result = (localarray1 /= localarray2)
RETURN VALUE:
   logical :: result
ARGUMENTS:
   type(ESMF_LocalArray), intent(in) :: localarray1
   type(ESMF_LocalArray), intent(in) :: localarray2
STATUS:

DESCRIPTION:

Test whether localarray1 and localarray2 are not valid aliases to the same ESMF LocalArray object in memory. For a more general comparison of two ESMF LocalArrays, going beyond the simple alias test, the ESMF_LocalArrayMatch() function (not yet implemented) must be used.

The arguments are:

localarray1
The ESMF_LocalArray object on the left hand side of the non-equality operation.
localarray2
The ESMF_LocalArray object on the right hand side of the non-equality operation.

29.3.4 ESMF_LocalArrayCreate - Create a LocalArray by explicitly specifying typekind and rank arguments


INTERFACE:

   ! Private name; call using ESMF_LocalArrayCreate()
   function ESMF_LocalArrayCreateByTKR(typekind, rank, totalCount, &
     totalLBound, totalUBound, rc)
RETURN VALUE:
     type(ESMF_LocalArray) :: ESMF_LocalArrayCreateByTKR
ARGUMENTS:
     type(ESMF_TypeKind_Flag), intent(in) :: typekind
     integer, intent(in) :: rank
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: totalCount(:)
     integer, intent(in), optional :: totalLBound(:)
     integer, intent(in), optional :: totalUBound(:)
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create a new ESMF_LocalArray and allocate data space, which remains uninitialized. The return value is a new LocalArray.

The arguments are:

typekind
Array typekind. See section 54.61 for valid values.
rank
Array rank (dimensionality, 1D, 2D, etc). Maximum allowed is 7D.
[totalCount]
The number of items in each dimension of the array. This is a 1D integer array the same length as the rank. The count argument may be omitted if both totalLBound and totalUBound arguments are present.
[totalLBound]
An integer array of length rank, with the lower index for each dimension.
[totalUBound]
An integer array of length rank, with the upper index for each dimension.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.5 ESMF_LocalArrayCreate - Create a LocalArray by specifying an ArraySpec


INTERFACE:

   ! Private name; call using ESMF_LocalArrayCreate()
   function ESMF_LocalArrayCreateBySpec(arrayspec, totalCount, &
     totalLBound, totalUBound, rc)
RETURN VALUE:
     type(ESMF_LocalArray) :: ESMF_LocalArrayCreateBySpec
ARGUMENTS:
     type(ESMF_ArraySpec), intent(in) :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(in), optional :: totalCount(:)
     integer, intent(in), optional :: totalLBound(:)
     integer, intent(in), optional :: totalUBound(:)
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create a new ESMF_LocalArray and allocate data space, which remains uninitialized. The return value is a new LocalArray.

The arguments are:

arrayspec
ArraySpec object specifying typekind and rank.
[totalCount]
The number of items in each dimension of the array. This is a 1D integer array the same length as the rank. The count argument may be omitted if both totalLBound and totalUBound arguments are present.
[totalLBound]
An integer array of length rank, with the lower index for each dimension.
[totalUBound]
An integer array of length rank, with the upper index for each dimension.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.6 ESMF_LocalArrayCreate - Create a LocalArray from pre-existing LocalArray


INTERFACE:

   ! Private name; call using ESMF_LocalArrayCreate()
   function ESMF_LocalArrayCreateCopy(localarray, rc)
RETURN VALUE:
     type(ESMF_LocalArray) :: ESMF_LocalArrayCreateCopy
ARGUMENTS:
     type(ESMF_LocalArray), intent(in) :: localarray
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Perform a deep copy of an existing ESMF_LocalArray object. The return value is a new LocalArray.

The arguments are:

localarray
Existing LocalArray to be copied.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.7 ESMF_LocalArrayCreate - Create a LocalArray from a Fortran pointer (associated or unassociated)


INTERFACE:

   ! Private name; call using ESMF_LocalArrayCreate() 
   function ESMF_LocalArrCreateByPtr<rank><type><kind>(farrayPtr, & 
   datacopyflag, totalCount, totalLBound, totalUBound, rc)
RETURN VALUE:
   type(ESMF_LocalArray) :: ESMF_LocalArrCreateByPtr<rank><type><kind>
ARGUMENTS:
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(in), optional :: totalCount(:) 
   integer, intent(in), optional :: totalLBound(:) 
   integer, intent(in), optional :: totalUBound(:) 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Creates an ESMF_LocalArray based on a Fortran array pointer. Two cases must be distinguished.

First, if farrayPtr is associated the optional datacopyflag argument may be used to indicate whether the associated data is to be copied or referenced. For associated farrayPtr the optional totalCount, totalLBound and totalUBound arguments need not be specified. However, all present arguments will be checked against farrayPtr for consistency.

Second, if farrayPtr is unassociated the optional argument datacopyflag must not be specified. However, in this case a complete set of totalCount and bounds information must be provided. Any combination of present totalCount totalLBound and totalUBound arguments that provides a complete specification is valid. All input information will be checked for consistency.

The arguments are:

farrayPtr
A Fortran array pointer (associated or unassociated).
[datacopyflag]
Indicate copy vs. reference behavior in case of associated farrayPtr. This argument must not be present for unassociated farrayPtr. Default to ESMF_DATACOPY_REFERENCE, makes the ESMF_LocalArray reference the associated data array. If set to ESMF_DATACOPY_VALUE this routine allocates new memory and copies the data from the pointer into the new LocalArray allocation.
[totalCount]
The number of items in each dimension of the array. This is a 1D integer array the same length as the rank. The count argument may be omitted if both totalLBound and totalUBound arguments are present.
[totalLBound]
An integer array of lower index values. Must be the same length as the rank.
[totalUBound]
An integer array of upper index values. Must be the same length as the rank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.8 ESMF_LocalArrayDestroy - Release resources associated with a LocalArray


INTERFACE:

   subroutine ESMF_LocalArrayDestroy(localarray, rc)
ARGUMENTS:
     type(ESMF_LocalArray), intent(inout) :: localarray
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Destroys an ESMF_LocalArray, releasing all resources associated with the object.

The arguments are:

localarray
Destroy contents of this ESMF_LocalArray.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.9 ESMF_LocalArrayGet - Get object-wide LocalArray information


INTERFACE:

   ! Private name; call using ESMF_LocalArrayGet()
   subroutine ESMF_LocalArrayGetDefault(localarray, &
     typekind, rank, totalCount, totalLBound, totalUBound, isESMFAllocated, rc)
ARGUMENTS:
     type(ESMF_LocalArray), intent(in) :: localarray
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_TypeKind_Flag), intent(out), optional :: typekind
     integer, intent(out), optional :: rank
     integer, intent(out), optional :: totalCount(:)
     integer, intent(out), optional :: totalLBound(:)
     integer, intent(out), optional :: totalUBound(:)
     logical, intent(out), optional :: isESMFAllocated
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Returns information about the ESMF_LocalArray.

The arguments are:

localarray
Queried ESMF_LocalArray object.
[typekind]
TypeKind of the LocalArray object.
[rank]
Rank of the LocalArray object.
[totalCount]
Count per dimension.
[totalLBound]
Lower bound per dimension.
[totalUBound]
Upper bound per dimension.
[isESMFAllocated]
Set to .true. for data allocations held by ESMF. Set to .false. otherwise.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.10 ESMF_LocalArrayGet - Get a Fortran array pointer from a LocalArray


INTERFACE:

   ! Private name; call using ESMF_LocalArrayGet() 
   subroutine ESMF_LocalArrayGetData<rank><type><kind>(localarray, farrayPtr, & 
   datacopyflag, rc)
ARGUMENTS:
   type(ESMF_LocalArray) :: localarray 
   <type> (ESMF_KIND_<kind>), pointer :: farrayPtr 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag 
   integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Return a Fortran pointer to the data buffer, or return a Fortran pointer to a new copy of the data.

The arguments are:

localarray
The ESMF_LocalArray to get the value from.
farrayPtr
An unassociated or associated Fortran pointer correctly allocated.
[datacopyflag]
An optional copy flag which can be specified. Can either make a new copy of the data or reference existing data. See section 54.13 for a list of possible values.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

29.3.11 ESMF_LocalArrayIsCreated - Check whether a LocalArray object has been created


INTERFACE:

   function ESMF_LocalArrayIsCreated(localarray, rc)
RETURN VALUE:
     logical :: ESMF_LocalArrayIsCreated
ARGUMENTS:
     type(ESMF_LocalArray), intent(in) :: localarray
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
DESCRIPTION:

Return .true. if the localarray has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

localarray
ESMF_LocalArray queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

30 ArraySpec Class

30.1 Description

An ArraySpec is a very simple class that contains type, kind, and rank information about an Array. This information is stored in two parameters. TypeKind describes the data type of the elements in the Array and their precision. Rank is the number of dimensions in the Array.

The only methods that are associated with the ArraySpec class are those that allow you to set and retrieve this information.

30.2 Use and Examples

The ArraySpec is passed in as an argument at Field and FieldBundle creation in order to describe an Array that will be allocated or attached at a later time. There are any number of situations in which this approach is useful. One common example is a case in which the user wants to create a very flexible export State with many diagnostic variables predefined, but only a subset desired and consequently allocated for a particular run.

! !PROGRAM: ESMF_ArraySpecEx - ArraySpec manipulation examples
!
! !DESCRIPTION:
!
! This program shows examples of ArraySpec set and get usage
!-----------------------------------------------------------------------------
#include "ESMF.h"

      ! ESMF Framework module
      use ESMF
      use ESMF_TestMod
      implicit none

      ! local variables
      type(ESMF_ArraySpec) :: arrayDS
      integer :: myrank
      type(ESMF_TypeKind_Flag) :: mytypekind


      ! return code
      integer:: rc, result
      character(ESMF_MAXSTR) :: testname
      character(ESMF_MAXSTR) :: failMsg

      ! initialize ESMF framework
      call ESMF_Initialize(defaultlogfilename="ArraySpecEx.Log", &
                    logkindflag=ESMF_LOGKIND_MULTI, rc=rc)

30.2.1 Set ArraySpec values

This example shows how to set values in an ESMF_ArraySpec.

      call ESMF_ArraySpecSet(arrayDS, rank=2, &
                             typekind=ESMF_TYPEKIND_R8, rc=rc)

30.2.2 Get ArraySpec values

This example shows how to query an ESMF_ArraySpec.

      call ESMF_ArraySpecGet(arrayDS, rank=myrank, &
        typekind=mytypekind, rc=rc)
      print *, "Returned values from ArraySpec:"
      print *, "rank =", myrank

      ! finalize ESMF framework
      call ESMF_Finalize(rc=rc)

      end program ESMF_ArraySpecEx

30.3 Restrictions and Future Work

  1. Limit on rank. The values for type, kind and rank passed into the ArraySpec class are subject to the same limitations as Arrays. The maximum array rank is 7, which is the highest rank supported by Fortran.

30.4 Design and Implementation Notes

The information contained in an ESMF_ArraySpec is used to create ESMF_Array objects.

ESMF_ArraySpec is a shallow class, and only set and get methods are needed. They do not need to be created or destroyed.

30.5 Class API

30.5.1 ESMF_ArraySpecAssignment(=) - Assign an ArraySpec to another ArraySpec


INTERFACE:

   interface assignment(=)
     arrayspec1 = arrayspec2
ARGUMENTS:
     type(ESMF_ArraySpec) :: arrayspec1
     type(ESMF_ArraySpec) :: arrayspec2
STATUS:

DESCRIPTION:

Set arrayspec1 equal to arrayspec2. This is the default Fortran assignment, which creates a complete, independent copy of arrayspec2 as arrayspec1. If arrayspec2 is an invalid ESMF_ArraySpec object then arrayspec1 will be equally invalid after the assignment.

The arguments are:

arrayspec1
The ESMF_ArraySpec to be set.
arrayspec2
The ESMF_ArraySpec to be copied.

30.5.2 ESMF_ArraySpecOperator(==) - Test if ArraySpec 1 is equal to ArraySpec 2


INTERFACE:

   interface operator(==)
     if (arrayspec1 == arrayspec2) then ... endif
                  OR
     result = (arrayspec1 == arrayspec2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_ArraySpec), intent(in) :: arrayspec1
     type(ESMF_ArraySpec), intent(in) :: arrayspec2
STATUS:

DESCRIPTION:

Overloads the (==) operator for the ESMF_ArraySpec class to return .true. if arrayspec1 and arrayspec2 specify the same type, kind and rank, and .false. otherwise.

The arguments are:

arrayspec1
First ESMF_ArraySpec in comparison.
arrayspec2
Second ESMF_ArraySpec in comparison.

30.5.3 ESMF_ArraySpecOperator(/=) - Test if ArraySpec 1 is not equal to ArraySpec 2


INTERFACE:

   interface operator(/=)
     if (arrayspec1 /= arrayspec2) then ... endif
                  OR
     result = (arrayspec1 /= arrayspec2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_ArraySpec), intent(in) :: arrayspec1
     type(ESMF_ArraySpec), intent(in) :: arrayspec2
STATUS:

DESCRIPTION:

Overloads the (/=) operator for the ESMF_ArraySpec class to return .true. if arrayspec1 and arrayspec2 do not specify the same type, kind or rank, and .false. otherwise.

The arguments are:

arrayspec1
First ESMF_ArraySpec in comparison.
arrayspec2
Second ESMF_ArraySpec in comparison.

30.5.4 ESMF_ArraySpecGet - Get values from an ArraySpec


INTERFACE:

   subroutine ESMF_ArraySpecGet(arrayspec, rank, typekind, rc)
ARGUMENTS:
     type(ESMF_ArraySpec),     intent(in)            :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                  intent(out), optional :: rank
     type(ESMF_TypeKind_Flag), intent(out), optional :: typekind
     integer,                  intent(out), optional :: rc
STATUS:

DESCRIPTION:

Returns information about the contents of an ESMF_ArraySpec.

The arguments are:

arrayspec
The ESMF_ArraySpec to query.
[rank]
Array rank (dimensionality - 1D, 2D, etc). Maximum possible is 7D.
[typekind]
Array typekind. See section 54.61 for valid values.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

30.5.5 ESMF_ArraySpecPrint - Print ArraySpec information


INTERFACE:

   subroutine ESMF_ArraySpecPrint(arrayspec, rc)
ARGUMENTS:
     type(ESMF_ArraySpec), intent(in)            :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(out), optional :: rc
STATUS:

DESCRIPTION:

Print ArraySpec internals.

The arguments are:

arrayspec
Specified ESMF_ArraySpec object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

30.5.6 ESMF_ArraySpecSet - Set values for an ArraySpec


INTERFACE:

   subroutine ESMF_ArraySpecSet(arrayspec, rank, typekind, rc)
ARGUMENTS:
     type(ESMF_ArraySpec),     intent(out)           :: arrayspec
     integer,                  intent(in)            :: rank
     type(ESMF_TypeKind_Flag), intent(in)            :: typekind
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                  intent(out), optional :: rc
STATUS:

DESCRIPTION:

Creates a description of the data - the typekind, the rank, and the dimensionality.

The arguments are:

arrayspec
The ESMF_ArraySpec to set.
rank
Array rank (dimensionality - 1D, 2D, etc). Maximum allowed is 7D.
typekind
Array typekind. See section 54.61 for valid values.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

30.5.7 ESMF_ArraySpecValidate - Validate ArraySpec internals


INTERFACE:

   subroutine ESMF_ArraySpecValidate(arrayspec, rc)
ARGUMENTS:
     type(ESMF_ArraySpec), intent(in)            :: arrayspec
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(out), optional :: rc
STATUS:

DESCRIPTION:

Validates that the arrayspec is internally consistent. The method returns an error code if problems are found.

The arguments are:

arrayspec
Specified ESMF_ArraySpec object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31 Grid Class

31.1 Description

The ESMF Grid class is used to describe the geometry and discretization of logically rectangular physical grids. It also contains the description of the grid's underlying topology and the decomposition of the physical grid across the available computational resources. The most frequent use of the Grid class is to describe physical grids in user code so that sufficient information is available to perform ESMF methods such as regridding.


Key Features
Representation of grids formed by logically rectangular regions, including uniform and rectilinear grids (e.g. lat-lon grids), curvilinear grids (e.g. displaced pole grids), and grids formed by connected logically rectangular regions (e.g. cubed sphere grids).
Support for 1D, 2D, 3D, and higher dimension grids.
Distribution of grids across computational resources for parallel operations - users set which grid dimensions are distributed.
Grids can be created already distributed, so that no single resource needs global information during the creation process.
Options to define periodicity and other edge connectivities either explicitly or implicitly via shape shortcuts.
Options for users to define grid coordinates themselves or to call prefabricated coordinate generation routines for standard grids.
Options for incremental construction of grids.
Options for using a set of pre-defined stagger locations or for setting custom stagger locations.

31.1.1 Grid Representation in ESMF

ESMF Grids are based on the concepts described in A Standard Description of Grids Used in Earth System Models [Balaji 2006]. In this document Balaji introduces the mosaic concept as a means of describing a wide variety of Earth system model grids. A mosaic is composed of grid tiles connected at their edges. Mosaic grids includes simple, single tile grids as a special case.

The ESMF Grid class is a representation of a mosaic grid. Each ESMF Grid is constructed of one or more logically rectangular Tiles. A Tile will usually have some physical significance (e.g. the region of the world covered by one face of a cubed sphere grid).

The piece of a Tile that resides on one DE (for simple cases, a DE can be thought of as a processor - see section on the DELayout) is called a LocalTile. For example, the six faces of a cubed sphere grid are each Tiles, and each Tile can be divided into many LocalTiles.

Every ESMF Grid contains a DistGrid object, which defines the Grid's index space, topology, distribution, and connectivities. It enables the user to define the complex edge relationships of tripole and other grids. The DistGrid can be created explicitly and passed into a Grid creation routine, or it can be created implicitly if the user takes a Grid creation shortcut. The DistGrid used in Grid creation describes the properties of the Grid cells. In addition to this one, the Grid internally creates DistGrids for each stagger location. These stagger DistGrids are related to the original DistGrid, but may contain extra padding to represent the extent of the index space of the stagger. These DistGrids are what are used when a Field is created on a Grid.

31.1.2 Supported Grids

The range of supported grids in ESMF can be defined by:


31.1.3 Grid Topologies and Periodicity

ESMF has shortcuts for the creation of standard Grid topologies or shapes up to 3D. In many cases, these enable the user to bypass the step of creating a DistGrid before creating the Grid. There are two sets of methods which allow the user to do this. These two sets of methods cover the same set of topologies, but allow the user to specify them in different ways.

The first set of these are a group of overloaded calls broken up by the number of periodic dimensions they specify. With these the user can pick the method which creates a Grid with the number of periodic dimensions they need, and then specify other connectivity options via arguments to the method. The following is a description of these methods:


ESMF_GridCreateNoPeriDim()
Allows the user to create a Grid with no edge connections, for example, a regional Grid with closed boundaries.

ESMF_GridCreate1PeriDim()
Allows the user to create a Grid with 1 periodic dimension and supports a range of options for what to do at the pole (see Section 31.2.5). Some examples of Grids which can be created here are tripole spheres, bipole spheres, cylinders with open poles.

ESMF_GridCreate2PeriDim()
Allows the user to create a Grid with 2 periodic dimensions, for example a torus, or a regional Grid with doubly periodic boundaries.

More detailed information can be found in the API description of each.


The second set of shortcut methods is a set of methods overloaded under the name ESMF_GridCreate(). These methods allow the user to specify the connectivites at the end of each dimension, by using the ESMF_GridConn_Flag flag. The table below shows the ESMF_GridConn_Flag settings used to create standard shapes in 2D using the ESMF_GridCreate() call. Two values are specified for each dimension, one for the low end and one for the high end of the dimension's index values.


2D Shape connflagDim1(1) connflagDim1(2) connflagDim2(1) connflagDim2(2)
Rectangle NONE NONE NONE NONE
Bipole Sphere POLE POLE PERIODIC PERIODIC
Tripole Sphere POLE BIPOLE PERIODIC PERIODIC
Cylinder NONE NONE PERIODIC PERIODIC
Torus PERIODIC PERIODIC PERIODIC PERIODIC


If the user's grid shape is too complex for an ESMF shortcut routine, or involves more than three dimensions, a DistGrid can be created to specify the shape in detail. This DistGrid is then passed into a Grid create call.


31.1.4 Grid Distribution

ESMF Grids have several options for data distribution (also referred to as decomposition). As ESMF Grids are cell based, these options are all specified in terms of how the cells in the Grid are broken up between DEs.

The main distribution options are regular, irregular, and arbitrary. A regular distribution is one in which the same number of contiguous grid cells are assigned to each DE in the distributed dimension. An irregular distribution is one in which unequal numbers of contiguous grid cells are assigned to each DE in the distributed dimension. An arbitrary distribution is one in which any grid cell can be assigned to any DE. Any of these distribution options can be applied to any of the grid shapes (i.e., rectangle) or types (i.e., rectilinear). Support for arbitrary distribution is limited in the current version of ESMF, see Section 31.3.7 for an example of creating a Grid with an arbitrary distribution.

Figure 13 illustrates options for distribution.

Figure 13: Examples of regular and irregular decomposition of a grid a that is 6x6, and an arbitrary decomposition of a grid b that is 6x3.
\scalebox{0.9}{\includegraphics{GridDecomps}}

A distribution can also be specified using the DistGrid, by passing object into a Grid create call.


31.1.5 Grid Coordinates

Grid Tiles can have uniform, rectilinear, or curvilinear coordinates. The coordinates of uniform grids are equally spaced along their axes, and can be fully specified by the coordinates of the two opposing points that define the grid's physical span. The coordinates of rectilinear grids are unequally spaced along their axes, and can be fully specified by giving the spacing of grid points along each axis. The coordinates of curvilinear grids must be specified by giving the explicit set of coordinates for each grid point. Curvilinear grids are often uniform or rectilinear grids that have been warped; for example, to place a pole over a land mass so that it does not affect the computations performed on an ocean model grid. Figure 14 shows examples of each type of grid.

Figure 14: Types of logically rectangular grid tiles. Red circles show the values needed to specify grid coordinates for each type.
\scalebox{0.9}{\includegraphics{LogRectGrids}}

Each of these coordinate types can be set for each of the standard grid shapes described in section 31.1.3.

The table below shows how examples of common single Tile grids fall into this shape and coordinate taxonomy. Note that any of the grids in the table can have a regular or arbitrary distribution.


  Uniform Rectilinear Curvilinear
Sphere Global uniform lat-lon grid Gaussian grid Displaced pole grid
Rectangle Regional uniform lat-lon grid Gaussian grid section Polar stereographic grid section

31.1.6 Coordinate Specification and Generation

There are two ways of specifying coordinates in ESMF. The first way is for the user to set the coordinates. The second way is to take a shortcut and have the framework generate the coordinates.

See Section 31.3.13 for more description and examples of setting coordinates.

31.1.7 Staggering

Staggering is a finite difference technique in which the values of different physical quantities are placed at different locations within a grid cell.

The ESMF Grid class supports a variety of stagger locations, including cell centers, corners, and edge centers. The default stagger location in ESMF is the cell center, and cell counts in Grid are based on this assumption. Combinations of the 2D ESMF stagger locations are sufficient to specify any of the Arakawa staggers. ESMF also supports staggering in 3D and higher dimensions. There are shortcuts for standard staggers, and interfaces through which users can create custom staggers.

As a default the ESMF Grid class provides symmetric staggering, so that cell centers are enclosed by cell perimeter (e.g. corner) stagger locations. This means the coordinate arrays for stagger locations other than the center will have an additional element of padding in order to enclose the cell center locations. However, to achieve other types of staggering, the user may alter or eliminate this padding by using the appropriate options when adding coordinates to a Grid.

In the current release, only the cell center stagger location is supported for an arbitrarily distributed grid. For examples and a full description of the stagger interface see Section 31.3.13.

31.1.8 Masking

Masking is the process whereby parts of a Grid can be marked to be ignored during an operation. For a description of how to set mask information in the Grid, see here 31.3.17. For a description of how masking works in regridding, see here 24.2.10.

31.2 Constants


31.2.1 ESMF_GRIDCONN

DESCRIPTION:
The ESMF_GridCreateShapeTile command has three specific arguments connflagDim1, connflagDim2, and connflagDim3. These can be used to setup different types of connections at the ends of each dimension of a Tile. Each of these parameters is a two element array. The first element is the connection type at the minimum end of the dimension and the second is the connection type at the maximum end. The default value for all the connections is ESMF_GRIDCONN_NONE, specifying no connection.

The type of this flag is:

type(ESMF_GridConn_Flag)

The valid values are:

ESMF_GRIDCONN_NONE
No connection.

ESMF_GRIDCONN_PERIODIC
Periodic connection.

ESMF_GRIDCONN_POLE
This edge is connected to itself. Given that the edge is n elements long, then element i is connected to element ((i+n/2) mod n).

ESMF_GRIDCONN_BIPOLE
This edge is connected to itself. Given that the edge is n elements long, element i is connected to element n-i+1.


31.2.2 ESMF_GRIDITEM

DESCRIPTION:
The ESMF Grid can contain other kinds of data besides coordinates. This data is referred to as Grid “items”. Some items may be used by ESMF for calculations involving the Grid. The following are the valid values of ESMF_GridItem_Flag.

The type of this flag is:

type(ESMF_GridItem_Flag)

The valid values are:
Item Label Type Restriction Type Default ESMF Uses Controls
ESMF_GRIDITEM_MASK ESMF_TYPEKIND_I4 ESMF_TYPEKIND_I4 YES Masking in Regrid
ESMF_GRIDITEM_AREA NONE ESMF_TYPEKIND_R8 YES Conservation in Regrid


NOTE: One important thing to consider when setting areas in the Grid using ESMF_GRIDITEM_AREA, ESMF doesn't currently do unit conversion on areas. If these areas are going to be used in a process that also involves the areas of another Grid or Mesh (e.g. conservative regridding), then it is the user's responsibility to make sure that the area units are consistent between the two sides. If ESMF calculates an area on the surface of a sphere, then it is in units of square radians. If it calculates the area for a Cartesian grid, then it is in the same units as the coordinates, but squared.


31.2.3 ESMF_GRIDMATCH

DESCRIPTION:
This type is used to indicate the level to which two grids match.

The type of this flag is:

type(ESMF_GridMatch_Flag)

The valid values are:

ESMF_GRIDMATCH_INVALID:
Indicates a non-valid matching level. Returned if an error occurs in the matching function. If a higher matching level is returned then no error occurred.
ESMF_GRIDMATCH_NONE:
The lowest level of grid matching. This indicates that the Grid's don't match at any of the higher levels.
ESMF_GRIDMATCH_EXACT:
All the pieces of the Grid (e.g. distgrids, coordinates, etc.) except the name, match between the two Grids.
ESMF_GRIDMATCH_ALIAS:
Both Grid variables are aliases to the exact same Grid object in memory.


31.2.4 ESMF_GRIDSTATUS

DESCRIPTION:
The ESMF Grid class can exist in two states. These states are present so that the library code can detect if a Grid has been appropriately setup for the task at hand. The following are the valid values of ESMF_GRIDSTATUS.

The type of this flag is:

type(ESMF_GridStatus_Flag)

The valid values are:

ESMF_GRIDSTATUS_EMPTY:
Status after a Grid has been created with ESMF_GridEmptyCreate. A Grid object container is allocated but space for internal objects is not. Topology information and coordinate information is incomplete. This object can be used in ESMF_GridEmptyComplete() methods in which additional information is added to the Grid.
ESMF_GRIDSTATUS_COMPLETE:
The Grid has a specific topology and distribution, but incomplete coordinate arrays. The Grid can be used as the basis for allocating a Field, and coordinates can be added via ESMF_GridCoordAdd() to allow other functionality.


31.2.5 ESMF_POLEKIND

DESCRIPTION:
This type describes the type of connection that occurs at the pole when a Grid is created with ESMF_GridCreate1PeriodicDim().

The type of this flag is:

type(ESMF_PoleKind_Flag)

The valid values are:

ESMF_POLEKIND_NONE
No connection at pole.

ESMF_POLEKIND_MONOPOLE
This edge is connected to itself. Given that the edge is n elements long, then element i is connected to element i+n/2.

ESMF_POLEKIND_BIPOLE
This edge is connected to itself. Given that the edge is n elements long, element i is connected to element n-i+1.


31.2.6 ESMF_STAGGERLOC

DESCRIPTION:
In the ESMF Grid class, data can be located at different positions in a Grid cell. When setting or retrieving coordinate data the stagger location is specified to tell the Grid method from where in the cell to get the data. Although the user may define their own custom stagger locations, ESMF provides a set of predefined locations for ease of use. The following are the valid predefined stagger locations.


Figure 15: 2D Predefined Stagger Locations
\scalebox{0.75}{\includegraphics{GridStaggerLoc2D}}

The 2D predefined stagger locations (illustrated in figure 15) are:

ESMF_STAGGERLOC_CENTER:
The center of the cell.
ESMF_STAGGERLOC_CORNER:
The corners of the cell.
ESMF_STAGGERLOC_EDGE1:
The edges offset from the center in the 1st dimension.
ESMF_STAGGERLOC_EDGE2:
The edges offset from the center in the 2nd dimension.


Figure 16: 3D Predefined Stagger Locations
\scalebox{1.0}{\includegraphics{GridStaggerLoc3D}}

The 3D predefined stagger locations (illustrated in figure 16) are:

ESMF_STAGGERLOC_CENTER_VCENTER:
The center of the 3D cell.
ESMF_STAGGERLOC_CORNER_VCENTER:
Half way up the vertical edges of the cell.
ESMF_STAGGERLOC_EDGE1_VCENTER:
The center of the face bounded by edge 1 and the vertical dimension.
ESMF_STAGGERLOC_EDGE2_VCENTER:
The center of the face bounded by edge 2 and the vertical dimension.
ESMF_STAGGERLOC_CORNER_VFACE:
The corners of the 3D cell.
ESMF_STAGGERLOC_EDGE1_VFACE:
The center of the edges of the 3D cell parallel offset from the center in the 1st dimension.
ESMF_STAGGERLOC_EDGE2_VFACE:
The center of the edges of the 3D cell parallel offset from the center in the 2nd dimension.
ESMF_STAGGERLOC_CENTER_VFACE:
The center of the top and bottom face. The face bounded by the 1st and 2nd dimensions.

31.3 Use and Examples

This section describes the use of the ESMF Grid class. It first discusses the more user friendly shape specific interface to the Grid. During this discussion it covers creation and options, adding stagger locations, coordinate data access, and other grid functionality. After this initial phase the document discusses the more advanced options which the user can employ should they need more customized interaction with the Grid class.

31.3.1 Create single-tile Grid shortcut method

The set of methods ESMF_GridCreateNoPeriDim(), ESMF_GridCreate1PeriDim(), ESMF_GridCreate2PeriDim(), and ESMF_GridCreate() are shortcuts for building 2D or 3D single tile logically rectangular Grids. These methods support all three types of distributions described in Section 31.1.4: regular, irregular and arbitrary.

The ESMF Grid is cell based and so for all distribution options the methods take as input the number of cells to describe the total index space and the number of cells to specify distribution.

To create a Grid with a regular distribution the user specifies the global maximum and minimum ranges of the Grid cell index space (maxIndex and minIndex), and the number of pieces in which to partition each dimension (via a regDecomp argument). ESMF then divides the index space as evenly as possible into the specified number of pieces. If there are cells left over then they are distributed one per DE starting from the first DE until they are gone.

If minIndex is not specified, then the bottom of the Grid cell index range is assumed to be (1,1,...,1). If regDecomp is not specified, then by default ESMF creates a distribution that partitions the grid cells in the first dimension (e.g. NPx1x1...1) as evenly as possible by the number of PETs NP. The remaining dimensions are not partitioned. The dimension of the Grid is the size of maxIndex. The following is an example of creating a 10x20x30 3D grid where the first dimensions is broken into 2 pieces, the second is broken into 4 pieces, and the third is not divided (i.e. every DE will have length 30 in the 3rd dimension).

  grid3D=ESMF_GridCreateNoPeriDim(regDecomp=(/2,4,1/), maxIndex=(/10,20,30/), &
           rc=rc)

Irregular distribution requires the user to specify the exact number of Grid cells per DE in each dimension. In the ESMF_GridCreateNoPeriDim() call the countsPerDEDim1, countsPerDim2, and countsPerDim3 arguments are used to specify a rectangular distribution containing size(countsPerDEDim1) by size(countsPerDEDim2) by size(countsPerDEDim3) DEs. The entries in each of these arrays specify the number of grid cells per DE in that dimension. The dimension of the grid is determined by the presence of countsPerDEDim3. If it's present the Grid will be 3D. If just countsPerDEDim1 and countsPerDEDim2 are specified the Grid will be 2D.

The following call illustrates the creation of a 10x20 two dimensional rectangular Grid distributed across six DEs that are arranged 2x3. In the first dimension there are 3 grid cells on the first DE and 7 cells on the second DE. The second dimension has 3 DEs with 11,2, and 7 cells, respectively.

   grid2D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/3,7/), &
          countsPerDEDim2=(/11,2,7/), rc=rc)

To add a distributed third dimension of size 30, broken up into two groups of 15, the above call would be altered as follows.

   grid3d=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/3,7/), &
          countsPerDEDim2=(/11,2,7/), countsPerDEDim3=(/15,15/), rc=rc)

To make a third dimension distributed across only 1 DE, then countsPerDEDim3 in the call should only have a single term.

   grid3D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/3,7/),  &
          countsPerDEDim2=(/11,2,7/), countsPerDEDim3=(/30/), rc=rc)

The petMap parameter may be used to specify on to which specific PETs the DEs in the Grid are assigned. Each entry in petMap specifies to which PET the corresponding DE should be assigned. For example, petMap(3,2)=4 tells the Grid create call to put the DE located at column 3 row 2 on PET 4. Note that this parameter is only available for the regular and irregular distribution types. The petMap array is a 3D array, for a 3D Grid each of its dimensions correspond to a Grid dimension. If the Grid is 2D, then the first two dimensions correspond to Grid dimensions and the last dimension should be of size 1. The size of each petMap dimension is the number of DE's along that dimension in the Grid. For a regular Grid, the size is equal to the number in regDecomp (i.e. size(petMap,d)=regDecomp(d) for all dimensions d in the Grid). For an irregular Grid the size is equal to the number of items in the corresponding countsPerDEDim variable (i.e. size(petMap,d)=size(countsPerDEDimd) for all dimensions d in the Grid). The following example demonstrates how to specify the PET to DE association for an ESMF_GridCreateNoPeriDim() call.

   ! allocate memory for petMap
   allocate( petMap(2,2,1) )

   ! Set petMap
   petMap(:,1,1) = (/3,2/) ! DE (1,1,1) on PET 3 and DE (2,1,1) on PET 2
   petMap(:,2,1) = (/1,0/) ! DE (1,2,1) on PET 1 and DE (2,2,1) on PET 0


   ! Let the 3D grid be be distributed only in the first two dimensions.
   grid2D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/3,7/), &
           countsPerDEDim2=(/7,6/), petMap=petMap, rc=rc)

To create an grid with arbitrary distribution, the user specifies the global minimum and maximum ranges of the index space with the arguments minIndex and maxIndex, the total number of cells and their index space locations residing on the local PET through a localArbIndexCount and a localArbIndex argument. localArbIndex is a 2D array with size (localArbIndexCount, n) where n is the total number dimensions distributed arbitrarily. Again, if minIndex is not specified, then the bottom of the index range is assumed to be (1,1,...). The dimension of the Grid is equal to the size of maxIndex. If n (number of arbitrarily distributed dimension) is less than the grid dimension, an optional argument distDim is used to specify which of the grid dimension is arbitrarily distributed. If not given, the first n dimensions are assumed to be distributed.

The following example creates a 2D Grid of dimensions 5x5, and places the diagonal elements (i.e. indices (i,i) where i goes from 1 to 5) on the local PET. The remaining PETs would individually declare the remainder of the Grid locations.

   ! allocate memory for localArbIndex
   allocate( localArbIndex(5,2) )
   ! Set local indices
   localArbIndex(1,:)=(/1,1/)
   localArbIndex(2,:)=(/2,2/)
   localArbIndex(3,:)=(/3,3/)
   localArbIndex(4,:)=(/4,4/)
   localArbIndex(5,:)=(/5,5/)

   ! Create a 2D Arbitrarily distributed Grid
   grid2D=ESMF_GridCreateNoPeriDim(maxIndex=(/5,5/), &
         arbIndexList=localArbIndex, arbIndexCount=5, rc=rc)

To create a 3D Grid of dimensions 5x6x5 with the first and the third dimensions distributed arbitrarily, distDim is used.

   ! Create a 3D Grid with the 1st and 3rd dimension arbitrarily distributed
   grid3D=ESMF_GridCreateNoPeriDim(maxIndex=(/5,6,5/), &
         arbIndexList=localArbIndex, arbIndexCount=5, &
         distDim=(/1,3/), rc=rc)


31.3.2 Create a 2D regularly distributed rectilinear Grid with uniformly spaced coordinates

The following is an example of creating a simple rectilinear grid and loading in a set of coordinates. It illustrates a straightforward use of the ESMF_GridCreateNoPeriDim() call described in the previous section. This code creates a 10x20 2D grid with uniformly spaced coordinates varying from (10,10) to (100,200). The grid is partitioned using a regular distribution. The first dimension is divided into two pieces, and the second dimension is divided into 3. This example assumes that the code is being run with a 1-1 mapping between PETs and DEs because we are only accessing the first DE on each PET (localDE=0). Because we have 6 DEs (2x3), this example would only work when run on 6 PETs. The Grid is created with global indices. After Grid creation the local bounds and native Fortran arrays are retrieved and the coordinates are set by the user.

   !-------------------------------------------------------------------
   ! Create the Grid:  Allocate space for the Grid object, define the
   ! topology and distribution of the Grid, and specify that it
   ! will have global indices.  Note that here aperiodic bounds are
   ! specified by the argument name. In this call the minIndex hasn't
   ! been set, so it defaults to (1,1,...). The default is to
   ! divide the index range as equally as possible among the DEs
   ! specified in regDecomp. This behavior can be changed by
   ! specifying decompFlag.
   !-------------------------------------------------------------------
   grid2D=ESMF_GridCreateNoPeriDim(          &
         ! Define a regular distribution
         maxIndex=(/10,20/), & ! define index space
         regDecomp=(/2,3/),  & ! define how to divide among DEs
         coordSys=ESMF_COORDSYS_CART, &
         ! Specify mapping of coords dim to Grid dim
         coordDep1=(/1/), & ! 1st coord is 1D and depends on 1st Grid dim
         coordDep2=(/2/), & ! 2nd coord is 1D and depends on 2nd Grid dim
         indexflag=ESMF_INDEX_GLOBAL, &
         rc=rc)

   !-------------------------------------------------------------------
   ! Allocate coordinate storage and associate it with the center
   ! stagger location.  Since no coordinate values are specified in
   ! this call no coordinate values are set yet.
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid2D,  &
          staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)

   !-------------------------------------------------------------------
   ! Get the pointer to the first coordinate array and the bounds
   ! of its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=1, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordX, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the first dimension [10-100].
   !-------------------------------------------------------------------
   do i=lbnd(1),ubnd(1)
        coordX(i) = i*10.0
   enddo

   !-------------------------------------------------------------------
   ! Get the pointer to the second coordinate array and the bounds of
   ! its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=2, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordY, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the second dimension [10-200]
   !-------------------------------------------------------------------
   do j=lbnd(1),ubnd(1)
        coordY(j) = j*10.0
   enddo


31.3.3 Create a periodic 2D regularly distributed rectilinear Grid

The following is an example of creating a simple rectilinear grid with a periodic dimension and loading in a set of coordinates. It illustrates a straightforward use of the ESMF_GridCreate1PeriDim() call described in the previous section. This code creates a 360x180 2D grid with uniformly spaced coordinates varying from (1,1) to (360,180). The grid is partitioned using a regular distribution. The first dimension is divided into two pieces, and the second dimension is divided into 3. This example assumes that the code is being run with a 1-1 mapping between PETs and DEs because we are only accessing the first DE on each PET (localDE=0). Because we have 6 DEs (2x3), this example would only work when run on 6 PETs. The Grid is created with global indices. After Grid creation the local bounds and native Fortran arrays are retrieved and the coordinates are set by the user.

   !-------------------------------------------------------------------
   ! Create the Grid:  Allocate space for the Grid object, define the
   ! topology and distribution of the Grid, and specify that it
   ! will have global indices.  Note that here a single periodic connection
   ! is specified by the argument name. In this call the minIndex hasn't
   ! been set, so it defaults to (1,1,...). The default is to
   ! divide the index range as equally as possible among the DEs
   ! specified in regDecomp. This behavior can be changed by
   ! specifying decompFlag. Since the coordinate system is
   ! not specified, it defaults to ESMF_COORDSYS_SPH_DEG.
   !-------------------------------------------------------------------
   grid2D=ESMF_GridCreate1PeriDim(          &
         ! Define a regular distribution
         maxIndex=(/360,180/), & ! define index space
         regDecomp=(/2,3/),  & ! define how to divide among DEs
         ! Specify mapping of coords dim to Grid dim
         coordDep1=(/1/), & ! 1st coord is 1D and depends on 1st Grid dim
         coordDep2=(/2/), & ! 2nd coord is 1D and depends on 2nd Grid dim
         indexflag=ESMF_INDEX_GLOBAL, &
         rc=rc)

   !-------------------------------------------------------------------
   ! Allocate coordinate storage and associate it with the center
   ! stagger location.  Since no coordinate values are specified in
   ! this call no coordinate values are set yet.
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid2D,  &
          staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)

   !-------------------------------------------------------------------
   ! Get the pointer to the first coordinate array and the bounds
   ! of its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=1, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordX, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the first dimension [10-100].
   !-------------------------------------------------------------------
   do i=lbnd(1),ubnd(1)
        coordX(i) = i*1.0
   enddo

   !-------------------------------------------------------------------
   ! Get the pointer to the second coordinate array and the bounds of
   ! its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=2, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordY, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the second dimension [10-200]
   !-------------------------------------------------------------------
   do j=lbnd(1),ubnd(1)
        coordY(j) = j*1.0
   enddo

The remaining examples in this section will use the irregular distribution because of its greater generality. To create code similar to these, but using a regular distribution, replace the countsPerDEDim arguments in the Grid create with the appropriate maxIndex and regDecomp arguments.


31.3.4 Create a 2D irregularly distributed rectilinear Grid with uniformly spaced coordinates

This example serves as an illustration of the difference between using a regular and irregular distribution. It repeats the previous example except using an irregular distribution to give the user more control over how the cells are divided between the DEs. As before, this code creates a 10x20 2D Grid with uniformly spaced coordinates varying from (10,10) to (100,200). In this example, the Grid is partitioned using an irregular distribution. The first dimension is divided into two pieces, the first with 3 Grid cells per DE and the second with 7 Grid cells per DE. In the second dimension, the Grid is divided into 3 pieces, with 11, 2, and 7 cells per DE respectively. This example assumes that the code is being run with a 1-1 mapping between PETs and DEs because we are only accessing the first DE on each PET (localDE=0). Because we have 6 DEs (2x3), this example would only work when run on 6 PETs. The Grid is created with global indices. After Grid creation the local bounds and native Fortran arrays are retrieved and the coordinates are set by the user.

   !-------------------------------------------------------------------
   ! Create the Grid:  Allocate space for the Grid object, define the
   ! topology and distribution of the Grid, and specify that it
   ! will have global coordinates.  Note that aperiodic bounds are
   ! indicated by the method name. In this call the minIndex hasn't
   ! been set, so it defaults to (1,1,...).
   !-------------------------------------------------------------------
   grid2D=ESMF_GridCreateNoPeriDim(          &
            ! Define an irregular distribution
            countsPerDEDim1=(/3,7/),    &
            countsPerDEDim2=(/11,2,7/), &
            ! Specify mapping of coords dim to Grid dim
            coordDep1=(/1/), & ! 1st coord is 1D and depends on 1st Grid dim
            coordDep2=(/2/), & ! 2nd coord is 1D and depends on 2nd Grid dim
            indexflag=ESMF_INDEX_GLOBAL, &
            rc=rc)

   !-------------------------------------------------------------------
   ! Allocate coordinate storage and associate it with the center
   ! stagger location.  Since no coordinate values are specified in
   ! this call no coordinate values are set yet.
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid2D,  &
          staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)

   !-------------------------------------------------------------------
   ! Get the pointer to the first coordinate array and the bounds
   ! of its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=1, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordX, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the first dimension [10-100].
   !-------------------------------------------------------------------
   do i=lbnd(1),ubnd(1)
        coordX(i) = i*10.0
   enddo

   !-------------------------------------------------------------------
   ! Get the pointer to the second coordinate array and the bounds of
   ! its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=2, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordY, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the second dimension [10-200]
   !-------------------------------------------------------------------
   do j=lbnd(1),ubnd(1)
        coordY(j) = j*10.0
   enddo


31.3.5 Create a 2D irregularly distributed Grid with curvilinear coordinates

The following is an example of creating a simple curvilinear Grid and loading in a set of coordinates. It creates a 10x20 2D Grid where the coordinates vary along every dimension. The Grid is partitioned using an irregular distribution. The first dimension is divided into two pieces, the first with 3 Grid cells per DE and the second with 7 Grid cells per DE. In the second dimension, the Grid is divided into 3 pieces, with 11, 2, and 7 cells per DE respectively. This example assumes that the code is being run with a 1-1 mapping between PETs and DEs because we are only accessing the first DE on each PET (localDE=0). Because we have 6 DEs (2x3), this example would only work when run on 6 PETs. The Grid is created with global indices. After Grid creation the local bounds and native Fortran arrays are retrieved and the coordinates are set by the user.

   !-------------------------------------------------------------------
   ! Create the Grid:  Allocate space for the Grid object, define the
   ! distribution of the Grid, and specify that it
   ! will have global indices.  Note that aperiodic bounds are
   ! indicated by the method name. If periodic bounds were desired they
   ! could be specified by using the ESMF_GridCreate1PeriDim() call.
   ! In this call the minIndex hasn't been set, so it defaults to (1,1,...).
   !-------------------------------------------------------------------
   grid2D=ESMF_GridCreateNoPeriDim(      &
        ! Define an irregular distribution
        countsPerDEDim1=(/3,7/),     &
        countsPerDEDim2=(/11,2,7/),   &
        ! Specify mapping of coords dim to Grid dim
        coordDep1=(/1,2/), & ! 1st coord is 2D and depends on both Grid dim
        coordDep2=(/1,2/), & ! 2nd coord is 2D and depends on both Grid dim
        indexflag=ESMF_INDEX_GLOBAL, &
        rc=rc)

   !-------------------------------------------------------------------
   ! Allocate coordinate storage and associate it with the center
   ! stagger location.  Since no coordinate values are specified in
   ! this call no coordinate values are set yet.
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid2D,  &
          staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)

   !-------------------------------------------------------------------
   ! Get the pointer to the first coordinate array and the bounds
   ! of its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=1, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordX2D, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the first dimension [10-100].
   !-------------------------------------------------------------------
   do j=lbnd(2),ubnd(2)
   do i=lbnd(1),ubnd(1)
        coordX2D(i,j) = i+j
   enddo
   enddo

   !-------------------------------------------------------------------
   ! Get the pointer to the second coordinate array and the bounds of
   ! its global indices on the local DE.
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid2D, coordDim=2, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CENTER, &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=coordY2D, rc=rc)

   !-------------------------------------------------------------------
   ! Calculate and set coordinates in the second dimension [10-200]
   !-------------------------------------------------------------------
   do j=lbnd(2),ubnd(2)
   do i=lbnd(1),ubnd(1)
        coordY2D(i,j) = j-i/100.0
   enddo
   enddo


31.3.6 Create an irregularly distributed rectilinear Grid with a non-distributed vertical dimension

This example demonstrates how a user can build a rectilinear horizontal Grid with a non-distributed vertical dimension. The Grid contains both the center and corner stagger locations (i.e. Arakawa B-Grid). In contrast to the previous examples, this example doesn't assume that the code is being run with a 1-1 mapping between PETs and DEs. It should work when run on any number of PETs.

   !-------------------------------------------------------------------
   ! Create the Grid:  Allocate space for the Grid object.  The
   ! Grid is defined to be 180 Grid cells in the first dimension
   ! (e.g. longitude), 90 Grid cells in the second dimension
   ! (e.g. latitude), and 40 Grid cells in the third dimension
   ! (e.g. height).  The first dimension is decomposed over 4 DEs,
   ! the second over 3 DEs, and the third is not distributed.
   ! The connectivities in each dimension are set to aperiodic
   ! by this method. In this call the minIndex hasn't been set,
   ! so it defaults to (1,1,...).
   !-------------------------------------------------------------------
   grid3D=ESMF_GridCreateNoPeriDim( &
            ! Define an irregular distribution
            countsPerDEDim1=(/45,75,40,20/), &
            countsPerDEDim2=(/30,40,20/),    &
            countsPerDEDim3=(/40/),          &
            ! Specify mapping of coords dim to Grid dim
            coordDep1=(/1/), & ! 1st coord is 1D and depends on 1st Grid dim
            coordDep2=(/2/), & ! 2nd coord is 1D and depends on 2nd Grid dim
            coordDep3=(/3/), & ! 3rd coord is 1D and depends on 3rd Grid dim
            indexflag=ESMF_INDEX_GLOBAL,     & ! Use global indices
            rc=rc)

        

   !-------------------------------------------------------------------
   ! Allocate coordinate storage for both center and corner stagger
   ! locations.  Since no coordinate values are specified in this
   ! call no coordinate values are set yet.
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid3D, &
          staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER, rc=rc)

   call ESMF_GridAddCoord(grid3D, &
          staggerloc=ESMF_STAGGERLOC_CORNER_VCENTER, rc=rc)

   !-------------------------------------------------------------------
   ! Get the number of DEs on this PET, so that the program
   ! can loop over them when accessing data.
   !-------------------------------------------------------------------
   call ESMF_GridGet(grid3D, localDECount=localDECount, rc=rc)

   !-------------------------------------------------------------------
   ! Loop over each localDE when accessing data
   !-------------------------------------------------------------------
   do lDE=0,localDECount-1

    !------------------------------------------------------------------
    ! Fill in the coordinates for the corner stagger location first.
    !------------------------------------------------------------------
      !----------------------------------------------------------------
      ! Get the local bounds of the global indexing for the first
      ! coordinate array on the local DE. If the number of PETs
      ! is less than the total number of DEs then the rest of this
      ! example would be in a loop over the local DEs.  Also get the
      ! pointer to the first coordinate array.
      !----------------------------------------------------------------
      call ESMF_GridGetCoord(grid3D, coordDim=1, localDE=lDE, &
             staggerLoc=ESMF_STAGGERLOC_CORNER_VCENTER,       &
             computationalLBound=lbnd_corner,                 &
             computationalUBound=ubnd_corner,                 &
             farrayPtr=cornerX, rc=rc)

      !----------------------------------------------------------------
      ! Calculate and set coordinates in the first dimension.
      !----------------------------------------------------------------
      do i=lbnd_corner(1),ubnd_corner(1)
         cornerX(i) = (i-1)*(360.0/180.0)
      enddo

      !----------------------------------------------------------------
      ! Get the local bounds of the global indexing for the second
      ! coordinate array on the local DE.  Also get the pointer to the
      ! second coordinate array.
      !----------------------------------------------------------------
      call ESMF_GridGetCoord(grid3D, coordDim=2, localDE=lDE,   &
             staggerLoc=ESMF_STAGGERLOC_CORNER_VCENTER,         &
             computationalLBound=lbnd_corner,                   &
             computationalUBound=ubnd_corner,                   &
             farrayPtr=cornerY, rc=rc)

      !----------------------------------------------------------------
      ! Calculate and set coordinates in the second dimension.
      !----------------------------------------------------------------
      do j=lbnd_corner(1),ubnd_corner(1)
         cornerY(j) = (j-1)*(180.0/90.0)
      enddo

      !----------------------------------------------------------------
      ! Get the local bounds of the global indexing for the third
      ! coordinate array on the local DE, and the pointer to the array.
      !----------------------------------------------------------------
      call ESMF_GridGetCoord(grid3D, coordDim=3, localDE=lDE,   &
             staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER,         &
             computationalLBound=lbnd, computationalUBound=ubnd,&
             farrayPtr=cornerZ, rc=rc)

      !----------------------------------------------------------------
      ! Calculate and set the vertical coordinates
      !----------------------------------------------------------------
      do k=lbnd(1),ubnd(1)
         cornerZ(k) = 4000.0*( (1./39.)*(k-1)  )**2
      enddo

    !------------------------------------------------------------------
    ! Now fill the coordinates for the center stagger location with
    ! the average of the corner coordinate location values.
    !------------------------------------------------------------------
      !----------------------------------------------------------------
      ! Get the local bounds of the global indexing for the first
      ! coordinate array on the local DE, and the pointer to the array.
      !----------------------------------------------------------------
      call ESMF_GridGetCoord(grid3D, coordDim=1, localDE=lDE,    &
             staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER,          &
             computationalLBound=lbnd, computationalUBound=ubnd, &
             farrayPtr=centerX, rc=rc)

      !----------------------------------------------------------------
      ! Calculate and set coordinates in the first dimension.
      !----------------------------------------------------------------
      do i=lbnd(1),ubnd(1)
         centerX(i) = 0.5*(i-1 + i)*(360.0/180.0)
      enddo

      !----------------------------------------------------------------
      ! Get the local bounds of the global indexing for the second
      ! coordinate array on the local DE, and the pointer to the array.
      !----------------------------------------------------------------
       call ESMF_GridGetCoord(grid3D, coordDim=2, localDE=lDE,    &
              staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER,          &
              computationalLBound=lbnd, computationalUBound=ubnd, &
              farrayPtr=centerY, rc=rc)

      !----------------------------------------------------------------
      ! Calculate and set coordinates in the second dimension.
      !----------------------------------------------------------------
      do j=lbnd(1),ubnd(1)
         centerY(j) = 0.5*(j-1 + j)*(180.0/90.0)
      enddo

      !----------------------------------------------------------------
      ! Get the local bounds of the global indexing for the third
      ! coordinate array on the local DE, and the pointer to the array.
      !----------------------------------------------------------------
      call ESMF_GridGetCoord(grid3D, coordDim=3, localDE=lDE,   &
             staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER,         &
             computationalLBound=lbnd, computationalUBound=ubnd,&
             farrayPtr=centerZ, rc=rc)

      !----------------------------------------------------------------
      ! Calculate and set the vertical coordinates
      !----------------------------------------------------------------
      do k=lbnd(1),ubnd(1)
         centerZ(k) = 4000.0*( (1./39.)*(k-1)  )**2
      enddo

   !-------------------------------------------------------------------
   ! End of loop over DEs
   !-------------------------------------------------------------------
   enddo


31.3.7 Create an arbitrarily distributed rectilinear Grid with a non-distributed vertical dimension

There are more restrictions in defining an arbitrarily distributed grid. First, there is always one DE per PET. Secondly, only local index (ESMF_INDEX_LOCAL) is supported. Third, only one stagger location, i.e. ESMF_STAGGERLOC_CENTER is allowed and last there is no extra paddings on the edge of the grid.

This example demonstrates how a user can build a 3D grid with its rectilinear horizontal Grid distributed arbitrarily and a non-distributed vertical dimension.

   !-------------------------------------------------------------------
   ! Set up the local index array:  Assuming the grid is 360x180x10.  First
   ! calculate the localArbIndexCount and localArbIndex array for each PET
   ! based on the total number of PETs. The cells are evenly distributed in
   ! all the PETs. If the total number of cells are not divisible by the
   ! total PETs, the remaining cells are assigned to the last PET.  The
   ! cells are card dealt to each PET in y dimension first,
   ! i.e. (1,1) -> PET 0, (1,2)->  PET 1, (1,3)-> PET 2, and so forth.
   !-------------------------------------------------------------------
   xdim = 360
   ydim = 180
   zdim = 10
   localArbIndexCount = (xdim*ydim)/petCount
   remain = (xdim*ydim)-localArbIndexCount*petCount
   if (localPet == petCount-1) localArbIndexCount = localArbIndexCount+remain

   allocate(localArbIndex(localArbIndexCount,2))
   ind = localPet
   do i=1, localArbIndexCount
      localArbIndex(i,1)=mod(ind,ydim)+1
      localArbIndex(i,2)=ind/ydim + 1
      ind = ind + petCount
   enddo
   if (localPet == petCount-1) then
      ind = xdim*ydim-remain+1
      do i=localArbIndexCount-remain+1,localArbIndexCount
         localArbIndex(i,1)=mod(ind,ydim)+1
         localArbIndex(i,2)=ind/ydim+1
         ind = ind + 1
      enddo
   endif

   !-------------------------------------------------------------------
   ! Create the Grid:  Allocate space for the Grid object.
   ! the minIndex hasn't been set, so it defaults to (1,1,...). The
   ! default coordDep1 and coordDep2 are (/ESMF_DIM_ARB/) where
   ! ESMF_DIM_ARB represents the collapsed dimension for the
   ! arbitrarily distributed grid dimensions.  For the undistributed
   ! grid dimension, the default value for coordDep3 is (/3/).  The
   ! default values for coordDepX in the arbitrary distribution are
   ! different from the non-arbitrary distributions.
   !-------------------------------------------------------------------
   grid3D=ESMF_GridCreateNoPeriDim( &
            maxIndex = (/xdim, ydim, zdim/), &
            arbIndexList = localArbIndex, &
            arbIndexCount = localArbIndexCount, &
            rc=rc)

   !-------------------------------------------------------------------
   ! Allocate coordinate storage for the center stagger location, the
   ! only stagger location supported for the arbitrary distribution.
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid3D, &
          staggerloc=ESMF_STAGGERLOC_CENTER_VCENTER, rc=rc)

   !------------------------------------------------------------------
   ! Fill in the coordinates for the center stagger location. There is
   ! always one DE per PET, so localDE is always 0
   !------------------------------------------------------------------
   call ESMF_GridGetCoord(grid3D, coordDim=1, localDE=0, &
          staggerLoc=ESMF_STAGGERLOC_CENTER,       &
          computationalLBound=lbnd,                 &
          computationalUBound=ubnd,                 &
          farrayPtr=centerX, rc=rc)

   !----------------------------------------------------------------
   ! Calculate and set coordinates in the first dimension.
   !----------------------------------------------------------------
   do i=lbnd(1),ubnd(1)
      centerX(i) = (localArbIndex(i,1)-0.5)*(360.0/xdim)
   enddo


   !----------------------------------------------------------------
   ! Get the local bounds of the global indexing for the second
   ! coordinate array on the local DE, and the pointer to the array.
   !----------------------------------------------------------------
   call ESMF_GridGetCoord(grid3D, coordDim=2, localDE=0,    &
          staggerloc=ESMF_STAGGERLOC_CENTER,                  &
          computationalLBound=lbnd, computationalUBound=ubnd, &
          farrayPtr=centerY, rc=rc)

   !----------------------------------------------------------------
   ! Calculate and set coordinates in the second dimension.
   !----------------------------------------------------------------
   do j=lbnd(1),ubnd(1)
      centerY(j) = (localArbIndex(j,2)-0.5)*(180.0/ydim)-90.0
   enddo

   !----------------------------------------------------------------
   ! Get the local bounds of the global indexing for the third
   ! coordinate array on the local DE, and the pointer to the array.
   !----------------------------------------------------------------
   call ESMF_GridGetCoord(grid3D, coordDim=3, localDE=0,   &
          staggerloc=ESMF_STAGGERLOC_CENTER,               &
          computationalLBound=lbnd, computationalUBound=ubnd,&
          farrayPtr=centerZ, rc=rc)

   !----------------------------------------------------------------
   ! Calculate and set the vertical coordinates
   !----------------------------------------------------------------
   do k=lbnd(1),ubnd(1)
      centerZ(k) = 4000.0*( (1./zdim)*(k-1))**2
   enddo


31.3.8 Create a curvilinear Grid using the coordinates defined in a SCRIP file

ESMF supports the creation of a 2D curvilinear Grid using the coordinates defined in a SCRIP format Grid file [13]. The grid contained in the file must be a 2D logically rectangular grid with grid_rank in the file set to 2. The center coordinates variables grid_center_lat and grid_center_lon in the file are placed in the ESMF_STAGGERLOC_CENTER location. If the parameter addCornerStagger in the ESMF_GridCreate call is set to .true., then the variables grid_corner_lat and grid_corner_lon in the file are used to set the ESMF_STAGGERLOC_CORNER coordinates, otherwise they are ignored. The values in the grid_imask variable in the file are used to set the ESMF_GRIDITEM_MASK in the Grid.

The following example code shows you how to create a 2D Grid with both center and corner coordinates using a SCRIP file and a row only regular distribution:

   grid2D = ESMF_GridCreate(filename="data/T42_grid.nc",  &
              fileFormat=ESMF_FILEFORMAT_SCRIP,  &
              regDecomp=(/PetCount,1/), addCornerStagger=.true., rc=rc)

where T42_grid.nc is a 2D global grid of size (128x64) and the resulting Grid is distributed by partitioning the rows evenly over all the PETs.

ESMF also support the creation of a 2D Grid from the SCRIP format Grid file using a user specified ESMF_DistGrid. The following example code demonstrates the creation of an Grid object using a pre-defined DistGrid. The resulting Grid is the same as the one created above:

   distgrid = ESMF_DistGridCreate((/1,1/), (/128,64/), &
              regDecomp=(/PetCount,1/), rc=rc)
   grid2D = ESMF_GridCreate(filename="data/T42_grid.nc",  &
              fileFormat=ESMF_FILEFORMAT_SCRIP,  &
              distGrid=distgrid, addCornerStagger=.true., rc=rc)


31.3.9 Create an empty Grid in a parent Component for completion in a child Component

ESMF Grids can be created incrementally. To do this, the user first calls ESMF_GridEmptyCreate() to allocate the shell of a Grid. Next, we use the ESMF_GridEmptyComplete() call that fills in the Grid and does an internal commit to make it usable. For consistency's sake the ESMF_GridSetCommitShapeTile() call must occur on the same or a subset of the PETs as the ESMF_GridEmptyCreate() call. The ESMF_GridEmptyComplete() call uses the VM for the context in which it's executed and the "empty" Grid contains no information about the VM in which its create was run. This means that if the ESMF_GridEmptyComplete() call occurs in a subset of the PETs in which the ESMF_GridEmptyCreate() was executed that the Grid is created only in that subset. Inside the subset the Grid will be fine, but outside the subset the Grid objects will still be "empty" and not usable. The following example uses the incremental technique to create a rectangular 10x20 Grid with coordinates at the center and corner stagger locations.

!---------------------------------------------------------------------------
! IN THE PARENT COMPONENT:
! Create an empty Grid in the parent component for use in a child component.
! The parent may be defined on more PETs than the child component.
! The child's [vm or pet list] is passed into the create call so that
! the Grid is defined on the appropriate subset of the parent's PETs.
!---------------------------------------------------------------------------
   grid2D=ESMF_GridEmptyCreate(rc=rc)

!---------------------------------------------------------------------------
! IN THE CHILD COMPONENT:
! Set the Grid topology.  Here we define an irregularly distributed
! rectangular Grid.
!---------------------------------------------------------------------------
   call ESMF_GridEmptyComplete(grid2D,             &
                          countsPerDEDim1=(/6,4/),      &
                          countsPerDEDim2=(/10,3,7/), rc=rc)

!---------------------------------------------------------------------------
! Add Grid coordinates at the cell center location.
!---------------------------------------------------------------------------
   call ESMF_GridAddCoord(grid2D, staggerLoc=ESMF_STAGGERLOC_CENTER, rc=rc)

!---------------------------------------------------------------------------
! Add Grid coordinates at the corner stagger location.
!---------------------------------------------------------------------------
   call ESMF_GridAddCoord(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER, rc=rc)


31.3.10 Create a six-tile cubed sphere Grid

This example creates a multi-tile Grid to represent a cubed sphere grid. Each of the six tiles making up the cubed sphere has 45 elements on each side, so the total number of elements is 45x45x6=12150. Each tile is decomposed using a regular decomposition. The first two tiles are decomposed into 2x2 blocks each and the remaining 4 tiles are decomposed into 1x2 block. A total of 16 DEs are used.

In this example, both the center and corner coordinates will be added to the grid.

     ! Set up decomposition for each tile
     allocate(decomptile(2,6))
     decomptile(:,1)=(/2,2/) ! Tile 1
     decomptile(:,2)=(/2,2/) ! Tile 2
     decomptile(:,3)=(/1,2/) ! Tile 3
     decomptile(:,4)=(/1,2/) ! Tile 4
     decomptile(:,5)=(/1,2/) ! Tile 5
     decomptile(:,6)=(/1,2/) ! Tile 6

     ! Create cubed sphere grid
     grid2D = ESMF_GridCreateCubedSphere(tileSize=45, regDecompPTile=decomptile, &
                 staggerLocList=(/ESMF_STAGGERLOC_CENTER, ESMF_STAGGERLOC_CORNER/), rc=rc)


31.3.11 Create a six-tile cubed sphere Grid and apply Schmidt transform

This example creates the same cubed sphere grid with the same regular decomposition as in 31.3.10 with a few differences. First, the coordinates of the grid are of type ESMF_TYPEKIND_R4 instead of the default ESMF_TYPEKIND_R8. Secondly, the coordinate system is ESMF_COORDSYS_SPH_RAD instead of the default ESMF_COORDSYS_SPH_DEG. Lastly, the grid was then transformed using Schmidt Transformation algorithm on an arbitrary target point and a streatching factor. An optional argument TransformArgs of type ESMF_CubedSphereTransform_Args is used to pass the Schmidt Transform arguments. ESMF_CubedSphereTransform_Args is defined as follows:

     type ESMF_CubedSphereTransform_Args
        real(ESMF_KIND_R8) :: stretch_factor, target_lat, target_lon
     end type

Note target_lat and target_lon are in radians.

     transformArgs%stretch_factor = 0.5;
     transformArgs%target_lon = 0.0; ! in radians
     transformArgs%target_lat = 1.3; ! in radians
     grid2D = ESMF_GridCreateCubedSphere(tileSize=45, regDecompPTile=decomptile, &
                 staggerLocList = (/ESMF_STAGGERLOC_CENTER, ESMF_STAGGERLOC_CORNER/), &
                 coordTypeKind = ESMF_TYPEKIND_R4, &
                 coordSys = ESMF_COORDSYS_SPH_RAD, &
                 transformArgs=transformArgs, &
                 rc=rc)


31.3.12 Create a six-tile cubed sphere Grid from a GRIDSPEC Mosaic file

This example creates a six-tile Grid to represent a cubed sphere grid defined in a GRIDSPEC Mosaic file C48_mosaic.nc. The GRIDSPEC mosaic file format is defined in the document GRIDSPEC: A standard for the description of grids used in Earth System models by V. Balaji, Alistair Adcroft and Zhi Liang.

The mosaic file contains the following information:

  netcdf C48_mosaic {
  dimensions:
         ntiles = 6 ;
         ncontact = 12 ;
         string = 255 ;
  variables:
         char mosaic(string) ;
                 mosaic:standard_name = "grid_mosaic_spec" ;
                 mosaic:children = "gridtiles" ;
                 mosaic:contact_regions = "contacts" ;
                 mosaic:grid_descriptor = "" ;
         char gridlocation(string) ;
                 gridlocation:standard_name = "grid_file_location" ;
         char gridfiles(ntiles, string) ;
         char gridtiles(ntiles, string) ;
         char contacts(ncontact, string) ;
                 contacts:standard_name = "grid_contact_spec" ;
                 contacts:contact_type = "boundary" ;
                 contacts:alignment = "true" ;
                 contacts:contact_index = "contact_index" ;
                 contacts:orientation = "orient" ;
         char contact_index(ncontact, string) ;
                 contact_index:standard_name = "starting_ending_point_index_of_contact" ;
 
  // global attributes:
                 :grid_version = "0.2" ;
                 :code_version = "$Name: testing $" ;
  data:
  
  mosaic = "C48_mosaic" ;
  
  gridlocation = "/archive/z1l/tools/test_20091028/output_all/" ;
  
  gridfiles =
    "horizontal_grid.tile1.nc",
    "horizontal_grid.tile2.nc",
    "horizontal_grid.tile3.nc",
    "horizontal_grid.tile4.nc",
    "horizontal_grid.tile5.nc",
    "horizontal_grid.tile6.nc" ;
  
  gridtiles =
    "tile1",
    "tile2",
    "tile3",
    "tile4",
    "tile5",
    "tile6" ;
  
  contacts =
    "C48_mosaic:tile1::C48_mosaic:tile2",
    "C48_mosaic:tile1::C48_mosaic:tile3",
    "C48_mosaic:tile1::C48_mosaic:tile5",
    "C48_mosaic:tile1::C48_mosaic:tile6",
    "C48_mosaic:tile2::C48_mosaic:tile3",
    "C48_mosaic:tile2::C48_mosaic:tile4",
    "C48_mosaic:tile2::C48_mosaic:tile6",
    "C48_mosaic:tile3::C48_mosaic:tile4",
    "C48_mosaic:tile3::C48_mosaic:tile5",
    "C48_mosaic:tile4::C48_mosaic:tile5",
    "C48_mosaic:tile4::C48_mosaic:tile6",
    "C48_mosaic:tile5::C48_mosaic:tile6" ;
  
   contact_index =
    "96:96,1:96::1:1,1:96",
    "1:96,96:96::1:1,96:1",
    "1:1,1:96::96:1,96:96",
    "1:96,1:1::1:96,96:96",
    "1:96,96:96::1:96,1:1",
    "96:96,1:96::96:1,1:1",
    "1:96,1:1::96:96,96:1",
    "96:96,1:96::1:1,1:96",
    "1:96,96:96::1:1,96:1",
    "1:96,96:96::1:96,1:1",
    "96:96,1:96::96:1,1:1",
    "96:96,1:96::1:1,1:96" ;
  }

A dummy variable with its standard_name attribute set to grid_mosaic_spec is required. The children attribute of this dummy variable provides the variable name that contains the tile names and the contact_region attribute points to the variable name that defines a list of tile pairs that are connected to each other. For a Cubed Sphere grid, there are six tiles and 12 connections. The contacts variable has three required attributes: standard_name, contact_type, and contact_index. startand_name has to be set to grid_contact_spec. contact_type has to be boundary. ESMF does not support overlapping contact regions. contact_index defines the variable name that contains the information how the two adjacent tiles are connected to each other. The contact_index variable contains 12 entries. Each entry contains the index of four points that defines the two edges that contact to each other from the two neighboring tiles. Assuming the four points are A, B, C, and D. A and B defines the edge of tile 1 and C and D defines the edge of tile2. A is the same point as C and B is the same as D. (Ai, Aj) is the index for point A. The entry looks like this:

    Ai:Bi,Aj:Bj::Ci:Di,Cj:Dj

The associated tile file names are defined in variable gridfiles and the directory path is defined in variable gridlocation. The gridlocation can be overwritten with an optional arguemnt TileFilePath. Each tile is decomposed using a regular decomposition. The first two tiles are decomposed into 2x2 blocks each and the remaining 4 tiles are decomposed into 1x2 block. A total of 16 DEs are used.

ESMF_GridCreateMosaic() first reads in the mosaic file and defines the tile connections in the ESMF_DistGrid using the information defined in variables contacts and contact_index. Then it reads in the coordinates defined in the tile files if the optional argument staggerLocList is provided. The coordinates defined in the tile file are a supergrid. A supergrid contains all the stagger locations in one grid. It contains the corner, edge and center coordinates all in one 2D array. In this example, there are 48 elements in each side of a tile, therefore, the size of the supergrid is 48*2+1=97, i.e. 97x97.

Here is the header of one of the tile files:

  netcdf horizontal_grid.tile1 {
  dimensions:
         string = 255 ;
         nx = 96 ;
         ny = 96 ;
         nxp = 97 ;
         nyp = 97 ;
  variables:
         char tile(string) ;
                 tile:standard_name = "grid_tile_spec" ;
                 tile:geometry = "spherical" ;
                 tile:north_pole = "0.0 90.0" ;
                 tile:projection = "cube_gnomonic" ;
                 tile:discretization = "logically_rectangular" ;
                 tile:conformal = "FALSE" ;
         double x(nyp, nxp) ;
                 x:standard_name = "geographic_longitude" ;
                 x:units = "degree_east" ;
         double y(nyp, nxp) ;
                 y:standard_name = "geographic_latitude" ;
                 y:units = "degree_north" ;
         double dx(nyp, nx) ;
                 dx:standard_name = "grid_edge_x_distance" ;
                 dx:units = "meters" ;
         double dy(ny, nxp) ;
                 dy:standard_name = "grid_edge_y_distance" ;
                 dy:units = "meters" ;
         double area(ny, nx) ;
                 area:standard_name = "grid_cell_area" ;
                 area:units = "m2" ;
         double angle_dx(nyp, nxp) ;
                 angle_dx:standard_name = "grid_vertex_x_angle_WRT_geographic_east" ;
                 angle_dx:units = "degrees_east" ;
         double angle_dy(nyp, nxp) ;
                 angle_dy:standard_name = "grid_vertex_y_angle_WRT_geographic_north" ;
                 angle_dy:units = "degrees_north" ;
         char arcx(string) ;
                 arcx:standard_name = "grid_edge_x_arc_type" ;
                 arcx:north_pole = "0.0 90.0" ;
  
  // global attributes:
                 :grid_version = "0.2" ;
                 :code_version = "$Name: testing $" ;
                 :history = "/home/z1l/bin/tools_20091028/make_hgrid --grid_type gnomonic_ed --nlon 96" ;
  }

The tile file not only defines the coordinates at all staggers, it also has a complete specification of distances, angles, and areas. In ESMF, we currently only use the geographic_longitude and geographic_latitude variables.

     ! Set up decomposition for each tile
     allocate(decomptile(2,6))
     decomptile(:,1)=(/2,2/) ! Tile 1
     decomptile(:,2)=(/2,2/) ! Tile 2
     decomptile(:,3)=(/1,2/) ! Tile 3
     decomptile(:,4)=(/1,2/) ! Tile 4
     decomptile(:,5)=(/1,2/) ! Tile 5
     decomptile(:,6)=(/1,2/) ! Tile 6

     ! Create cubed sphere grid without reading in the coordinates
     grid2D = ESMF_GridCreateMosaic(filename='data/C48_mosaic.nc', &
                tileFilePath='./data/', regDecompPTile=decomptile, rc=rc)

     ! Create cubed sphere grid and read in the center and corner stagger coordinates
     ! from the tile files

     grid2D = ESMF_GridCreateMosaic(filename='data/C48_mosaic.nc', &
                staggerLocList=(/ESMF_STAGGERLOC_CENTER, ESMF_STAGGERLOC_CORNER/), &
                tileFilePath='./data/', regDecompPTile=decomptile, rc=rc)

     ! Create cubed sphere grid and read in the edge staggers' coordinates
     ! from the tile files, set the coordTypeKind to ESMF_TYPEKIND_R4

     grid2D = ESMF_GridCreateMosaic(filename='data/C48_mosaic.nc', &
                staggerLocList=(/ESMF_STAGGERLOC_EDGE1, ESMF_STAGGERLOC_EDGE2/), &
                coordTypeKind = ESMF_TYPEKIND_R4, &
                tileFilePath='./data/', regDecompPTile=decomptile, rc=rc)


31.3.13 Grid stagger locations

A useful finite difference technique is to place different physical quantities at different locations within a grid cell. This staggering of the physical variables on the mesh is introduced so that the difference of a field is naturally defined at the location of another variable. This method was first formalized by Mesinger and Arakawa (1976).

To support the staggering of variables, the Grid provides the idea of stagger locations. Stagger locations refer to the places in a Grid cell that can contain coordinates or other data and once a Grid is associated with a Field object, field data. Typically Grid data can be located at the cell center, at the cell corners, or at the cell faces, in 2D, 3D, and higher dimensions. (Note that any Arakawa stagger can be constructed of a set of Grid stagger locations.) There are predefined stagger locations (see Section 31.2.6), or, should the user wish to specify their own, there is also a set of methods for generating custom locations (See Section 31.3.25). Users can put Grid data (e.g. coordinates) at multiple stagger locations in a Grid. In addition, the user can create a Field at any of the stagger locations in a Grid.

By default the Grid data array at the center stagger location starts at the bottom index of the Grid (default (1,1..,1)) and extends up to the maximum cell index in the Grid (e.g. given by the maxIndex argument). Other stagger locations also start at the bottom index of the Grid, however, they can extend to +1 element beyond the center in some dimensions to allow for the extra space to surround the center elements. See Section 31.3.25 for a description of this extra space and how to adjust if it necessary. There are ESMF_GridGet subroutines (e.g. ESMF_GridGetCoord() or ESMF_GridGetItem()) which can be used to retrieve the stagger bounds for the piece of Grid data on a particular DE.

31.3.14 Associate coordinates with stagger locations

The primary type of data the Grid is responsible for storing is coordinates. The coordinate values in a Grid can be employed by the user in calculations or to describe the geometry of a Field. The Grid coordinate values are also used by ESMF_FieldRegridStore() when calculating the interpolation matrix between two Fields. The user can allocate coordinate arrays without setting coordinate values using the ESMF_GridAddCoord() call. (See Section 31.3.16 for a discussion of setting/getting coordinate values.) When adding or accessing coordinate data, the stagger location is specified to tell the Grid method where in the cell to get the data. The different stagger locations may also have slightly different index ranges and sizes. Please see Section 31.3.13 for a discussion of Grid stagger locations.

The following example adds coordinate storage to the corner stagger location in a Grid using one of the predefined stagger locations.

   call ESMF_GridAddCoord(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER, rc=rc)

Note only the center stagger location ESMF_STAGGERLOC_CENTER is supported in an arbitrarily distributed Grid.

31.3.15 Specify the relationship of coordinate Arrays to index space dimensions

To specify how the coordinate arrays are mapped to the index dimensions the arguments coordDep1, coordDep2, and coordDep3 are used, each of which is a Fortran array. The values of the elements in a coordDep array specify which index dimension the corresponding coordinate dimension maps to. For example, coordDep1=(/1,2/) means that the first dimension of coordinate 1 maps to index dimension 1 and the second maps to index dimension 2. For a grid with non-arbitrary distribution, the default values for coordDep1, coordDep2 and coordDep3 are /1,2..,gridDimCount/. This default thus specifies a curvilinear grid.

The following call demonstrates the creation of a 10x20 2D rectilinear grid where the first coordinate component is mapped to the second index dimension (i.e. is of size 20) and the second coordinate component is mapped to the first index dimension (i.e. is of size 10).

   grid2D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/5,5/), &
          countsPerDEDim2=(/7,7,6/),                    &
          coordDep1=(/2/),                              &
          coordDep2=(/1/), rc=rc)

The following call demonstrates the creation of a 10x20x30 2D plus 1 curvilinear grid where coordinate component 1 and 2 are still 10x20, but coordinate component 3 is mapped just to the third index dimension.

   grid2D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/6,4/), &
          countsPerDEDim2=(/10,7,3/), countsPerDEDim3=(/30/), &
          coordDep1=(/1,2/), coordDep2=(/1,2/), &
          coordDep3=(/3/), rc=rc)

By default the local piece of the array on each PET starts at (1,1,..), however, the indexing for each grid coordinate array on each DE may be shifted to the global indices by using the indexflag. For example, the following call switches the grid to use global indices.

   grid2D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/6,4/), &
           countsPerDEDim2=(/10,7,3/), indexflag=ESMF_INDEX_GLOBAL, rc=rc)

For an arbitrarily distributed grid, the default value of a coordinate array dimension is ESMF_DIM_ARB if the index dimension is arbitrarily distributed and is n where n is the index dimension itself when it is not distributed. The following call is equivalent to the example in Section 31.3.7

   grid3D=ESMF_GridCreateNoPeriDim( &
            maxIndex = (/xdim, ydim, zdim/), &
            arbIndexList = localArbIndex, &
            arbIndexCount = localArbIndexCount,          &
            coordDep1 = (/ESMF_DIM_ARB/), &
            coordDep2 = (/ESMF_DIM_ARB/), &
            coordDep3 = (/3/), &
            rc=rc)

The following call uses non-default coordDep1, coordDep2, and coordDep3 to create a 3D curvilinear grid with its horizontal dimensions arbitrarily distributed.

   grid3D=ESMF_GridCreateNoPeriDim( &
            maxIndex = (/xdim, ydim, zdim/), &
            arbIndexList = localArbIndex, &
            arbIndexCount = localArbIndexCount,          &
            coordDep1 = (/ESMF_DIM_ARB, 3/), &
            coordDep2 = (/ESMF_DIM_ARB, 3/), &
            coordDep3 = (/ESMF_DIM_ARB, 3/), &
            rc=rc)


31.3.16 Access coordinates

Once a Grid has been created, the user has several options to access the Grid coordinate data. The first of these, ESMF_GridSetCoord(), enables the user to use ESMF Arrays to set data for one stagger location across the whole Grid. For example, the following sets the coordinates in the first dimension (e.g. x) for the corner stagger location to those in the ESMF Array arrayCoordX.

   call ESMF_GridSetCoord(grid2D, &
          staggerLoc=ESMF_STAGGERLOC_CORNER, &
          coordDim=1, array=arrayCoordX, rc=rc)

The method ESMF_GridGetCoord() allows the user to obtain a reference to an ESMF Array which contains the coordinate data for a stagger location in a Grid. The user can then employ any of the standard ESMF_Array tools to operate on the data. The following copies the coordinates from the second component of the corner and puts it into the ESMF Array arrayCoordY.

   call ESMF_GridGetCoord(grid2D,    &
          staggerLoc=ESMF_STAGGERLOC_CORNER,    &
          coordDim=2,                           &
          array=arrayCoordY, rc=rc)

Alternatively, the call ESMF_GridGetCoord() gets a Fortran pointer to the coordinate data. The user can then operate on this array in the usual manner. The following call gets a reference to the Fortran array which holds the data for the second coordinate (e.g. y).

   call ESMF_GridGetCoord(grid2D, coordDim=2, localDE=0, &
          staggerloc=ESMF_STAGGERLOC_CORNER, farrayPtr=coordY2D, rc=rc)


31.3.17 Associate items with stagger locations

The ESMF Grids contain the ability to store other kinds of data beyond coordinates. These kinds of data are referred to as "items". Although the user is free to use this data as they see fit, the user should be aware that this data may also be used by other parts of ESMF (e.g. the ESMF_GRIDITEM_MASK item is used in regridding). Please see Section 31.2.2 for a list of valid items.

Like coordinates items are also created on stagger locations. When adding or accessing item data, the stagger location is specified to tell the Grid method where in the cell to get the data. The different stagger locations may also have slightly different index ranges and sizes. Please see Section 31.3.13 for a discussion of Grid stagger locations. The user can allocate item arrays without setting item values using the ESMF_GridAddItem() call. (See Section 31.3.18 for a discussion of setting/getting item values.)

The following example adds mask item storage to the corner stagger location in a grid.

   call ESMF_GridAddItem(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER, &
          itemflag=ESMF_GRIDITEM_MASK, rc=rc)


31.3.18 Access items

Once an item has been added to a Grid, the user has several options to access the data. The first of these, ESMF_GridSetItem(), enables the user to use ESMF Arrays to set data for one stagger location across the whole Grid. For example, the following sets the mask item in the corner stagger location to those in the ESMF Array arrayMask.

   call ESMF_GridSetItem(grid2D,             &
          staggerLoc=ESMF_STAGGERLOC_CORNER, &
          itemflag=ESMF_GRIDITEM_MASK, array=arrayMask, rc=rc)

The method ESMF_GridGetItem() allows the user to get a reference to the Array which contains item data for a stagger location on a Grid. The user can then employ any of the standard ESMF_Array tools to operate on the data. The following gets the mask data from the corner and puts it into the ESMF Array arrayMask.

   call ESMF_GridGetItem(grid2D,             &
          staggerLoc=ESMF_STAGGERLOC_CORNER, &
          itemflag=ESMF_GRIDITEM_MASK,           &
          array=arrayMask, rc=rc)

Alternatively, the call ESMF_GridGetItem() gets a Fortran pointer to the item data. The user can then operate on this array in the usual manner. The following call gets a reference to the Fortran array which holds the data for the mask data.

   call ESMF_GridGetItem(grid2D, localDE=0,   &
          staggerloc=ESMF_STAGGERLOC_CORNER,  &
          itemflag=ESMF_GRIDITEM_MASK, farrayPtr=mask2D, rc=rc)


31.3.19 Grid regions and bounds

Like an Array or a Field, the index space of each stagger location in the Grid contains an exclusive region, a computational region and a total region. Please see Section 28.2.6 for an in depth description of these regions.

The exclusive region is the index space defined by the distgrid of each stagger location of the Grid. This region is the region which is owned by the DE and is the region operated on by communication methods such as ESMF_FieldRegrid(). The exclusive region for a stagger location is based on the exclusive region defined by the DistGrid used to create the Grid. The size of the stagger exclusive region is the index space for the Grid cells, plus the stagger padding.

The default stagger padding depends on the topology of the Grid. For an unconnected dimension the stagger padding is a width of 1 on the upper side (i.e. gridEdgeUWidth=(1,1,1,1...)). For a periodic dimension there is no stagger padding. By adjusting gridEdgeLWidth and gridEdgeUWidth, the user can set the stagger padding for the whole Grid and thus the exclusive region can be adjusted at will around the index space corresponding to the cells. The user can also use staggerEdgeLWidth and staggerEdgeUWidth to adjust individual stagger location padding within the Grid's padding (Please see Section 31.3.26 for further discussion of customizing the stagger padding).

Figure 17: An example of a Grid's exclusive region for the corner stagger
\scalebox{0.75}{\includegraphics{GridExclusiveReg}}

Figure 17 shows an example of a Grid exclusive region for the ESMF_STAGGERLOC_CORNER stagger with default stagger padding. This exclusive region would be for a Grid generated by either of the following calls:

  grid2D=ESMF_GridCreateNoPeriDim(regDecomp=(/2,4/), maxIndex=(/5,15/), &
           indexflag=ESMF_INDEX_GLOBAL, rc=rc)

  grid2D=ESMF_GridCreateNoPeriDim(countsPerDEDim1=(/4,4,4,3/), &
           countsPerDEDim2=(/3,2/), indexflag=ESMF_INDEX_GLOBAL, rc=rc)

Each rectangle in this diagram represents a DE and the numbers along the sides are the index values of the locations in the DE. Note that the exclusive region has one extra index location in each dimension than the number of cells because of the padding for the larger corner stagger location.

The computational region is a user-settable region which can be used to distinguish a particular area for computation. The Grid doesn't currently contain functionality to let the user set the computational region so it defaults to the exclusive region. However, if the user sets an Array holding different computational bounds into the Grid then that Array's computational bounds will be used.

The total region is the outermost boundary of the memory allocated on each DE to hold the data for the stagger location on that DE. This region can be as small as the exclusive region, but may be larger to include space for halos, memory padding, etc. The total region is what is enlarged to include space for halos, and the total region must be large enough to contain the maximum halo operation on the Grid. The Grid doesn't currently contain functionality to let the user set the total region so it defaults to the exclusive region. However, if the user sets an Array holding different total bounds into the Grid then that Array's total bounds will be used.

The user can retrieve a set of bounds for each index space region described above: exclusive bounds, computational bounds, and total bounds. Note that although some of these are similar to bounds provided by ESMF_Array subroutines (see Section 28.2.6) the format here is different. The Array bounds are only for distributed dimensions and are ordered to correspond to the dimension order in the associated DistGrid. The bounds provided by the Grid are ordered according to the order of dimensions of the data in question. This means that the bounds provided should be usable "as is" to access the data.

Each of the three types of bounds refers to the maximum and minimum per dimension of the index ranges of a particular region. The parameters referring to the maximums contain a 'U' for upper. The parameters referring to the minimums contain an 'L' for lower. The bounds and associated quantities are almost always given on a per DE basis. The three types of bounds exclusiveBounds, computationalBounds, and totalBounds refer to the ranges of the exclusive region, the computational region, and the total region. Each of these bounds also has a corresponding count parameter which gives the number of items across that region (on a DE) in each dimension. (e.g. totalCount(d)=totallUBound(i)-totalLBound(i)+1). Width parameters give the spacing between two different types of region. The computationalWidth argument gives the spacing between the exclusive region and the computational region. The totalWidth argument gives the spacing between the total region and the computational region. Like the other bound information these are typically on a per DE basis, for example specifying totalLWidth=(1,1) makes the bottom of the total region one lower in each dimension than the computational region on each DE. The exceptions to the per DE rule are staggerEdgeWidth, and gridEdgeWidth which give the spacing only on the DEs along the boundary of the Grid.

All the above bound discussions only apply to the grid with non-arbitrary distributions, i.e., regular or irregular distributions. For an arbitrarily distributed grid, only center stagger location is supported and there is no padding around the grid. Thus, the exclusive bounds, the total bounds and the computational bounds are identical and staggerEdgeWidth, and gridEdgeWidth are all zeros.

31.3.20 Get Grid coordinate bounds

When operating on coordinates the user may often wish to retrieve the bounds of the piece of coordinate data on a particular local DE. This is useful for iterating through the data to set coordinates, retrieve coordinates, or do calculations. The method ESMF_GridGetCoord allows the user to retrieve bound information for a particular coordinate array.

As described in the previous section there are three types of bounds the user can get: exclusive bounds, computational bounds, and total bounds. The bounds provided by ESMF_GridGetCoordBounds are for both distributed and undistributed dimensions and are ordered according to the order of dimensions in the coordinate. This means that the bounds provided should be usable "as is" to access data in the coordinate array. In the case of factorized coordinate Arrays where a coordinate may have a smaller dimension than its associated Grid, then the dimension of the coordinate's bounds are the dimension of the coordinate, not the Grid.

The following is an example of retrieving the bounds for localDE 0 for the first coordinate array from the corner stagger location.

   call ESMF_GridGetCoordBounds(grid2D, coordDim=1, localDE=0,  &
          staggerLoc=ESMF_STAGGERLOC_CORNER,                         &
          exclusiveLBound=elbnd, exclusiveUBound=eubnd,              &
          computationalLBound=clbnd, computationalUBound=cubnd,      &
          totalLBound=tlbnd, totalUBound=tubnd, rc=rc)

31.3.21 Get Grid stagger location bounds

When operating on data stored at a particular stagger in a Grid the user may find it useful to be able to retrieve the bounds of the data on a particular local DE. This is useful for iterating through the data for computations or allocating arrays to hold the data. The method ESMF_GridGet allows the user to retrieve bound information for a particular stagger location.

As described in Section 31.3.19 there are three types of bounds the user can typically get, however, the Grid doesn't hold data at a stagger location (that is the job of the Field), and so no Array is contained there and so no total region exists, so the user may only retrieve exclusive and computational bounds from a stagger location. The bounds provided by ESMF_GridGet are ordered according to the order of dimensions in the Grid.

The following is an example of retrieving the bounds for localDE 0 from the corner stagger location.

   call ESMF_GridGet(grid2D, localDE=0,                         &
          staggerLoc=ESMF_STAGGERLOC_CORNER,                    &
          exclusiveLBound=elbnd, exclusiveUBound=eubnd,         &
          computationalLBound=clbnd, computationalUBound=cubnd, rc=rc)

31.3.22 Get Grid stagger location information

In addition to the per DE information that can be accessed about a stagger location there is some global information that can accessed by using ESMF_GridGet without specifying a localDE. One of the uses of this information is to create an ESMF Array to hold data for a stagger location.

The information currently available from a stagger location is the distgrid. The distgrid gives the distgrid which describes the size and distribution of the elements in the stagger location.

The following is an example of retrieving information for localDE 0 from the corner stagger location.

    ! Get info about staggerloc
    call ESMF_GridGet(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER,  &
           distgrid=staggerDistgrid, &
           rc=rc)

31.3.23 Create an Array at a stagger location

In order to create an Array to correspond to a Grid stagger location several pieces of information need to be obtained from both the Grid and the stagger location in the Grid.

The information that needs to be obtained from the Grid is the distgridToGridMap to ensure that the new Array has its dimensions are mapped correctly to the Grid. These are obtained using the ESMF_GridGet method.

The information that needs to be obtained from the stagger location is the distgrid that describes the size and distribution of the elements in the stagger location. This information can be obtained using the stagger location specific ESMF_GridGet method.

The following is an example of using information from a 2D Grid with non-arbitrary distribution to create an Array corresponding to a stagger location.

    ! Get info from Grid
    call ESMF_GridGet(grid2D, distgridToGridMap=distgridToGridMap, rc=rc)

    ! Get info about staggerloc
    call ESMF_GridGet(grid2D, staggerLoc=ESMF_STAGGERLOC_CORNER, &
           distgrid=staggerDistgrid, &
           rc=rc)

    ! construct ArraySpec
    call ESMF_ArraySpecSet(arrayspec, rank=2, typekind=ESMF_TYPEKIND_R8, rc=rc)

    ! Create an Array based on info from grid
    array=ESMF_ArrayCreate(arrayspec=arrayspec, &
            distgrid=staggerDistgrid, distgridToArrayMap=distgridToGridMap, &
            rc=rc)

Creating an Array for a Grid with arbitrary distribution is different. For a 2D Grid with both dimension arbitrarily distributed, the Array dimension is 1. For a 3D Grid with two arbitrarily distributed dimensions and one undistributed dimension, the Array dimension is 2. In general, if the Array does not have any ungridded dimension, the Array dimension should be 1 plus the number of undistributed dimensions of the Grid.

The following is an example of creating an Array for a 3D Grid with 2 arbitrarily distributed dimensions such as the one defined in Section 31.3.7.

    ! Get distGrid from Grid
    call ESMF_GridGet(grid3D, distgrid=distgrid, rc=rc)

    ! construct ArraySpec
    call ESMF_ArraySpecSet(arrayspec, rank=2, typekind=ESMF_TYPEKIND_R8, rc=rc)

    ! Create an Array based on the presence of distributed dimensions
    array=ESMF_ArrayCreate(arrayspec=arrayspec,distgrid=distgrid, rc=rc)


31.3.24 Create more complex Grids using DistGrid

Besides the shortcut methods for creating a Grid object such as ESMF_GridCreateNoPeriDim(), there is a set of methods which give the user more control over the specifics of the grid. The following describes the more general interface, using DistGrid. The basic idea is to first create an ESMF DistGrid object describing the distribution and shape of the Grid, and then to employ that to either directly create the Grid or first create Arrays and then create the Grid from those. This method gives the user maximum control over the topology and distribution of the Grid. See the DistGrid documentation in Section 36.1 for an in-depth description of its interface and use.

As an example, the following call constructs a 10x20 Grid with a lower bound of (1,2).

   ! Create DistGrid
   distgrid2D = ESMF_DistGridCreate(minIndex=(/1,2/), maxIndex=(/11,22/), &
           rc=rc)

   ! Create Grid
   grid3D=ESMF_GridCreate(distGrid=distgrid2D, rc=rc)

To alter which dimensions are distributed, the distgridToGridMap argument can be used. The distgridToGridMap is used to set which dimensions of the Grid are mapped to the dimensions described by maxIndex. In other words, it describes how the dimensions of the underlying default DistGrid are mapped to the Grid. Each entry in distgridToGridMap contains the Grid dimension to which the corresponding DistGrid dimension should be mapped. The following example illustrates the creation of a Grid where the largest dimension is first. To accomplish this the two dimensions are swapped.

   ! Create DistGrid
   distgrid2D = ESMF_DistGridCreate(minIndex=(/1,2/), maxIndex=(/11,22/), &
        rc=rc)

   ! Create Grid
   grid2D=ESMF_GridCreate(distGrid=distgrid2D, distgridToGridMap=(/2,1/), &
        rc=rc)


31.3.25 Specify custom stagger locations

Although ESMF provides a set of predefined stagger locations (See Section 31.2.6), the user may need one outside this set. This section describes the construction of custom stagger locations.

To completely specify a stagger for an arbitrary number of dimensions, we define the stagger location in terms of a set of cartesian coordinates. The cell is represented by a n-dimensional cube with sides of length 2, and the coordinate origin located at the center of the cell. The geometry of the cell is for reference purposes only, and does not literally represent the actual shape of the cell. Think of this method instead as an easy way to specify a part (e.g. center, corner, face) of a higher dimensional cell which is extensible to any number of dimensions.

To illustrate this approach, consider a 2D cell. In 2 dimensions the cell is represented by a square. An xy axis is placed at its center, with the positive x-axis oriented East and the positive y-axis oriented North. The resulting coordinate for the lower left corner is at $(-1,-1)$, and upper right corner at $(1,1)$. However, because our staggers are symmetric they don't need to distinguish between the $-1$, and the $1$, so we only need to concern ourselves with the first quadrant of this cell. We only need to use the $1$, and the $0$, and many of the cell locations collapse together (e.g. we only need to represent one corner). See figure 18 for an illustration of these concepts.

Figure 18: An example of specifying 2D stagger locations using coordinates.
\scalebox{0.75}{\includegraphics{GridCustStaggerLoc}}

The cell center is represented by the coordinate pair $(0,0)$ indicating the origin. The cell corner is $+1$ in each direction, giving a coordinate pair of $(1,1)$. The edges are each $+1$ in one dimension and $0$ in the other indicating that they're even with the center in one dimension and offset in the other.

For three dimensions, the vertical component of the stagger location can be added by simply adding an additional coordinate. The three dimensional generalization of the cell center becomes $(0,0,0)$ and the cell corner becomes $(1,1,1)$. The rest of the 3D stagger locations are combinations of $+1$ offsets from the center.

To generalize this to $d$ dimensions, to represent a $d$ dimensional stagger location. A set of $d$ $0$ and $1$ is used to specify for each dimension whether a stagger location is aligned with the cell center in that dimension ($0$), or offset by $+1$ in that dimension ($1$). Using this scheme we can represent any symmetric stagger location.

To construct a custom stagger location in ESMF the subroutine ESMF_StaggerLocSet() is used to specify, for each dimension, whether the stagger is located at the interior (0) or on the boundary (1) of the cell. This method allows users to construct stagger locations for which there is no predefined value. In this example, it's used to set the 4D center and 4D corner locations.

   ! Set Center
   call ESMF_StaggerLocSet(staggerLoc,loc=(/0,0,0,0/),rc=rc)

   call ESMF_GridAddCoord(grid4D, staggerLoc=staggerLoc, rc=rc)

   ! Set Corner
   call ESMF_StaggerLocSet(staggerLoc,loc=(/1,1,1,1/),rc=rc)

   call ESMF_GridAddCoord(grid4D, staggerLoc=staggerLoc, rc=rc)


31.3.26 Specify custom stagger padding

There is an added complication with the data (e.g. coordinates) stored at stagger locations in that they can require different amounts of storage depending on the underlying Grid type.

Figure 19: An example 2D Grid with cell centers and corners.
\scalebox{0.75}{\includegraphics{GridCellsAndCorners}}

Consider the example 2D grid in figure 19, where the dots represent the cell corners and the “+” represents the cell centers. For the corners to completely enclose the cell centers (symmetric stagger), the number of corners in each dimension needs to be one greater then the number of cell centers. In the above figure, there are two rows and three columns of cell centers. To enclose the cell centers, there must be three rows and four columns of cell corners. This is true in general for Grids without periodicity or other connections. In fact, for a symmetric stagger, given that the center location requires n x m storage, the corresponding corner location requires n+1 x m+1, and the edges, depending on the side, require n+1 x m or m+1 x n. In order to add the extra storage, a new DistGrid is created at each stagger location. This Distgrid is similar to the DistGrid used to create the Grid, but has an extra set of elements added to hold the index locations for the stagger padding. By default, when the coordinate arrays are created, one extra layer of padding is added to the index space to create symmetric staggers (i.e. the center location is surrounded). The default is to add this padding on the positive side, and to only add this padding where needed (e.g. no padding for the center, padding on both dimensions for the corner, in only one dimension for the edge in 2D.) There are two ways for the user to change these defaults.

One way is to use the GridEdgeWidth or GridAlign arguments when creating a Grid. These arguments can be used to change the default padding around the Grid cell index space. This extra padding is used by default when setting the padding for a stagger location.

The gridEdgeLWidth and gridEdgeUWidth arguments are both 1D arrays of the same size as the Grid dimension. The entries in the arrays give the extra offset from the outer boundary of the grid cell index space. The following example shows the creation of a Grid with all the extra space to hold stagger padding on the negative side of a Grid. This is the reverse of the default behavior. The resulting Grid will have an exclusive region which extends from $(-1,-1)$ to $(10,10)$, however, the cell center stagger location will still extend from $(1,1)$ to $(10,10)$.

   grid2D=ESMF_GridCreateNoPeriDim(minIndex=(/1,1/),maxIndex=(/10,10/), &
            gridEdgeLWidth=(/1,1/), gridEdgeUWidth=(/0,0/), rc=rc)

To indicate how the data in a Grid's stagger locations are aligned with the cell centers, the optional gridAlign parameter may be used. This parameter indicates which stagger elements in a cell share the same index values as the cell center. For example, in a 2D cell, it would indicate which of the four corners has the same index value as the center. To set gridAlign, the values -1,+1 are used to indicate the alignment in each dimension. This parameter is mostly informational, however, if the gridEdgeWidth parameters are not set then its value determines where the default padding is placed. If not specified, then the default is to align all staggers to the most negative, so the padding is on the positive side. The following code illustrates creating a Grid aligned to the reverse of default (with everything to the positive side). This creates a Grid identical to that created in the previous example.

   grid2D=ESMF_GridCreateNoPeriDim(minIndex=(/1,1/),maxIndex=(/10,10/), &
            gridAlign=(/1,1/), rc=rc)

The gridEdgeWidth and gridAlign arguments both allow the user to set the default padding to be used by stagger locations in a Grid. By default, stagger locations allocated in a Grid set their stagger padding based on these values. A stagger location's padding in each dimension is equal to the value of gridEdgeWidth (or the value implied by gridAlign), unless the stagger location is centered in a dimension in which case the stagger padding is 0. For example, the cell center stagger location has 0 stagger padding in all dimensions, whereas the edge stagger location lower padding is equal to gridEdgeLWidth and the upper padding is equal to gridEdgeUWidth in one dimension, but both are 0 in the other, centered, dimension. If the user wishes to set the stagger padding individually for each stagger location they may use the staggerEdgeWidth and staggerAlign arguments.

The staggerEdgeLWidth and staggerEdgeUWidth arguments are both 1D arrays of the same size as the Grid dimension. The entries in the arrays give the extra offset from the Grid cell index space for a stagger location. The following example shows the addition of two stagger locations. The corner location has no extra boundary and the center has a single layer of extra padding on the negative side and none on the positive. This is the reverse of the default behavior.

   grid2D=ESMF_GridCreate(distgrid=distgrid2D, &
            gridEdgeLWidth=(/1,1/), gridEdgeUWidth=(/0,0/), rc=rc)

   call ESMF_GridAddCoord(grid2D, &
          staggerLoc=ESMF_STAGGERLOC_CORNER, &
          staggerEdgeLWidth=(/0,0/), staggerEdgeUWidth=(/0,0/), rc=rc)

   call ESMF_GridAddCoord(grid2D, &
          staggerLoc=ESMF_STAGGERLOC_CENTER, &
          staggerEdgeLWidth=(/1,1/), staggerEdgeUWidth=(/0,0/), rc=rc)

To indicate how the data at a particular stagger location is aligned with the cell center, the optional staggerAlign parameter may be used. This parameter indicates which stagger elements in a cell share the same index values as the cell center. For example, in a 2D cell, it would indicate which of the four corners has the same index value as the center. To set staggerAlign, the values -1,+1 are used to indicate the alignment in each dimension. If a stagger location is centered in a dimension (e.g. an edge in 2D), then that dimension is ignored in the alignment. This parameter is mostly informational, however, if the staggerEdgeWidth parameters are not set then its value determines where the default padding is placed. If not specified, then the default is to align all staggers to the most negative, so the padding is on the positive side. The following code illustrates aligning the positive (northeast in 2D) corner with the center.

   call ESMF_GridAddCoord(grid2D, &
          staggerLoc=ESMF_STAGGERLOC_CORNER, staggerAlign=(/1,1/), rc=rc)

31.4 Restrictions and Future Work

31.5 Design and Implementation Notes

31.5.1 Grid Topology

The ESMF_Grid class depends upon the ESMF_DistGrid class for the specification of its topology. That is, when creating a Grid, first an ESMF_DistGrid is created to describe the appropriate index space topology. This decision was made because it seemed redundant to have a system for doing this in both classes. It also seems most appropriate for the machinary for topology creation to be located at the lowest level possible so that it can be used by other classes (e.g. the ESMF_Array class). Because of this, however, the authors recommend that as a natural part of the implementation of subroutines to generate standard grid shapes (e.g. ESMF_GridGenSphere) a set of standard topology generation subroutines be implemented (e.g. ESMF_DistGridGenSphere) for users who want to create a standard topology, but a custom geometry.

31.6 Class API: General Grid Methods

31.6.1 ESMF_GridAssignment(=) - Grid assignment


INTERFACE:

     interface assignment(=)
     grid1 = grid2
ARGUMENTS:
     type(ESMF_Grid) :: grid1
     type(ESMF_Grid) :: grid2
STATUS:

DESCRIPTION:

Assign grid1 as an alias to the same ESMF Grid object in memory as grid2. If grid2 is invalid, then grid1 will be equally invalid after the assignment.

The arguments are:

grid1
The ESMF_Grid object on the left hand side of the assignment.
grid2
The ESMF_Grid object on the right hand side of the assignment.

31.6.2 ESMF_GridOperator(==) - Grid equality operator


INTERFACE:

   interface operator(==)
     if (grid1 == grid2) then ... endif
               OR
     result = (grid1 == grid2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_Grid), intent(in) :: grid1
     type(ESMF_Grid), intent(in) :: grid2
STATUS:

DESCRIPTION:

Test whether grid1 and grid2 are valid aliases to the same ESMF Grid object in memory. For a more general comparison of two ESMF Grids, going beyond the simple alias test, the ESMF_GridMatch() function must be used.

The arguments are:

grid1
The ESMF_Grid object on the left hand side of the equality operation.
grid2
The ESMF_Grid object on the right hand side of the equality operation.

31.6.3 ESMF_GridOperator(/=) - Grid not equal operator


INTERFACE:

   interface operator(/=)
     if (grid1 /= grid2) then ... endif
               OR
     result = (grid1 /= grid2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_Grid), intent(in) :: grid1
     type(ESMF_Grid), intent(in) :: grid2
STATUS:

DESCRIPTION:

Test whether grid1 and grid2 are not valid aliases to the same ESMF Grid object in memory. For a more general comparison of two ESMF Grids, going beyond the simple alias test, the ESMF_GridMatch() function (not yet fully implemented) must be used.

The arguments are:

grid1
The ESMF_Grid object on the left hand side of the non-equality operation.
grid2
The ESMF_Grid object on the right hand side of the non-equality operation.

31.6.4 ESMF_GridAddCoord - Allocate coordinate arrays but don't set their values


INTERFACE:

   ! Private name; call using ESMF_GridAddCoord()
      subroutine ESMF_GridAddCoordNoValues(grid, staggerloc,  &
        staggerEdgeLWidth, staggerEdgeUWidth, staggerAlign, &
        staggerLBound,rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       integer,                intent(in),  optional :: staggerEdgeLWidth(:)
       integer,                intent(in),  optional :: staggerEdgeUWidth(:)
       integer,                intent(in),  optional :: staggerAlign(:)
       integer,                intent(in),  optional :: staggerLBound(:)
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

When a Grid is created all of its potential stagger locations can hold coordinate data, but none of them have storage allocated. This call allocates coordinate storage (creates internal ESMF_Arrays and associated memory) for a particular stagger location. Note that this call doesn't assign any values to the storage, it only allocates it. The remaining options staggerEdgeLWidth, etc. allow the user to adjust the padding on the coordinate arrays.

The arguments are:

grid
Grid to allocate coordinate storage in.
[staggerloc]
The stagger location to add. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[staggerEdgeLWidth]
This array should be the same dimCount as the grid. It specifies the lower corner of the stagger region with respect to the lower corner of the exclusive region.
[staggerEdgeUWidth]
This array should be the same dimCount as the grid. It specifies the upper corner of the stagger region with respect to the upper corner of the exclusive region.
[staggerAlign]
This array is of size grid dimCount. For this stagger location, it specifies which element has the same index value as the center. For example, for a 2D cell with corner stagger it specifies which of the 4 corners has the same index as the center. If this is set and either staggerEdgeUWidth or staggerEdgeLWidth is not, this determines the default array padding for a stagger. If not set, then this defaults to all negative. (e.g. The most negative part of the stagger in a cell is aligned with the center and the padding is all on the positive side.)
[staggerLBound]
Specifies the lower index range of the memory of every DE in this staggerloc in this Grid. Only used when Grid indexflag is ESMF_INDEX_USER.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.5 ESMF_GridAddItem - Allocate item array but don't set their values


INTERFACE:

   ! Private name; call using ESMF_GridAddItem()
      subroutine ESMF_GridAddItemNoValues(grid, itemflag,  &
        staggerloc, itemTypeKind, staggerEdgeLWidth, staggerEdgeUWidth, &
        staggerAlign, staggerLBound,rc)
ARGUMENTS:
       type(ESMF_Grid),          intent(in)           :: grid
       type (ESMF_GridItem_Flag),intent(in)           :: itemflag
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc)  , intent(in), optional :: staggerloc
       type (ESMF_TypeKind_Flag),intent(in), optional :: itemTypeKind
       integer,                  intent(in), optional :: staggerEdgeLWidth(:)
       integer,                  intent(in), optional :: staggerEdgeUWidth(:)
       integer,                  intent(in), optional :: staggerAlign(:)
       integer,                  intent(in), optional :: staggerLBound(:)
       integer,                  intent(out),optional :: rc
STATUS:

DESCRIPTION:

When a Grid is created all of its potential stagger locations can hold item data, but none of them have storage allocated. This call allocates item storage (creates an internal ESMF_Array and associated memory) for a particular stagger location. Note that this call doesn't assign any values to the storage, it only allocates it. The remaining options staggerEdgeLWidth, etc. allow the user to adjust the padding on the item array.

The arguments are:

grid
Grid to allocate coordinate storage in.
itemflag
The grid item to add. Please see Section 31.2.2 for a list of valid items.
[staggerloc]
The stagger location to add. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[itemTypeKind]
The typekind of the item to add.
[staggerEdgeLWidth]
This array should be the same dimCount as the grid. It specifies the lower corner of the stagger region with respect to the lower corner of the exclusive region.
[staggerEdgeUWidth]
This array should be the same dimCount as the grid. It specifies the upper corner of the stagger region with respect to the upper corner of the exclusive region.
[staggerAlign]
This array is of size grid dimCount. For this stagger location, it specifies which element has the same index value as the center. For example, for a 2D cell with corner stagger it specifies which of the 4 corners has the same index as the center. If this is set and either staggerEdgeUWidth or staggerEdgeLWidth is not, this determines the default array padding for a stagger. If not set, then this defaults to all negative. (e.g. The most negative part of the stagger in a cell is aligned with the center and the padding is all on the positive side.)
[staggerLBound]
Specifies the lower index range of the memory of every DE in this staggerloc in this Grid. Only used when Grid indexflag is ESMF_INDEX_USER.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.6 ESMF_GridCreate - Create a copy of a Grid with a new DistGrid


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateCopyFromNewDG(grid, distgrid, &
         name, copyAttributes, routehandle, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateCopyFromNewDG
ARGUMENTS:
        type(ESMF_Grid),       intent(in)              :: grid
        type(ESMF_DistGrid),   intent(in)              :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        character (len=*),     intent(in),   optional  :: name
        logical,               intent(in),   optional  :: copyAttributes
        type(ESMF_RouteHandle),intent(out),  optional  :: routehandle
        integer,               intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

This call allows the user to copy an existing ESMF Grid, but with a new distribution. All internal data from the old Grid (coords, items) are redistributed to the new Grid.

The arguments are:

grid
The existing ESMF_Grid being redistributed, i.e. the "source" grid.
distgrid
ESMF_DistGrid object which describes how the newly created Grid is decomposed and distributed.
[name]
Name of the new Grid. If not specified, a new unique name will be created for the Grid.
[copyAttributes]
A flag to indicate whether to copy the attributes of the existing grid to the new grid. The default value is .false..
[routehandle]
If provided holds the mapping of coordinates between the two grids. This can be used in the companion method ESMF_GridRedist() to update coordinates.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.7 ESMF_GridCreate - Create a copy of a Grid with a different regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateCopyFromReg(grid, &
         regDecomp, decompFlag, name, copyAttributes, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateCopyFromReg
ARGUMENTS:
        type(ESMF_Grid),        intent(in)              :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                intent(in),   optional  :: regDecomp(:)
        type(ESMF_Decomp_Flag), intent(in),   optional  :: decompflag(:)
        character (len=*),      intent(in),   optional  :: name
        logical,                intent(in),   optional  :: copyAttributes
        integer,                intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

This method creates a copy of an existing Grid, the new Grid is regularly distributed (see Figure 13). To specify the new distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible.

The arguments are:

grid
ESMF_Grid to copy.
[regDecomp]
List that has the same number of elements as maxIndex. Each entry is the number of decounts for that dimension. If not specified, the default decomposition will be petCountx1x1..x1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[name]
Name of the new Grid. If not specified, a new unique name will be created for the Grid.
[copyAttributes]
A flag to indicate whether to copy the attributes of the existing grid to the new grid. The default value is .false..
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.8 ESMF_GridCreate - Create a Grid with user set edge connections and an irregular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateEdgeConnI(minIndex,         &
         countsPerDEDim1,countsPerDeDim2,                  &
         countsPerDEDim3,                                  &
         connflagDim1, connflagDim2, connflagDim3,         &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,        &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateEdgeConnI
ARGUMENTS:
        integer,                  intent(in),  optional :: minIndex(:)
        integer,                  intent(in)            :: countsPerDEDim1(:)
        integer,                  intent(in)            :: countsPerDEDim2(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                  intent(in),  optional :: countsPerDEDim3(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim1(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim2(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim3(:)
        type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),  optional :: coordTypeKind
        integer,                  intent(in),  optional :: coordDep1(:)
        integer,                  intent(in),  optional :: coordDep2(:)
        integer,                  intent(in),  optional :: coordDep3(:)
        integer,                  intent(in),  optional :: gridEdgeLWidth(:)
        integer,                  intent(in),  optional :: gridEdgeUWidth(:)
        integer,                  intent(in),  optional :: gridAlign(:)
        integer,                  intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),    intent(in),  optional :: indexflag
        integer,                  intent(in),  optional :: petMap(:,:,:)
        character (len=*),        intent(in),  optional :: name
        integer,                  intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, irregularly distributed grid (see Figure 13). To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Currently this call only supports creating 2D or 3D Grids. A 2D Grid can be specified using the countsPerDEDim1 and countsPerDEDim2 arguments. A 3D Grid can be specified by also using the optional countsPerDEDim3 argument. The index of each array element in these arguments corresponds to a DE number. The array value at the index is the number of grid cells on the DE in that dimension.

Section 31.3.4 shows an example of using this method to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
countsPerDEDim1
This arrays specifies the number of cells per DE for index dimension 1 for the exclusive region (the center stagger location).
countsPerDEDim2
This array specifies the number of cells per DE for index dimension 2 for the exclusive region (center stagger location).
[countsPerDEDim3]
This array specifies the number of cells per DE for index dimension 3 for the exclusive region (center stagger location). If not specified then grid is 2D.
[connflagDim1]
Fortran array describing the index dimension 1 connections. The first element represents the minimum end of dimension 1. The second element represents the maximum end of dimension 1. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connflagDim2]
Fortran array describing the index dimension 2 connections. The first element represents the minimum end of dimension 2. The second element represents the maximum end of dimension 2. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connflagDim3]
Fortran array describing the index dimension 3 connections. The first element represents the minimum end of dimension 3. The second element represents the maximum end of dimension 3. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size size(countsPerDEDim1) x size(countsPerDEDim2) x size(countsPerDEDim3). If countsPerDEDim3 isn't present, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.9 ESMF_GridCreate - Create a Grid with user set edge connections and a regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateEdgeConnR(regDecomp, decompFlag, &
         minIndex, maxIndex,                                    &
         connflagDim1, connflagDim2, connflagDim3,                       &
         coordSys, coordTypeKind,                            &
         coordDep1, coordDep2, coordDep3,                    &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,          &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateEdgeConnR
ARGUMENTS:
        integer,                  intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),   intent(in),  optional :: decompflag(:)
        integer,                  intent(in),  optional :: minIndex(:)
        integer,                  intent(in)            :: maxIndex(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim1(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim2(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim3(:)
        type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),  optional :: coordTypeKind
        integer,                  intent(in),  optional :: coordDep1(:)
        integer,                  intent(in),  optional :: coordDep2(:)
        integer,                  intent(in),  optional :: coordDep3(:)
        integer,                  intent(in),  optional :: gridEdgeLWidth(:)
        integer,                  intent(in),  optional :: gridEdgeUWidth(:)
        integer,                  intent(in),  optional :: gridAlign(:)
        integer,                  intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),    intent(in),  optional :: indexflag
        integer,                  intent(in),  optional :: petMap(:,:,:)
        character (len=*),        intent(in),  optional :: name
        integer,                  intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13). To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[regDecomp]
List that has the same number of elements as maxIndex. Each entry is the number of decounts for that dimension. If not specified, the default decomposition will be petCountx1x1..x1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
[connflagDim1]
Fortran array describing the index dimension 1 connections. The first element represents the minimum end of dimension 1. The second element represents the maximum end of dimension 1. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connflagDim2]
Fortran array describing the index dimension 2 connections. The first element represents the minimum end of dimension 2. The second element represents the maximum end of dimension 2. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connflagDim3]
Fortran array describing the index dimension 3 connections. The first element represents the minimum end of dimension 3. The second element represents the maximum end of dimension 3. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.10 ESMF_GridCreate - Create a Grid with user set edge connections and an arbitrary distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateEdgeConnA(minIndex, maxIndex,  &
         arbIndexCount, arbIndexList,                         &
         connflagDim1, connflagDim2, connflagDim3,                     &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         distDim, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateEdgeConnA
ARGUMENTS:
        integer,                  intent(in),  optional :: minIndex(:)
        integer,                  intent(in)            :: maxIndex(:)
        integer,                  intent(in)            :: arbIndexCount
        integer,                  intent(in)            :: arbIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim1(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim2(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connflagDim3(:)
        type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),  optional :: coordTypeKind
        integer,                  intent(in),  optional :: coordDep1(:)
        integer,                  intent(in),  optional :: coordDep2(:)
        integer,                  intent(in),  optional :: coordDep3(:)
        integer,                  intent(in),  optional :: distDim(:)
        character (len=*),        intent(in),  optional :: name
        integer,                  intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, arbitrarily distributed grid (see Figure 13). To specify the arbitrary distribution, the user passes in an 2D array of local indices, where the first dimension is the number of local grid cells specified by localArbIndexCount and the second dimension is the number of distributed dimensions.

distDim specifies which grid dimensions are arbitrarily distributed. The size of distDim has to agree with the size of the second dimension of localArbIndex.

Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
maxIndex
The upper extend of the grid index ranges.
arbIndexCount
The number of grid cells in the local DE. It is okay to have 0 grid cell in a local DE.
arbIndexList
This 2D array specifies the indices of the PET LOCAL grid cells. The dimensions should be arbIndexCount * number of Distributed grid dimensions where arbIndexCount is the input argument specified below
[connflagDim1]
Fortran array describing the index dimension 1 connections. The first element represents the minimum end of dimension 1. The second element represents the maximum end of dimension 1. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connflagDim2]
Fortran array describing the index dimension 2 connections. The first element represents the minimum end of dimension 2. The second element represents the maximum end of dimension 2. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connflagDim3]
Fortran array describing the index dimension 3 connections. The first element represents the minimum end of dimension 3. The second element represents the maximum end of dimension 3. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if the first dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=1) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep2]
The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=2) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep3]
The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=3) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[distDim]
This array specifies which dimensions are arbitrarily distributed. The size of the array specifies the total distributed dimensions. if not specified, defaults is all dimensions will be arbitrarily distributed. The size has to agree with the size of the second dimension of localArbIndex.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.11 ESMF_GridCreate - Create a Grid from a DistGrid


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateFrmDistGrid(distgrid, &
         distgridToGridMap, &
         coordSys, coordTypeKind, coordDimCount, coordDimMap, &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign, &
         gridMemLBound, indexflag, name, vm, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateFrmDistGrid
ARGUMENTS:
        type(ESMF_DistGrid),     intent(in)            :: distgrid
        integer,                 intent(in),  optional :: distgridToGridMap(:)
        type(ESMF_CoordSys_Flag),intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),intent(in),  optional :: coordTypeKind
        integer,                 intent(in),  optional :: coordDimCount(:)
        integer,                 intent(in),  optional :: coordDimMap(:,:)
        integer,                 intent(in),  optional :: gridEdgeLWidth(:)
        integer,                 intent(in),  optional :: gridEdgeUWidth(:)
        integer,                 intent(in),  optional :: gridAlign(:)
        integer,                 intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),   intent(in),  optional :: indexflag
        character (len=*),       intent(in),  optional :: name
        type(ESMF_VM),           intent(in),  optional :: vm
        integer,                 intent(out), optional :: rc
DESCRIPTION:

This is the most general form of creation for an ESMF_Grid object. It allows the user to fully specify the topology and index space using the DistGrid methods and then build a grid out of the resulting DistGrid. Note that since the Grid created by this call uses distgrid as a description of its index space, the resulting Grid will have exactly the same number of dimensions (i.e. the same dimCount) as distgrid. The distgridToGridMap argument specifies how the Grid dimensions are mapped to the distgrid. The coordDimCount and coordDimMap arguments allow the user to specify how the coordinate arrays should map to the grid dimensions. (Note, though, that creating a grid does not allocate coordinate storage. A method such as ESMF_GridAddCoord() must be called before adding coordinate values.)

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs.
[distgridToGridMap]
List that has dimCount elements. The elements map each dimension of distgrid to a dimension in the grid. (i.e. the values should range from 1 to dimCount). If not specified, the default is to map all of distgrid's dimensions against the dimensions of the grid in sequence.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDimCount]
List that has dimCount elements. Gives the dimension of each component (e.g. x) array. This is to allow factorization of the coordinate arrays. If not specified all arrays are the same size as the grid.
[coordDimMap]
2D list of size dimCount x dimCount. This array describes the map of each component array's dimensions onto the grids dimensions. Each entry coordDimMap(i,j) tells which grid dimension component i's, jth dimension maps to. Note that if j is bigger than coordDimCount(i) it is ignored. The default for each row i is coordDimMap(i,:)=(1,2,3,4,...).
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[name]
ESMF_Grid name.
[vm]
If present, the Grid object is created on the specified ESMF_VM object. The default is to create on the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.12 ESMF_GridCreate - Create a Arbitrary Grid from a DistGrid


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
       function ESMF_GridCreateFrmDistGridArb(distgrid, &
         indexArray, distDim, &
         coordSys, coordTypeKind, coordDimCount, coordDimMap, &
         name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateFrmDistGridArb
ARGUMENTS:
        type(ESMF_DistGrid),      intent(in)              :: distgrid
        integer,                  intent(in)              :: indexArray(:,:)
        integer,                  intent(in),   optional  :: distDim(:)
        type(ESMF_CoordSys_Flag), intent(in),   optional  :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),   optional  :: coordTypeKind
        integer,                  intent(in),   optional  :: coordDimCount(:)
        integer,                  intent(in),   optional  :: coordDimMap(:,:)
        character (len=*),        intent(in),   optional  :: name
        integer,                  intent(out),  optional  :: rc
DESCRIPTION:

This is the lower level function to create an arbitrarily distributed ESMF_Grid object. It allows the user to fully specify the topology and index space (of the distributed dimensions) using the DistGrid methods and then build a grid out of the resulting distgrid. The indexArray(2,dimCount), argument is required to specifies the topology of the grid.

The arguments are:

distgrid
ESMF_DistGrid object that describes how the array is decomposed and distributed over DEs.
indexArray
The minIndex and maxIndex array of size 2 x dimCount indexArray(1,:) is the minIndex and indexArray(2,:) is the maxIndex
[distDim]
This array specifies which dimensions are arbitrarily distributed. The size of the array specifies the total distributed dimensions. if not specified, the default is that all dimensions will be arbitrarily distributed.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDimCount]
List that has dimCount elements. Gives the dimension of each component (e.g. x) array. This is to allow factorization of the coordinate arrays. If not specified each component is assumed to be size 1. Note, the default value is different from the same argument for a non-arbitrarily distributed grid.
[coordDimMap]
2D list of size dimCount x dimCount. This array describes the map of each coordinate array's dimensions onto the grids dimensions. coordDimMap(i,j) is the grid dimension of the jth dimension of the i'th coordinate array. If not specified, the default value of coordDimMap(i,1) is /ESMF_DIM_ARB/ if the ith dimension of the grid is arbitrarily distributed, or i if the ith dimension is not distributed. Note that if j is bigger than coordDimCount(i) then it's ignored. Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.13 ESMF_GridCreate - Create a Grid from a SCRIP or GRIDSPEC format grid file with a user specified distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
      function ESMF_GridCreateFrmNCFileDG(filename, fileformat, distgrid, &
        isSphere, polekindflag, addCornerStagger, coordTypeKind, addUserArea, indexflag, &
        addMask, varname, coordNames, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateFrmNCFileDG
ARGUMENTS:
 
     character(len=*),       intent(in)             :: filename
     type(ESMF_FileFormat_Flag), intent(in), optional :: fileformat
     type(ESMF_DistGrid),    intent(in)             :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),  optional  :: isSphere
     type(ESMF_PoleKind_Flag),  intent(in),  optional :: polekindflag(2)
     logical,                intent(in),  optional  :: addCornerStagger
     type(ESMF_TypeKind_Flag),intent(in), optional  :: coordTypeKind
     logical,                intent(in),  optional  :: addUserArea
     type(ESMF_Index_Flag),  intent(in),  optional  :: indexflag
     logical,                intent(in),  optional  :: addMask
     character(len=*),       intent(in),  optional  :: varname
     character(len=*),       intent(in),  optional  :: coordNames(:)
     integer,                intent(out), optional  :: rc
DESCRIPTION:

This function creates a ESMF_Grid object using the grid definition from a grid file in NetCDF that is either in the SCRIP format or in the CF convention. To specify the distribution, the user passes in a distGrid. The grid defined in the file has to be a 2D logically rectangular grid. This function first call ESMF_GridCreateFrmNCFile() to create a ESMF_Grid object using a pre-calculated block distribution, then redistribute the Grid to create a new Grid object using the user specified distGrid.

This call is collective across the current VM.

The arguments are:

filename
The NetCDF Grid filename.
[fileformat]
The file format. The valid options are ESMF_FILEFORMAT_SCRIP and ESMF_FILEFORMAT_GRIDSPEC. If it is the SCRIP format, the dimension grid_rank in the file has to be equal to 2. Please see section 54.20 for a detailed description of the options. If not specified, the file type will be detected automatically.
distGrid
A distGrid defines how the grid is distributed
[isSphere]
If .true., create a periodic Grid. If .false., create a regional Grid. Defaults to .true.
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[addCornerStagger]
Uses the information in the grid file to add the Corner stagger to the Grid. The coordinates for the corner stagger is required for conservative regridding. If not specified, defaults to false.
[coordTypeKind]
The type/kind of the grid coordinate data. Only ESMF_TYPEKIND_R4 and ESMF_TYPEKIND_R8 are allowed. Currently, ESMF_TYPEKIND_R4 is only supported for the GRIDSPEC fileformat. If not specified then defaults to ESMF_TYPEKIND_R8.
[addUserArea]
If .true., read in the cell area from the Grid file, otherwise, ESMF will calculate it. The feature is only supported when the grid file is in the SCRIP format. If not set, the default value is .false.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[addMask]
If .true., generate the mask using the missing_value attribute defined in 'varname'. This flag is only needed for the GRIDSPEC file format. If not set, the default value is .false.
[varname]
If addMask is true, provide a variable name stored in the grid file and the mask will be generated using the missing value of the data value of this variable. The first two dimensions of the variable has to be the the longitude and the latitude dimension and the mask is derived from the first 2D values of this variable even if this data is 3D, or 4D array.
[coordNames]
a two-element array containing the longitude and latitude variable names in a GRIDSPEC file if there are multiple coordinates defined in the file
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.14 ESMF_GridCreate - Create a Grid from a SCRIP or GRIDSPEC format grid file


INTERFACE:

   ! Private name; call using ESMF_GridCreate()
      function ESMF_GridCreateFrmNCFile(filename, fileformat, regDecomp, &
        decompflag, delayout, isSphere, polekindflag, addCornerStagger, coordTypeKind, &
        addUserArea, indexflag, addMask, varname, coordNames, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateFrmNCFile
ARGUMENTS:
 
     character(len=*),       intent(in)                :: filename
     type(ESMF_FileFormat_Flag), intent(in), optional  :: fileformat
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),  optional     :: regDecomp(:)
     type(ESMF_Decomp_Flag), intent(in),  optional     :: decompflag(:)
     type(ESMF_DELayout),    intent(in),  optional     :: delayout
     logical,                intent(in),  optional     :: isSphere
     type(ESMF_PoleKind_Flag),  intent(in),  optional  :: polekindflag(2)
     logical,                intent(in),  optional     :: addCornerStagger 
     type(ESMF_TypeKind_Flag),intent(in), optional     :: coordTypeKind
     logical,                intent(in),  optional     :: addUserArea
     type(ESMF_Index_Flag),  intent(in),  optional     :: indexflag
     logical,                intent(in),  optional     :: addMask
     character(len=*),       intent(in),  optional     :: varname
     character(len=*),       intent(in),  optional     :: coordNames(:)
     integer,                intent(out), optional     :: rc
DESCRIPTION:

This function creates a ESMF_Grid object using the grid definition from a grid file in NetCDF that is either in the SCRIP format or in the CF convention. To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompflag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. The grid defined in the file has to be a 2D logically rectangular grid.

This call is collective across the current VM.

The arguments are:

filename
The NetCDF Grid filename.
[fileformat]
The file format. The valid options are ESMF_FILEFORMAT_SCRIP and ESMF_FILEFORMAT_GRIDSPEC. If it is the SCRIP format, the dimension grid_rank in the file has to be equal to 2. Please see section 54.20 for a detailed description of the options. If not specified, the filetype will be automatically detected.
[regDecomp]
A 2 element array specifying how the grid is decomposed. Each entry is the number of decounts for that dimension. The total decounts cannot exceed the total number of PETs. In other word, at most one DE is allowed per processor. If not specified, the default decomposition will be petCountx1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[delayout]
The DELayout that determines DE layout of DEs across PETs. The default is to create a default DELayout with the correct number of DEs according to the regDecomp. See the documentation of the ESMF_DELayoutCreate() method for details about the default DELayout.
[isSphere]
If .true., create a periodic Grid. If .false., create a regional Grid. Defaults to .true.
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[addCornerStagger]
Uses the information in the grid file to add the Corner stagger to the Grid. The coordinates for the corner stagger is required for conservative regridding. If not specified, defaults to false.
[coordTypeKind]
The type/kind of the grid coordinate data. Only ESMF_TYPEKIND_R4 and ESMF_TYPEKIND_R8 are allowed. Currently, ESMF_TYPEKIND_R4 is only supported for the GRIDSPEC fileformat. If not specified then defaults to ESMF_TYPEKIND_R8.
[addUserArea]
If .true., read in the cell area from the Grid file, otherwise, ESMF will calculate it. The feature is only supported when the grid file is in the SCRIP format. If not set, the default value is .false.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[addMask]
If .true., generate the mask using the missing_value attribute defined in 'varname'. This flag is only needed for the GRIDSPEC file format. If not set, the default value is .false.
[varname]
If addMask is true, provide a variable name stored in the grid file and the mask will be generated using the missing value of the data value of this variable. The first two dimensions of the variable has to be the the longitude and the latitude dimension and the mask is derived from the first 2D values of this variable even if this data is 3D, or 4D array.
[coordNames]
a two-element array containing the longitude and latitude variable names in a GRIDSPEC file if there are multiple coordinates defined in the file
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.15 ESMF_GridCreate1PeriDim - Create a Grid with one periodic dim and an irregular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate1PeriDim()
       function ESMF_GridCreate1PeriDimI(minIndex,         &
         countsPerDEDim1,countsPerDeDim2,                  &
         countsPerDEDim3,                                  &
         polekindflag, periodicDim, poleDim,                   &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,        &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreate1PeriDimI
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: countsPerDEDim1(:)
        integer,                   intent(in)            :: countsPerDEDim2(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                   intent(in),  optional :: countsPerDEDim3(:)
        type(ESMF_PoleKind_Flag),  intent(in),  optional :: polekindflag(2)
        integer,                   intent(in),  optional :: periodicDim
        integer,                   intent(in),  optional :: poleDim
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: gridEdgeLWidth(:)
        integer,                   intent(in),  optional :: gridEdgeUWidth(:)
        integer,                   intent(in),  optional :: gridAlign(:)
        integer,                   intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),     intent(in),  optional :: indexflag
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, irregularly distributed grid (see Figure 13) with one periodic dimension. To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Currently this call only supports creating 2D or 3D Grids. A 2D Grid can be specified using the countsPerDEDim1 and countsPerDEDim2 arguments. A 3D Grid can be specified by also using the optional countsPerDEDim3 argument. The index of each array element in these arguments corresponds to a DE number. The array value at the index is the number of grid cells on the DE in that dimension.

Section 31.3.4 shows an example of using this method to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
countsPerDEDim1
This arrays specifies the number of cells per DE for index dimension 1 for the exclusive region (the center stagger location).
countsPerDEDim2
This array specifies the number of cells per DE for index dimension 2 for the exclusive region (center stagger location).
[countsPerDEDim3]
This array specifies the number of cells per DE for index dimension 3 for the exclusive region (center stagger location). If not specified then grid is 2D.
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[periodicDim]
The periodic dimension. If not specified, defaults to 1.
[poleDim]
The dimension at who's ends the poles are located. If not specified defaults to 2.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size size(countsPerDEDim1) x size(countsPerDEDim2) x size(countsPerDEDim3). If countsPerDEDim3 isn't present, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.16 ESMF_GridCreate1PeriDim - Create a Grid with one periodic dim and a regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate1PeriDim()
       function ESMF_GridCreate1PeriDimR(regDecomp, decompFlag, &
         minIndex, maxIndex,                                    &
         polekindflag, periodicDim, poleDim,                        &
         coordSys, coordTypeKind,                               &
         coordDep1, coordDep2, coordDep3,                       &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,             &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreate1PeriDimR
ARGUMENTS:
        integer,                   intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),    intent(in),  optional :: decompflag(:)
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_PoleKind_Flag),  intent(in),  optional :: polekindflag(2)
        integer,                   intent(in),  optional :: periodicDim
        integer,                   intent(in),  optional :: poleDim
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: gridEdgeLWidth(:)
        integer,                   intent(in),  optional :: gridEdgeUWidth(:)
        integer,                   intent(in),  optional :: gridAlign(:)
        integer,                   intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),     intent(in),  optional :: indexflag
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13) with one periodic dimension. To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[regDecomp]
List that has the same number of elements as maxIndex. Each entry is the number of decounts for that dimension. If not specified, the default decomposition will be petCountx1x1..x1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the ! possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[periodicDim]
The periodic dimension. If not specified, defaults to 1.
[poleDim]
The dimension at who's ends the poles are located. If not specified defaults to 2.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.17 ESMF_GridCreate1PeriDim - Create a Grid with one periodic dim and an arbitrary distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate1PeriDim()
       function ESMF_GridCreate1PeriDimA(minIndex, maxIndex,  &
         arbIndexCount, arbIndexList,                         &
         polekindflag, periodicDim, poleDim,                      &
         coordSys, coordTypeKind,                             &
         coordDep1, coordDep2, coordDep3,                     &
         distDim, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreate1PeriDimA
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
         integer,                   intent(in)            :: maxIndex(:)
        integer,                   intent(in)                    :: arbIndexCount
        integer,                   intent(in)            :: arbIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_PoleKind_Flag),  intent(in),  optional :: polekindflag(2)
        integer,                   intent(in),  optional :: periodicDim
        integer,                   intent(in),  optional :: poleDim
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: distDim(:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, arbitrarily distributed grid (see Figure 13) with one periodic dimension. To specify the arbitrary distribution, the user passes in an 2D array of local indices, where the first dimension is the number of local grid cells specified by localArbIndexCount and the second dimension is the number of distributed dimensions.

distDim specifies which grid dimensions are arbitrarily distributed. The size of distDim has to agree with the size of the second dimension of localArbIndex.

Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
maxIndex
The upper extend of the grid index ranges.
arbIndexCount
The number of grid cells in the local DE. It is okay to have 0 grid cell in a local DE.
arbIndexList
This 2D array specifies the indices of the PET LOCAL grid cells. The dimensions should be arbIndexCount * number of Distributed grid dimensions where arbIndexCount is the input argument specified below
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[periodicDim]
The periodic dimension. If not specified, defaults to 1.
[poleDim]
The dimension at who's ends the poles are located. If not specified defaults to 2.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if the first dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=1) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep2]
The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=2) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep3]
The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate ! arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=3) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[distDim]
This array specifies which dimensions are arbitrarily distributed. The size of the array specifies the total distributed dimensions. if not specified, defaults is all dimensions will be arbitrarily distributed. The size has to agree with the size of the second dimension of localArbIndex.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.18 ESMF_GridCreate2PeriDim - Create a Grid with two periodic dims and an irregular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate2PeriDim()
       function ESMF_GridCreate2PeriDimI(minIndex,         &
         countsPerDEDim1,countsPerDeDim2,                  &
         countsPerDEDim3,                                  &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,        &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreate2PeriDimI
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: countsPerDEDim1(:)
        integer,                   intent(in)            :: countsPerDEDim2(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                   intent(in),  optional :: countsPerDEDim3(:)
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: gridEdgeLWidth(:)
        integer,                   intent(in),  optional :: gridEdgeUWidth(:)
        integer,                   intent(in),  optional :: gridAlign(:)
        integer,                   intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),     intent(in),  optional :: indexflag
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, irregularly distributed grid (see Figure 13) with two periodic dimensions. To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Currently this call only supports creating 2D or 3D Grids. A 2D Grid can be specified using the countsPerDEDim1 and countsPerDEDim2 arguments. A 3D Grid can be specified by also using the optional countsPerDEDim3 argument. The index of each array element in these arguments corresponds to a DE number. The array value at the index is the number of grid ! cells on the DE in that dimension.

Section 31.3.4 shows an example of using this method to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
countsPerDEDim1
This arrays specifies the number of cells per DE for index dimension 1 for the exclusive region (the center stagger location).
countsPerDEDim2
This array specifies the number of cells per DE for index dimension 2 for the exclusive region (center stagger location).
[countsPerDEDim3]
This array specifies the number of cells per DE for index dimension 3 for the exclusive region (center stagger location). If not specified then grid is 2D.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank. !
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size size(countsPerDEDim1) x size(countsPerDEDim2) x size(countsPerDEDim3). If countsPerDEDim3 isn't present, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
! Return code; equals ESMF_SUCCESS if there are no errors.

31.6.19 ESMF_GridCreate2PeriDim - Create a Grid with two periodic dims and a regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate2PeriDim()
       function ESMF_GridCreate2PeriDimR(regDecomp, decompFlag, &
         minIndex, maxIndex,                                    &
         coordSys, coordTypeKind,                               &
         coordDep1, coordDep2, coordDep3,                       &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,             &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreate2PeriDimR
ARGUMENTS:
        integer,                   intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),    intent(in),  optional :: decompflag(:)
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: gridEdgeLWidth(:)
        integer,                   intent(in),  optional :: gridEdgeUWidth(:)
        integer,                   intent(in),  optional :: gridAlign(:)
        integer,                   intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),     intent(in),  optional :: indexflag
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13) with two periodic dimensions. To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[regDecomp]
List that has the same number of elements as maxIndex. Each entry is the number of decounts for that dimension. If not specified, the default decomposition will be petCountx1x1..x1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.20 ESMF_GridCreate2PeriDim - Create a Grid with two periodic dims and an arbitrary distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate2PeriDim()
       function ESMF_GridCreate2PeriDimA(minIndex, maxIndex, &
         arbIndexCount, arbIndexList,                        &
         coordSys, coordTypeKind,                            &
         coordDep1, coordDep2, coordDep3,                    &
         distDim, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreate2PeriDimA
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
        integer,                   intent(in)                    :: arbIndexCount
         integer,                   intent(in)            :: arbIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: distDim(:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, arbitrarily distributed grid (see Figure 13) with two periodic dimensions. To specify the arbitrary distribution, the user passes in an 2D array of local indices, where the first dimension is the number of local grid cells specified by localArbIndexCount and the second dimension is the number of distributed dimensions.

distDim specifies which grid dimensions are arbitrarily distributed. The size of distDim has to agree with the size of the second dimension of localArbIndex.

Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
maxIndex
The upper extend of the grid index ranges.
arbIndexCount
The number of grid cells in the local DE. It is okay to have 0 grid cell in a local DE.
arbIndexList
This 2D array specifies the indices of the PET LOCAL grid cells. The dimensions should be arbIndexCount * number of Distributed grid dimensions where arbIndexCount is the input argument specified below
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if the first dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=1) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep2]
The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=2) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep3]
The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=3) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[distDim]
This array specifies which dimensions are arbitrarily distributed. The size of the array specifies the total distributed dimensions. if not specified, defaults is all dimensions will be arbitrarily distributed. The size has to agree with the size of the second dimension of localArbIndex.
[name]
ESMF_Grid name. !
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.21 ESMF_GridCreateNoPeriDim - Create a Grid with no periodic dim and an irregular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreateNoPeriDim()
       function ESMF_GridCreateNoPeriDimI(minIndex,        &
         countsPerDEDim1,countsPerDeDim2,                  &
         countsPerDEDim3,                                  &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,        &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateNoPeriDimI
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: countsPerDEDim1(:)
        integer,                   intent(in)            :: countsPerDEDim2(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                   intent(in),  optional :: countsPerDEDim3(:)
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: gridEdgeLWidth(:)
        integer,                   intent(in),  optional :: gridEdgeUWidth(:)
        integer,                   intent(in),  optional :: gridAlign(:)
        integer,                   intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),     intent(in),  optional :: indexflag
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, irregularly distributed grid (see Figure 13) without a periodic dimension. To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Currently this call only supports creating 2D or 3D Grids. A 2D Grid can be specified using the countsPerDEDim1 and countsPerDEDim2 arguments. A 3D Grid can be specified by also using the optional countsPerDEDim3 argument. The index of each array element in these arguments corresponds to a DE number. The array value at the index is the number of grid cells on the DE in that dimension.

Section 31.3.4 shows an example of using this method to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
countsPerDEDim1
This arrays specifies the number of cells per DE for index dimension 1 for the exclusive region (the center stagger location).
countsPerDEDim2
This array specifies the number of cells per DE for index dimension 2 ! for the exclusive region (center stagger location).
[countsPerDEDim3]
This array specifies the number of cells per DE for index dimension 3 for the exclusive region (center stagger location). If not specified then grid is 2D.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank. !
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size size(countsPerDEDim1) x size(countsPerDEDim2) x size(countsPerDEDim3). If countsPerDEDim3 isn't present, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.22 ESMF_GridCreateNoPeriDim - Create a Grid with no periodic dim and a regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreateNoPeriDim()
       function ESMF_GridCreateNoPeriDimR(regDecomp, decompFlag, &
         minIndex, maxIndex,                                     &
         coordSys, coordTypeKind,                                &
         coordDep1, coordDep2, coordDep3,                        &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,              &
         gridMemLBound, indexflag, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateNoPeriDimR
ARGUMENTS:
        integer,                   intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),    intent(in),  optional :: decompflag(:)
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: gridEdgeLWidth(:)
        integer,                   intent(in),  optional :: gridEdgeUWidth(:)
        integer,                   intent(in),  optional :: gridAlign(:)
        integer,                   intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),     intent(in),  optional :: indexflag
         integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13) with no periodic dimension. To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[regDecomp]
List that has the same number of elements as maxIndex. Each entry is the number of decounts for that dimension. If not specified, the default decomposition will be petCountx1x1..x1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which ! of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D ! should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.23 ESMF_GridCreateNoPeriDim - Create a Grid with no periodic dim and an arbitrary distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreateNoPeriodic()
       function ESMF_GridCreateNoPeriDimA(minIndex, maxIndex, &
         arbIndexCount, arbIndexList,                         &
         coordSys, coordTypeKind,                             &
         coordDep1, coordDep2, coordDep3,                     &
         distDim, name, rc)
RETURN VALUE:
        type(ESMF_Grid) :: ESMF_GridCreateNoPeriDimA
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
        integer,                   intent(in)                    :: arbIndexCount
        integer,                   intent(in)            :: arbIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag),  intent(in),  optional :: coordTypeKind
        integer,                   intent(in),  optional :: coordDep1(:)
        integer,                   intent(in),  optional :: coordDep2(:)
        integer,                   intent(in),  optional :: coordDep3(:)
        integer,                   intent(in),  optional :: distDim(:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, arbitrarily distributed grid (see Figure 13) with no periodic dimension. To specify the arbitrary distribution, the user passes in an 2D array of local indices, where the first dimension is the number of local grid cells specified by localArbIndexCount and the second dimension is the number of distributed dimensions.

distDim specifies which grid dimensions are arbitrarily distributed. The size of distDim has to agree with the size of the second dimension of localArbIndex.

Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The arguments are:

[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
maxIndex
The upper extend of the grid index ranges.
arbIndexCount
The number of grid cells in the local DE. It is okay to have 0 grid cell in a local DE.
arbIndexList
! This 2D array specifies the indices of the PET LOCAL grid cells. The dimensions should be arbIndexCount * number of Distributed grid dimensions ! where arbIndexCount is the input argument specified below
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if the first dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=1) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep2]
The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=2) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[coordDep3]
The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_DIM_ARB/ where /ESMF_DIM_ARB/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_DIM_ARB/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=3) Please see Section 54.2 for a definition of ESMF_DIM_ARB.
[distDim]
This array specifies which dimensions are arbitrarily distributed. ! The size of the array specifies the total distributed dimensions. if not specified, defaults is all dimensions will be arbitrarily distributed. The size has to agree with the size of the second dimension of localArbIndex.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.24 ESMF_GridCreate1PeriDimUfrm - Create a uniform Grid with one periodic dim and a regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate1PeriDimUfrm()
       function ESMF_GridCreate1PeriDimUfrmR(minIndex, maxIndex, &
            minCornerCoord, maxCornerCoord, &
            regDecomp, decompFlag, &
            polekindflag, coordSys, staggerLocList, &
            ignoreNonPeriCoord, petMap, name, rc)
RETURN VALUE:
        type(ESMF_Grid) :: ESMF_GridCreate1PeriDimUfrmR
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
        real(ESMF_KIND_R8),        intent(in)            :: minCornerCoord(:)
        real(ESMF_KIND_R8),        intent(in)            :: maxCornerCoord(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                   intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),    intent(in),  optional :: decompflag(:)
        type(ESMF_PoleKind_Flag),  intent(in),  optional :: polekindflag(2)
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_StaggerLoc),     intent(in),  optional :: staggerLocList(:)
        logical,                   intent(in),  optional :: ignoreNonPeriCoord
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13) with one periodic dimension. The periodic dimension in the resulting grid will be dimension 1. The dimension with the poles at either end (i.e. the pole dimension) will be dimension 2.

The grid will have its coordinates uniformly spread between the ranges specified by the user. The coordinates are ESMF_TYPEKIND_R8. Currently, this method only fills the center stagger with coordinates, and the minCornerCoord and maxCornerCoord arguments give the boundaries of the center stagger.

To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The following arguments have been set to non-typical values and so there is a reasonable possibility that they may change in the future to be inconsistent with other Grid create interfaces:

The arguments coordDep1, coordDep2, and coordDep3 have internally been set to 1, 2, and 3 respectively. This was done because this call creates a uniform grid and so only 1D arrays are needed to hold the coordinates. This means the coordinate arrays will be 1D.

The argument indexFlag has internally been set to ESMF_INDEX_GLOBAL. This means that the grid created from this function will have a global index space.

The arguments are:

[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
minCornerCoord
The coordinates of the corner of the grid that corresponds to minIndex. size(minCornerCoord) must be equal to size(maxIndex).
maxCornerCoord
The coordinates of the corner of the grid that corresponds to maxIndex. size(maxCornerCoord) must be equal to size(maxIndex).
[regDecomp]
A ndims-element array specifying how the grid is decomposed. Each entry is the number of decounts for that dimension.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[staggerLocList]
The list of stagger locations to fill with coordinates. Please see Section 31.2.6 for a description of the available stagger locations. If not present, then no staggers are added or filled.
[ignoreNonPeriCoord]
If .true., do not check if the coordinates for the periodic dimension (i.e. dim=1) specify a full periodic range (e.g. 0 to 360 degrees). If not specified, defaults to .false. .
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.25 ESMF_GridCreate1PeriDimUfrm - Create a uniform Grid with one periodic dim and a block distribution


INTERFACE:

   ! Private name; call using ESMF_GridCreate1PeriDimUfrm()
   function ESMF_GridCreate1PeriDimUfrmB(minIndex, maxIndex, &
             minCornerCoord, maxCornerCoord, &
             deBlockList, deLabelList, &
             polekindflag, coordSys, staggerLocList, &
             ignoreNonPeriCoord, petMap, name, rc)
RETURN VALUE:
     type(ESMF_Grid) :: ESMF_GridCreate1PeriDimUfrmB
ARGUMENTS:
     integer,                   intent(in),  optional :: minIndex(:)
     integer,                   intent(in)            :: maxIndex(:)
     real(ESMF_KIND_R8),        intent(in)            :: minCornerCoord(:)
     real(ESMF_KIND_R8),        intent(in)            :: maxCornerCoord(:)
     integer,                   intent(in)            :: deBlockList(:,:,:)
     integer,                   intent(in),  optional :: deLabelList(:)
     type(ESMF_PoleKind_Flag),  intent(in),  optional :: polekindflag(2)
     type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
     type(ESMF_StaggerLoc),     intent(in),  optional :: staggerLocList(:)
     logical,                   intent(in),  optional :: ignoreNonPeriCoord
     integer,                   intent(in),  optional :: petMap(:,:,:)
     character (len=*),         intent(in),  optional :: name
     integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13) with one periodic dimension. The periodic dimension in the resulting grid will be dimension 1. The dimension with the poles at either end (i.e. the pole dimension) will be dimension 2.

The grid will have its coordinates uniformly spread between the ranges specified by the user. The coordinates are ESMF_TYPEKIND_R8. Currently, this method only fills the center stagger with coordinates, and the minCornerCoord and maxCornerCoord arguments give the boundaries of the center stagger.

To specify the distribution, the user passes in an array (deBlockList) specifying index space blocks for each DE.

The following arguments have been set to non-typical values and so there is a reasonable possibility that they may change in the future to be inconsistent with other Grid create interfaces:

The arguements coordDep1, coordDep2, and coordDep3 have internally been set to 1, 2, and 3 respectively. This was done because this call creates a uniform grid and so only 1D arrays are needed to hold the coordinates. This means the coordinate arrays will be 1D.

The argument indexFlag has internally been set to ESMF_INDEX_GLOBAL. This means that the grid created from this function will have a global index space.

The arguments are:

[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
minCornerCoord
The coordinates of the corner of the grid that corresponds to minIndex. size(minCornerCoord) must be equal to size(maxIndex).
maxCornerCoord
The coordinates of the corner of the grid that corresponds to maxIndex. size(maxCornerCoord) must be equal to size(maxIndex).
deBlockList
List of DE-local LR blocks. The third index of deBlockList steps through the deBlock elements, which are defined by the first two indices. The first index must be of size dimCount and the second index must be of size 2. Each 2D element of deBlockList defined by the first two indices hold the following information.
                 +---------------------------------------> 2nd index
                 |    1               2
                 | 1  minIndex(1)    maxIndex(1)
                 | 2  minIndex(2)    maxIndex(2)
                 | .  minIndex(.)    maxIndex(.)
                 | .
                 v
                1st index
It is required that there be no overlap between the LR segments defined by deBlockList.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the order of DEs in the deBlockList argument.
[polekindflag]
Two item array which specifies the type of connection which occurs at the pole. The value in polekindflag(1) specifies the connection that occurs at the minimum end of the pole dimension. The value in polekindflag(2) specifies the connection that occurs at the maximum end of the pole dimension. Please see Section 31.2.5 for a full list of options. If not specified, the default is ESMF_POLEKIND_MONOPOLE for both.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[staggerLocList]
The list of stagger locations to fill with coordinates. Please see Section 31.2.6 for a description of the available stagger locations. If not present, then no staggers are added or filled.
[ignoreNonPeriCoord]
If .true., do not check if the coordinates for the periodic dimension (i.e. dim=1) specify a full periodic range (e.g. 0 to 360 degrees). If not specified, defaults to .false. .
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.26 ESMF_GridCreateNoPeriDimUfrm - Create a uniform Grid with no periodic dim and a regular distribution


INTERFACE:

  ! Private name; call using ESMF_GridCreateNoPeriDimUfrm()
       function ESMF_GridCreateNoPeriDimUfrmR(minIndex, maxIndex, &
         minCornerCoord, maxCornerCoord, &
         regDecomp, decompFlag, &
         coordSys, staggerLocList, petMap, name, rc)
RETURN VALUE:
       type(ESMF_Grid) :: ESMF_GridCreateNoPeriDimUfrmR
ARGUMENTS:
        integer,                   intent(in),  optional :: minIndex(:)
        integer,                   intent(in)            :: maxIndex(:)
        real(ESMF_KIND_R8),        intent(in)            :: minCornerCoord(:)
        real(ESMF_KIND_R8),        intent(in)            :: maxCornerCoord(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                   intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),    intent(in),  optional :: decompflag(:)
        type(ESMF_CoordSys_Flag),  intent(in),  optional :: coordSys
        type(ESMF_StaggerLoc),     intent(in),  optional :: staggerLocList(:)
        integer,                   intent(in),  optional :: petMap(:,:,:)
        character (len=*),         intent(in),  optional :: name
        integer,                   intent(out), optional :: rc
DESCRIPTION:

This method creates a single tile, regularly distributed grid (see Figure 13) with no periodic dimension.

The resulting grid will have its coordinates uniformly spread between the ranges specified by the user. The coordinates are ESMF_TYPEKIND_R8. Currently, this method only fills the center stagger with coordinates, and the minCornerCoord and maxCornerCoord arguments give the boundaries of the center stagger.

To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

The following arguments have been set to non-typical values and so there is a reasonable possibility that they may change in the future to be inconsistent with other Grid create interfaces:

The arguements coordDep1, coordDep2, and coordDep3 have internally been set to 1, 2, and 3 respectively. This was done because this call creates a uniform grid and so only 1D arrays are needed to hold the coordinates. This means the coordinate arrays will be 1D.

The argument indexFlag has internally been set to ESMF_INDEX_GLOBAL. This means that the grid created from this function will have a global index space.

The arguments are:

[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
minCornerCoord
The coordinates of the corner of the grid that corresponds to minIndex. size(minCornerCoord) must be equal to size(maxIndex).
maxCornerCoord
The coordinates of the corner of the grid that corresponds to maxIndex. size(maxCornerCoord) must be equal to size(maxIndex).
[regDecomp]
A ndims-element array specifying how the grid is decomposed. Each entry is the number of decounts for that dimension.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[staggerLocList]
The list of stagger locations to fill with coordinates. Please see Section 31.2.6 for a description of the available stagger locations. If not present, then no staggers are added or filled.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.27 ESMF_GridCreateCubedSphere - Create a multi-tile cubed sphere Grid with regular decomposition


INTERFACE:

   ! Private name; call using ESMF_GridCreateCubedSphere()
   function ESMF_GridCreateCubedSphereReg(tileSize,&
        regDecompPTile, decompflagPTile,                            &
        coordSys, coordTypeKind,                                    &
        deLabelList, staggerLocList,                                &
        delayout, indexflag, name, transformArgs, coordCalcFlag,    &
        rc)
RETURN VALUE:
     type(ESMF_Grid) :: ESMF_GridCreateCubedSphereReg
ARGUMENTS:
     integer,                        intent(in)            :: tilesize
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                        intent(in),  optional :: regDecompPTile(:,:)
     type(ESMF_Decomp_Flag), target, intent(in),  optional :: decompflagPTile(:,:)
     type(ESMF_CoordSys_Flag),       intent(in),  optional :: coordSys
     type(ESMF_TypeKind_Flag),       intent(in),  optional :: coordTypeKind
     integer,                        intent(in),  optional :: deLabelList(:)
     type(ESMF_StaggerLoc),          intent(in),  optional :: staggerLocList(:)
     type(ESMF_DELayout),            intent(in),  optional :: delayout
     type(ESMF_Index_Flag),          intent(in),  optional :: indexflag
     character(len=*),               intent(in),  optional :: name
     type(ESMF_CubedSphereTransform_Args), intent(in),  optional :: transformArgs
     type(ESMF_CubedSphereCalc_Flag),intent(in),  optional :: coordCalcFlag           
     integer,                        intent(out), optional :: rc
DESCRIPTION:

Create a six-tile ESMF_Grid for a Cubed Sphere grid using regular decomposition. Each tile can have different decomposition. The grid coordinates are generated based on the algorithm used by GEOS-5, The tile resolution is defined by tileSize.

The arguments are:

tilesize
The number of elements on each side of the tile of the Cubed Sphere grid.
[regDecompPTile]
List of DE counts for each dimension. The second index steps through the tiles. The total deCount is determined as the sum over the products of regDecompPTile elements for each tile. By default every tile is decomposed in the same way. If the total PET count is less than 6, one tile will be assigned to one DE and the DEs will be assigned to PETs sequentially, therefore, some PETs may have more than one DE. If the total PET count is greater than 6, the total number of DEs will be a multiple of 6 and less than or equal to the total PET count. For instance, if the total PET count is 16, the total DE count will be 12 with each tile decomposed into 1x2 blocks. The 12 DEs are mapped to the first 12 PETs and the remaining 4 PETs have no DEs locally, unless an optional delayout is provided.
[decompflagPTile]
List of decomposition flags indicating how each dimension of each tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions for all tiles. See section 54.14 for a list of valid decomposition flag options. The second index indicates the tile number.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the column major order of the regDecompPTile elements in the sequence as they appear following the tile index.
[staggerLocList]
The list of stagger locations to fill with coordinates. Only ESMF_STAGGERLOC_CENTER and ESMF_STAGGERLOC_CORNER are supported. If not present, no coordinates will be added or filled.
[coordSys]
The coordinate system of the grid coordinate data. Only ESMF_COORDSYS_SPH_DEG and ESMF_COORDSYS_SPH_RAD are supported. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. Only ESMF_TYPEKIND_R4 and ESMF_TYPEKIND_R8 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[delayout]
Optional ESMF_DELayout object to be used. By default a new DELayout object will be created with as many DEs as there are PETs, or tiles, which ever is greater. If a DELayout object is specified, the number of DEs must match regDecompPTile, if present. In the case that regDecompPTile was not specified, the deCount must be at least that of the default DELayout. The regDecompPTile will be constructed accordingly.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[name]
ESMF_Grid name.
[transformArgs]
A data type containing the stretch factor, target longitude, and target latitude to perform a Schmidt transformation on the Cubed-Sphere grid. 31.3.11 for details.
[coordCalcFlag]
A flag which controls the method used to calculate the cubed sphere coordinates. Please see section 54.12 for a list of options. If not set, defaults to ESMF_CUBEDSPHERECALC_1TILE which was the original method used to calculate coordinates.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.28 ESMF_GridCreateCubedSphere - Create a multi-tile cubed sphere Grid with irregular decomposition


INTERFACE:

   ! Private name; call using ESMF_GridCreateCubedSphere()
   function ESMF_GridCreateCubedSphereIReg(tileSize,              &
         countsPerDEDim1PTile, countsPerDEDim2PTile,              &
                                                 &        
         coordSys, coordTypeKind,                                 &
         deLabelList, staggerLocList,                             &
         delayout, indexflag, name, transformArgs, coordCalcFlag, &
         rc)
RETURN VALUE:
     type(ESMF_Grid) :: ESMF_GridCreateCubedSphereIReg
ARGUMENTS:
     integer,                        intent(in)            :: tilesize
     integer,                        intent(in)            :: countsPerDEDim1PTile(:,:)
     integer,                        intent(in)            :: countsPerDEDim2PTile(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_CoordSys_Flag),       intent(in),  optional :: coordSys
     type(ESMF_TypeKind_Flag),       intent(in),  optional :: coordTypeKind
     integer,                        intent(in),  optional :: deLabelList(:)
     type(ESMF_StaggerLoc),          intent(in),  optional :: staggerLocList(:)
     type(ESMF_DELayout),            intent(in),  optional :: delayout
     type(ESMF_Index_Flag),          intent(in),  optional :: indexflag
     character(len=*),               intent(in),  optional :: name
     type(ESMF_CubedSphereTransform_Args), intent(in),  optional :: transformArgs
     type(ESMF_CubedSphereCalc_Flag),intent(in),  optional :: coordCalcFlag           
     integer,                        intent(out), optional :: rc
DESCRIPTION:

Create a six-tile ESMF_Grid for a Cubed Sphere grid using irregular decomposition. Each tile can have different decomposition. The grid coordinates are generated based on the algorithm used by GEOS-5, The tile resolution is defined by tileSize.

The arguments are:

tilesize
The number of elements on each side of the tile of the Cubed Sphere grid.
countsPerDEDim1PTile
This array specifies the number of cells per DE for index dimension 1 for the center stagger location. The second index steps through the tiles. If each tile is decomposed into different number of DEs, the first dimension is the maximal DEs of all the tiles.
countsPerDEDim2PTile
This array specifies the number of cells per DE for index dimension 2 for the center stagger location. The second index steps through the tiles. If each tile is decomposed into different number of DEs, the first dimension is the maximal DEs of all the tiles.
[coordSys]
The coordinate system of the grid coordinate data. Only ESMF_COORDSYS_SPH_DEG and ESMF_COORDSYS_SPH_RAD are supported. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. Only ESMF_TYPEKIND_R4 and ESMF_TYPEKIND_R8 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the column major order in the sequence as they appear in countsPerDEDim1PTile, followed by countsPerDEDim2PTile, then the tile index.
[staggerLocList]
The list of stagger locations to fill with coordinates. Only ESMF_STAGGERLOC_CENTER and ESMF_STAGGERLOC_CORNER are supported. If not present, no coordinates will be added or filled.
[delayout]
Optional ESMF_DELayout object to be used. If a delayout object is specified, the number of DEs must match with the total DEs defined in countsPerDEDim1PTile and countsPerDEDim2PTile.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[name]
ESMF_Grid name.
[transformArgs]
A data type containing the stretch factor, target longitude, and target latitude to perform a Schmidt transformation on the Cubed-Sphere grid. See section 31.3.11 for details.
[coordCalcFlag]
A flag which controls the method used to calculate the cubed sphere coordinates. Please see section 54.12 for a list of options. If not set, defaults to ESMF_CUBEDSPHERECALC_1TILE which was the original method used to calculate coordinates.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.29 ESMF_GridCreateMosaic - Create a multi-tile Grid object with regular decomposition using the grid definition from a GRIDSPEC Mosaic file.


INTERFACE:

   function ESMF_GridCreateMosaicReg(filename,regDecompPTile, decompflagPTile, &
         coordTypeKind, deLabelList, staggerLocList, delayout, indexflag, name, tileFilePath, rc)
RETURN VALUE:
     type(ESMF_Grid) :: ESMF_GridCreateMosaicReg
ARGUMENTS:
     character(len=*),               intent(in)            :: filename
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                        intent(in),  optional :: regDecompPTile(:,:)
     type(ESMF_Decomp_Flag), target, intent(in),  optional :: decompflagPTile(:,:)
     type(ESMF_TypeKind_Flag),       intent(in),  optional :: coordTypeKind
     integer,                        intent(in),  optional :: deLabelList(:)
     type(ESMF_StaggerLoc),          intent(in),  optional :: staggerLocList(:)
     type(ESMF_DELayout),            intent(in),  optional :: delayout
     type(ESMF_Index_Flag),          intent(in),  optional :: indexflag
     character(len=*),               intent(in),  optional :: name
     character(len=*),               intent(in),  optional :: tileFilePath
     integer,                        intent(out), optional :: rc
DESCRIPTION:

Create a multiple-tile ESMF_Grid based on the definition from a GRIDSPEC Mosaic file and its associated tile files using regular decomposition. Each tile can have different decomposition. The tile connections are defined in a GRIDSPEC format Mosaic file. And each tile's coordination is defined in a separate NetCDF file. The coordinates defined in the tile file is so-called "Super Grid". In other words, the dimensions of the coordinate variables are (2*xdim+1, 2*ydim+1) if (xdim, ydim) is the size of the tile. The Super Grid combines the corner, the edge and the center coordinates in one big array. A Mosaic file may contain just one tile. If a Mosaic contains multiple tiles. Each tile is a logically rectangular lat/lon grid. Currently, all the tiles have to be the same size. We will remove this limitation in the future release.

The arguments are:

filename
The name of the GRIDSPEC Mosaic file.
[regDecompPTile]
List of DE counts for each dimension. The second index steps through the tiles. The total deCount is determined as the sum over the products of regDecompPTile elements for each tile. By default every tile is decomposed in the same way. If the total PET count is less than the tile count, one tile will be assigned to one DE and the DEs will be assigned to PETs sequentially, therefore, some PETs may have more than one DE. If the total PET count is greater than the tile count, the total number of DEs will be a multiple of the tile count and less than or equal to the total PET count. For instance, if the total PET count is 16 and the tile count is 6, the total DE count will be 12 with each tile decomposed into 1x2 blocks. The 12 DEs are mapped to the first 12 PETs and the remaining 4 PETs have no DEs locally, unless an optional delayout is provided.
[decompflagPTile]
List of decomposition flags indicating how each dimension of each tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions for all tiles. See section 54.14 for a list of valid decomposition flag options. The second index indicates the tile number.
[coordTypeKind]
The type/kind of the grid coordinate data. Only ESMF_TYPEKIND_R4 and ESMF_TYPEKIND_R8 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the column major order of the regDecompPTile elements in the sequence as they appear following the tile index.
[staggerLocList]
The list of stagger locations to fill with coordinates. Please see Section 31.2.6 for a description of the available stagger locations. If not present, no coordinates will be added or filled.
[delayout]
Optional ESMF_DELayout object to be used. By default a new DELayout object will be created with as many DEs as there are PETs, or tiles, which ever is greater. If a DELayout object is specified, the number of DEs must match regDecompPTile, if present. In the case that regDecompPTile was not specified, the deCount must be at least that of the default DELayout. The regDecompPTile will be constructed accordingly.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[name]
ESMF_Grid name.
[tileFilePath]
Optional argument to define the path where the tile files reside. If it is given, it overwrites the path defined in gridlocation variable in the mosaic file.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.30 ESMF_GridCreateMosaic - Create a multi-tile Grid object with irregular decomposition using the grid definition from a GRIDSPEC Mosaic file.


INTERFACE:

   function ESMF_GridCreateMosaicIReg(filename,                  &
           countsPerDEDim1PTile, countsPerDEDim2PTile,           &
                                                &
           coordTypeKind,                                        &
           deLabelList, staggerLocList,                          &
           delayout, indexflag, name, tileFilePath, rc)
RETURN VALUE:
     type(ESMF_Grid) :: ESMF_GridCreateMosaicIReg
ARGUMENTS:
     character(len=*),               intent(in)            :: filename
     integer,                        intent(in)            :: countsPerDEDim1PTile(:,:)
     integer,                        intent(in)            :: countsPerDEDim2PTile(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_TypeKind_Flag),       intent(in),  optional :: coordTypeKind
     integer,                        intent(in),  optional :: deLabelList(:)
     type(ESMF_StaggerLoc),          intent(in),  optional :: staggerLocList(:)
     type(ESMF_DELayout),            intent(in),  optional :: delayout
     type(ESMF_Index_Flag),          intent(in),  optional :: indexflag
     character(len=*),               intent(in),  optional :: name
     character(len=*),               intent(in),  optional :: tileFilePath
     integer,                        intent(out), optional :: rc
DESCRIPTION:

Create a multiple-tile ESMF_Grid based on the definition from a GRIDSPEC Mosaic file and its associated tile files using irregular decomposition. Each tile can have different decomposition. The tile connections are defined in a GRIDSPEC format Mosaic file. And each tile's coordination is defined in a separate NetCDF file. The coordinates defined in the tile file is so-called "Super Grid". In other words, the dimensions of the coordinate variables are (2*xdim+1, 2*ydim+1) if (xdim, ydim) is the size of the tile. The Super Grid combines the corner, the edge and the center coordinates in one big array. A Mosaic file may contain just one tile. If a Mosaic contains multiple tiles. Each tile is a logically rectangular lat/lon grid. Currently, all the tiles have to be the same size. We will remove this limitation in the future release.

The arguments are:

filename
The name of the GRIDSPEC Mosaic file.
countsPerDEDim1PTile
This array specifies the number of cells per DE for index dimension 1 for the center stagger location. The second index steps through the tiles. If each tile is decomposed into different number of DEs, the first dimension is the maximal DEs of all the tiles.
countsPerDEDim2PTile
This array specifies the number of cells per DE for index dimension 2 for the center stagger location. The second index steps through the tiles. If each tile is decomposed into different number of DEs, the first dimension is the maximal DEs of all the tiles.
[coordTypeKind]
The type/kind of the grid coordinate data. Only ESMF_TYPEKIND_R4 and ESMF_TYPEKIND_R8 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the column major order in the sequence as they appear in countsPerDEDim1PTile, followed by countsPerDEDim2PTile, then the tile index.
[staggerLocList]
The list of stagger locations to fill with coordinates. Please see Section 31.2.6 for a description of the available stagger locations. If not present, no coordinates will be added or filled.
[delayout]
Optional ESMF_DELayout object to be used. If a delayout object is specified, the number of DEs must match with the total DEs defined in countsPerDEDim1PTile and countsPerDEDim2PTile.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[name]
ESMF_Grid name.
[tileFilePath]
Optional argument to define the path where the tile files reside. If it is given, it overwrites the path defined in gridlocation variable in the mosaic file.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.31 ESMF_GridDestroy - Release resources associated with a Grid


INTERFACE:

       subroutine ESMF_GridDestroy(grid, noGarbage, rc)
ARGUMENTS:
       type(ESMF_Grid), intent(inout)         :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical,         intent(in),  optional :: noGarbage
       integer,         intent(out), optional :: rc
STATUS:

DESCRIPTION:

Destroys an ESMF_Grid object and related internal structures. This call does destroy internally created DistGrid and DELayout classes, for example those created by ESMF_GridCreateShapeTile(). It also destroys internally created coordinate/item Arrays, for example those created by ESMF_GridAddCoord(). However, if the user uses an externally created class, for example creating an Array and setting it using ESMF_GridSetCoord(), then that class is not destroyed by this method.

By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.

The arguments are:

grid
ESMF_Grid to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.32 ESMF_GridEmptyComplete - Complete a Grid with user set edge connections and an irregular distribution


INTERFACE:

   ! Private name; call using ESMF_GridEmptyComplete()
       subroutine ESMF_GridEmptyCompleteEConnI(grid, minIndex,         &
         countsPerDEDim1,countsPerDeDim2,                  &
         countsPerDEDim3,                                  &
         connDim1, connDim2, connDim3,                     &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,        &
         gridMemLBound, indexflag, petMap, name, rc)
ARGUMENTS:
        type (ESMF_Grid)                                :: grid
        integer,                  intent(in),  optional :: minIndex(:)
        integer,                  intent(in)            :: countsPerDEDim1(:)
        integer,                  intent(in)            :: countsPerDEDim2(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                  intent(in),  optional :: countsPerDEDim3(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim1(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim2(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim3(:)
        type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),  optional :: coordTypeKind
        integer,                  intent(in),  optional :: coordDep1(:)
        integer,                  intent(in),  optional :: coordDep2(:)
        integer,                  intent(in),  optional :: coordDep3(:)
        integer,                  intent(in),  optional :: gridEdgeLWidth(:)
        integer,                  intent(in),  optional :: gridEdgeUWidth(:)
        integer,                  intent(in),  optional :: gridAlign(:)
        integer,                  intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),    intent(in),  optional :: indexflag
        integer,                  intent(in),  optional :: petMap(:,:,:)
        character (len=*),        intent(in),  optional :: name
        integer,                  intent(out), optional :: rc
DESCRIPTION:

This method takes in an empty Grid created by ESMF_GridEmptyCreate(). It then completes the grid to form a single tile, irregularly distributed grid (see Figure 13). To specify the irregular distribution, the user passes in an array for each grid dimension, where the length of the array is the number of DEs in the dimension. Currently this call only supports creating 2D or 3D Grids. A 2D Grid can be specified using the countsPerDEDim1 and countsPerDEDim2 arguments. A 3D Grid can be specified by also using the optional countsPerDEDim3 argument. The index of each array element in these arguments corresponds to a DE number. The array value at the index is the number of grid cells on the DE in that dimension.

Section 31.3.4 shows an example of using an irregular distribution to create a 2D Grid with uniformly spaced coordinates. This creation method can also be used as the basis for grids with rectilinear coordinates or curvilinear coordinates.

For consistency's sake the ESMF_GridEmptyComplete() call should be executed in the same set or a subset of the PETs in which the ESMF_GridEmptyCreate() call was made. If the call is made in a subset, the Grid objects outside that subset will still be "empty" and not usable.

The arguments are:

grid
The empty ESMF_Grid to set information into and then commit.
[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
countsPerDEDim1
This arrays specifies the number of cells per DE for index dimension 1 for the exclusive region (the center stagger location).
countsPerDEDim2
This array specifies the number of cells per DE for index dimension 2 for the exclusive region (center stagger location).
[countsPerDEDim3]
This array specifies the number of cells per DE for index dimension 3 for the exclusive region (center stagger location). If not specified then grid is 2D.
[connDim1]
Fortran array describing the index dimension 1 connections. The first element represents the minimum end of dimension 1. The second element represents the maximum end of dimension 1. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connDim2]
Fortran array describing the index dimension 2 connections. The first element represents the minimum end of dimension 2. The second element represents the maximum end of dimension 2. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connDim3]
Fortran array describing the index dimension 3 connections. The first element represents the minimum end of dimension 3. The second element represents the maximum end of dimension 3. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size size(countsPerDEDim1) x size(countsPerDEDim2) x size(countsPerDEDim3). If countsPerDEDim3 isn't present, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.33 ESMF_GridEmptyComplete - Complete a Grid with user set edge connections and a regular distribution


INTERFACE:

   ! Private name; call using ESMF_GridEmptyComplete()
      subroutine ESMF_GridEmptyCompleteEConnR(grid, regDecomp, decompFlag, &
         minIndex, maxIndex,                                    &
         connDim1, connDim2, connDim3,                       &
         coordSys, coordTypeKind,                            &
         coordDep1, coordDep2, coordDep3,                    &
         gridEdgeLWidth, gridEdgeUWidth, gridAlign,          &
         gridMemLBound, indexflag, petMap, name, rc)
 
  !
ARGUMENTS:
        type (ESMF_Grid)                                :: grid
        integer,                  intent(in),  optional :: regDecomp(:)
        type(ESMF_Decomp_Flag),   intent(in),  optional :: decompflag(:)
        integer,                  intent(in),  optional :: minIndex(:)
        integer,                  intent(in)            :: maxIndex(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim1(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim2(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim3(:)
        type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),  optional :: coordTypeKind
        integer,                  intent(in),  optional :: coordDep1(:)
        integer,                  intent(in),  optional :: coordDep2(:)
        integer,                  intent(in),  optional :: coordDep3(:)
        integer,                  intent(in),  optional :: gridEdgeLWidth(:)
        integer,                  intent(in),  optional :: gridEdgeUWidth(:)
        integer,                  intent(in),  optional :: gridAlign(:)
        integer,                  intent(in),  optional :: gridMemLBound(:)
        type(ESMF_Index_Flag),    intent(in),  optional :: indexflag
        integer,                  intent(in),  optional :: petMap(:,:,:)
        character (len=*),        intent(in),  optional :: name
        integer,                  intent(out), optional :: rc
DESCRIPTION:

This method takes in an empty Grid created by ESMF_GridEmptyCreate(). It then completes the grid to form a single tile, regularly distributed grid (see Figure 13). To specify the distribution, the user passes in an array (regDecomp) specifying the number of DEs to divide each dimension into. The array decompFlag indicates how the division into DEs is to occur. The default is to divide the range as evenly as possible. Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

For consistency's sake the ESMF_GridEmptyComplete() call should be executed in the same set or a subset of the PETs in which the ESMF_GridEmptyCreate() call was made. If the call is made in a subset, the Grid objects outside that subset will still be "empty" and not usable.

The arguments are:

grid
The empty ESMF_Grid to set information into and then commit.
[regDecomp]
List that has the same number of elements as maxIndex. Each entry is the number of decounts for that dimension. If not specified, the default decomposition will be petCountx1x1..x1.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. Please see Section 54.14 for a full description of the possible options. Note that currently the option ESMF_DECOMP_CYCLIC isn't supported in Grid creation.
[minIndex]
The bottom extent of the grid array. If not given then the value defaults to /1,1,1,.../.
maxIndex
The upper extent of the grid array.
[connDim1]
Fortran array describing the index dimension 1 connections. The first element represents the minimum end of dimension 1. The second element represents the maximum end of dimension 1. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connDim2]
Fortran array describing the index dimension 2 connections. The first element represents the minimum end of dimension 2. The second element represents the maximum end of dimension 2. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connDim3]
Fortran array describing the index dimension 3 connections. The first element represents the minimum end of dimension 3. The second element represents the maximum end of dimension 3. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
This array specifies the dependence of the first coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep2]
This array specifies the dependence of the second coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[coordDep3]
This array specifies the dependence of the third coordinate component on the three index dimensions described by coordsPerDEDim1,2,3. The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. If not present the default is 1,2,...,grid rank.
[gridEdgeLWidth]
The padding around the lower edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 0, 0, ..., 0 (all zeros).
[gridEdgeUWidth]
The padding around the upper edges of the grid. This padding is between the index space corresponding to the cells and the boundary of the the exclusive region. This extra space is to contain the extra padding for non-center stagger locations, and should be big enough to hold any stagger in the grid. If this and gridAlign are not present then defaults to 1, 1, ..., 1 (all ones).
[gridAlign]
Specification of how the stagger locations should align with the cell index space (can be overridden by the individual staggerAligns). If the gridEdgeWidths are not specified than this argument implies the gridEdgeWidths. If the gridEdgeWidths are specified and this argument isn't then this argument is implied by the gridEdgeWidths. If this and the gridEdgeWidths are not specified, then defaults to -1, -1, ..., -1 (all negative ones).
[gridMemLBound]
Specifies the lower index range of the memory of every DE in this Grid. Only used when indexflag is ESMF_INDEX_USER. May be overridden by staggerMemLBound.
[indexflag]
Indicates the indexing scheme to be used in the new Grid. Please see Section 54.29 for the list of options. If not present, defaults to ESMF_INDEX_DELOCAL.
[petMap]
Sets the mapping of pets to the created DEs. This 3D should be of size regDecomp(1) x regDecomp(2) x regDecomp(3) If the Grid is 2D, then the last dimension is of size 1.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.34 ESMF_GridEmptyComplete - Complete a Grid with user set edge connections and an arbitrary distribution


INTERFACE:

   ! Private name; call using ESMF_GridEmptyComplete()
       subroutine ESMF_GridEmptyCompleteEConnA(grid, minIndex, maxIndex,  &
         arbIndexCount, arbIndexList,                         &
         connDim1, connDim2, connDim3,                     &
         coordSys, coordTypeKind,                          &
         coordDep1, coordDep2, coordDep3,                  &
         distDim, name, rc)
  !
ARGUMENTS:
        type (ESMF_Grid)                                :: grid
        integer,                  intent(in),  optional :: minIndex(:)
        integer,                  intent(in)            :: maxIndex(:)
        integer,                  intent(in)            :: arbIndexCount
        integer,                  intent(in)            :: arbIndexList(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim1(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim2(:)
        type(ESMF_GridConn_Flag), intent(in),  optional :: connDim3(:)
        type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
        type(ESMF_TypeKind_Flag), intent(in),  optional :: coordTypeKind
        integer,                  intent(in),  optional :: coordDep1(:)
        integer,                  intent(in),  optional :: coordDep2(:)
        integer,                  intent(in),  optional :: coordDep3(:)
        integer,                  intent(in),  optional :: distDim(:)
        character (len=*),        intent(in),  optional :: name
        integer,                  intent(out), optional :: rc
DESCRIPTION:

This method takes in an empty Grid created by ESMF_GridEmptyCreate(). It then completes the grid to form a single tile, arbitrarily distributed grid (see Figure 13). To specify the arbitrary distribution, the user passes in an 2D array of local indices, where the first dimension is the number of local grid cells specified by localArbIndexCount and the second dimension is the number of distributed dimensions.

distDim specifies which grid dimensions are arbitrarily distributed. The size of distDim has to agree with the size of the second dimension of localArbIndex.

Currently this call only supports creating a 2D or 3D Grid, and thus, for example, maxIndex must be of size 2 or 3.

For consistency's sake the ESMF_GridEmptyComplete() call should be executed in the same set or a subset of the PETs in which the ESMF_GridEmptyCreate() call was made. If the call is made in a subset, the Grid objects outside that subset will still be "empty" and not usable.

The arguments are:

grid
The empty ESMF_Grid to set information into and then commit.
[minIndex]
Tuple to start the index ranges at. If not present, defaults to /1,1,1,.../.
maxIndex
The upper extend of the grid index ranges.
arbIndexCount
The number of grid cells in the local DE. It is okay to have 0 grid cell in a local DE.
arbIndexList
This 2D array specifies the indices of the PET LOCAL grid cells. The dimensions should be arbIndexCount * number of Distributed grid dimensions where arbIndexCount is the input argument specified below
[connDim1]
Fortran array describing the index dimension 1 connections. The first element represents the minimum end of dimension 1. The second element represents the maximum end of dimension 1. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connDim2]
Fortran array describing the index dimension 2 connections. The first element represents the minimum end of dimension 2. The second element represents the maximum end of dimension 2. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[connDim3]
Fortran array describing the index dimension 3 connections. The first element represents the minimum end of dimension 3. The second element represents the maximum end of dimension 3. If array is only one element long, then that element is used for both the minimum and maximum end. Please see Section 31.2.1 for a list of valid options. If not present, defaults to ESMF_GRIDCONN_NONE.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported. If not specified then defaults to ESMF_TYPEKIND_R8.
[coordDep1]
The size of the array specifies the number of dimensions of the first coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_GRID_ARBDIM/ where /ESMF_GRID_ARBDIM/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_GRID_ARBDIM/ if the first dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=1) Please see Section 54.2 for a definition of ESMF_GRID_ARBDIM.
[coordDep2]
The size of the array specifies the number of dimensions of the second coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_GRID_ARBDIM/ where /ESMF_GRID_ARBDIM/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_GRID_ARBDIM/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=2) Please see Section 54.2 for a definition of ESMF_GRID_ARBDIM.
[coordDep3]
The size of the array specifies the number of dimensions of the third coordinate component array. The values specify which of the index dimensions the corresponding coordinate arrays map to. The format should be /ESMF_GRID_ARBDIM/ where /ESMF_GRID_ARBDIM/ is mapped to the collapsed 1D dimension from all the arbitrarily distributed dimensions. n is the dimension that is not distributed (if exists). If not present the default is /ESMF_GRID_ARBDIM/ if this dimension is arbitararily distributed, or /n/ if not distributed (i.e. n=3) Please see Section 54.2 for a definition of ESMF_GRID_ARBDIM.
[distDim]
This array specifies which dimensions are arbitrarily distributed. The size of the array specifies the total distributed dimensions. if not specified, defaults is all dimensions will be arbitrarily distributed. The size has to agree with the size of the second dimension of localArbIndex.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.35 ESMF_GridEmptyCreate - Create a Grid that has no contents


INTERFACE:

      function ESMF_GridEmptyCreate(vm, rc)
RETURN VALUE:
      type(ESMF_Grid) :: ESMF_GridEmptyCreate
ARGUMENTS:
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        type(ESMF_VM),           intent(in),  optional :: vm
        integer,                 intent(out), optional :: rc
STATUS:

DESCRIPTION:

Partially create an ESMF_Grid object. This function allocates an ESMF_Grid object, but doesn't allocate any coordinate storage or other internal structures. The ESMF_GridEmptyComplete() calls can be used to set the values in the grid object and to construct the internal structure.

The arguments are:

[vm]
If present, the Grid object is created on the specified ESMF_VM object. The default is to create on the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.36 ESMF_GridGet - Get object-wide Grid information


INTERFACE:

   ! Private name; call using ESMF_GridGet()
       subroutine ESMF_GridGetDefault(grid, coordTypeKind, &
         dimCount, tileCount, staggerlocCount, localDECount, distgrid, &
         distgridToGridMap, coordSys, coordDimCount, coordDimMap, arbDim, &
         rank, arbDimCount, gridEdgeLWidth, gridEdgeUWidth, gridAlign,  &
         indexFlag, status, name, rc)
ARGUMENTS:
       type(ESMF_Grid),            intent(in)            :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_TypeKind_Flag),  intent(out), optional :: coordTypeKind
       integer,                   intent(out), optional :: dimCount
       integer,                   intent(out), optional :: tileCount
       integer,                   intent(out), optional :: staggerlocCount
       integer,                   intent(out), optional :: localDECount
       type(ESMF_DistGrid),       intent(out), optional :: distgrid
       integer,       target,     intent(out), optional :: distgridToGridMap(:)
       type(ESMF_CoordSys_Flag),  intent(out), optional :: coordSys
       integer,       target,     intent(out), optional :: coordDimCount(:)
       integer,       target,     intent(out), optional :: coordDimMap(:,:)
       integer,                   intent(out), optional :: arbDim
       integer,                   intent(out), optional :: rank
       integer,                   intent(out), optional :: arbDimCount
       integer,       target,     intent(out), optional :: gridEdgeLWidth(:)
       integer,       target,     intent(out), optional :: gridEdgeUWidth(:)
       integer,       target,     intent(out), optional :: gridAlign(:)
       type(ESMF_Index_Flag),     intent(out), optional :: indexflag
       type(ESMF_GridStatus_Flag),intent(out), optional :: status
       character (len=*),         intent(out), optional :: name
       integer,                   intent(out), optional :: rc
STATUS:

DESCRIPTION:

Gets various types of information about a grid.

The arguments are:

grid
Grid to get the information from.
[coordTypeKind]
The type/kind of the grid coordinate data. All numerical types listed under section 54.61 are supported.
[dimCount]
DimCount of the Grid object.
[tileCount]
The number of logically rectangular tiles in the grid.
[staggerlocCount]
The number of stagger locations.
[localDECount]
The number of DEs in this grid on this PET.
[distgrid]
The structure describing the distribution of the grid.
[distgridToGridMap]
List that has as many elements as the distgrid dimCount. This array describes mapping between the grids dimensions and the distgrid.
[coordSys]
The coordinate system of the grid coordinate data.
[coordDimCount]
This argument needs to be of size equal to the Grid's dimCount. Each entry in the argument will be filled with the dimCount of the corresponding coordinate component (e.g. the dimCount of coordDim=1 will be put into entry 1). This is useful because it describes the factorization of the coordinate components in the Grid.
[coordDimMap]
2D list of size grid dimCount x grid dimCount. This array describes the map of each component array's dimensions onto the grids dimensions.
[arbDim]
The distgrid dimension that is mapped by the arbitrarily distributed grid dimensions.
[rank]
The count of the memory dimensions, it is the same as dimCount for a non-arbitrarily distributed grid, and equal or less for a arbitrarily distributed grid.
[arbDimCount]
The number of dimensions distributed arbitrarily for an arbitrary grid, 0 if the grid is non-arbitrary.
[gridEdgeLWidth]
The padding around the lower edges of the grid. The array should be of size greater or equal to the Grid dimCount.
[gridEdgeUWidth]
The padding around the upper edges of the grid. The array should be of size greater or equal to the Grid dimCount.
[gridAlign]
Specification of how the stagger locations should align with the cell index space. The array should be of size greater or equal to the Grid dimCount.
[indexflag]
Flag indicating the indexing scheme being used in the Grid. Please see Section 54.29 for the list of options.
[status]
Flag indicating the status of the Grid. Please see Section 31.2.4 for the list of options.
[name]
ESMF_Grid name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.37 ESMF_GridGet - Get DE-local Grid information


INTERFACE:

   ! Private name; call using ESMF_GridGet()
       subroutine ESMF_GridGetPLocalDe(grid, localDE, &
         isLBound,isUBound, arbIndexCount, arbIndexList, tile, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       integer,                intent(in)            :: localDE
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical,                intent(out), optional :: isLBound(:)
       logical,                intent(out), optional :: isUBound(:)
       integer,                intent(out), optional :: arbIndexCount
       integer,        target, intent(out), optional :: arbIndexList(:,:)
       integer,                intent(out), optional :: tile
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This call gets information about a particular local DE in a Grid.

The arguments are:

grid
Grid to get the information from.
localDE
The local DE from which to get the information. [0,..,localDECount-1]
[isLBound]
Upon return, for each dimension this indicates if the DE is a lower bound of the Grid. isLBound must be allocated to be of size equal to the Grid dimCount.
[isUBound]
Upon return, for each dimension this indicates if the DE is an upper bound of the Grid. isUBound must be allocated to be of size equal to the Grid dimCount.
[arbIndexCount]
The number of local cells for an arbitrarily distributed grid
[arbIndexList]
The 2D array storing the local cell indices for an arbitrarily distributed grid. The size of the array is arbIndexCount * arbDimCount
[tile]
The number of the tile in which localDE is contained. Tile numbers range from 1 to TileCount.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.38 ESMF_GridGet - Get DE-local information for a specific stagger location in a Grid


INTERFACE:

   ! Private name; call using ESMF_GridGet()
       subroutine ESMF_GridGetPLocalDePSloc(grid, staggerloc, localDE, &
         exclusiveLBound, exclusiveUBound, exclusiveCount,  &
         computationalLBound, computationalUBound, computationalCount, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       type (ESMF_StaggerLoc), intent(in)            :: staggerloc
       integer,                intent(in)            :: localDE
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,        target, intent(out), optional :: exclusiveLBound(:)
       integer,        target, intent(out), optional :: exclusiveUBound(:)
       integer,        target, intent(out), optional :: exclusiveCount(:)
       integer,        target, intent(out), optional :: computationalLBound(:)
       integer,        target, intent(out), optional :: computationalUBound(:)
       integer,        target, intent(out), optional :: computationalCount(:)
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method gets information about the range of index space which a particular stagger location occupies. This call differs from the coordinate bound calls (e.g. ESMF_GridGetCoord) in that a given coordinate array may only occupy a subset of the Grid's dimensions, and so these calls may not give all the bounds of the stagger location. The bounds from this call are the full bounds, and so for example, give the appropriate bounds for allocating a Fortran array to hold data residing on the stagger location. Note that unlike the output from the Array, these values also include the undistributed dimensions and are ordered to reflect the order of the indices in the Grid. This call will still give correct values even if the stagger location does not contain coordinate arrays (e.g. if ESMF_GridAddCoord hasn't yet been called on the stagger location).

The arguments are:

grid
Grid to get the information from.
staggerloc
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations.
localDE
The local DE from which to get the information. [0,..,localDECount-1]
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to the Grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to the Grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveCount]
Upon return this holds the number of items,exclusiveUBound-exclusiveLBound+1, in the exclusive region per dimension. exclusiveCount must be allocated to be of size equal to the Grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the computational region. computationalLBound must be allocated to be of size equal to the Grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the upper bounds of the computational region. computationalUBound must be allocated to be of size equal to the Grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension. (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to the Grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.39 ESMF_GridGet - Get information about a specific stagger location in a Grid


INTERFACE:

   ! Private name; call using ESMF_GridGet()
       subroutine ESMF_GridGetPSloc(grid, staggerloc, &
         distgrid, &
         staggerEdgeLWidth, staggerEdgeUWidth, &
         staggerAlign, staggerLBound, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       type (ESMF_StaggerLoc), intent(in)            :: staggerloc
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_DistGrid),    intent(out), optional :: distgrid
       integer,                intent(out), optional :: staggerEdgeLWidth(:)
       integer,                intent(out), optional :: staggerEdgeUWidth(:)
       integer,                intent(out), optional :: staggerAlign(:)
       integer,                intent(out), optional :: staggerLBound(:)
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method gets information about a particular stagger location. This information is useful for creating an ESMF Array to hold the data at the stagger location.

The arguments are:

grid
Grid to get the information from.
staggerloc
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations.
[distgrid]
The structure describing the distribution of this staggerloc in this grid.
[staggerEdgeLWidth]
This array should be the same dimCount as the grid. It specifies the lower corner of the stagger region with respect to the lower corner of the exclusive region.
[staggerEdgeUWidth]
This array should be the same dimCount as the grid. It specifies the upper corner of the stagger region with respect to the upper corner of the exclusive region.
[staggerAlign]
This array is of size grid dimCount. For this stagger location, it specifies which element has the same index value as the center. For example, for a 2D cell with corner stagger it specifies which of the 4 corners has the same index as the center.
[staggerLBound]
Specifies the lower index range of the memory of every DE in this staggerloc in this Grid. Only used when Grid indexflag is ESMF_INDEX_USER.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.40 ESMF_GridGet - Get information about a specific stagger location and tile in a Grid


INTERFACE:

   ! Private name; call using ESMF_GridGet()
       subroutine ESMF_GridGetPSlocPTile(grid, tile, staggerloc, &
         minIndex, maxIndex, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       integer,                intent(in)            :: tile
       type (ESMF_StaggerLoc), intent(in)            :: staggerloc
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,        target, intent(out), optional :: minIndex(:)
       integer,        target, intent(out), optional :: maxIndex(:)
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method gets information about a particular stagger location. This information is useful for creating an ESMF Array to hold the data at the stagger location.

The arguments are:

grid
Grid to get the information from.
tile
The tile number to get the data from. Tile numbers range from 1 to TileCount.
staggerloc
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations.
[minIndex]
Upon return this holds the global lower index of this stagger location. minIndex must be allocated to be of size equal to the grid DimCount. Note that this value is only for the first Grid tile, as multigrid support is added, this interface will likely be changed or moved to adapt.
[maxIndex]
Upon return this holds the global upper index of this stagger location. maxIndex must be allocated to be of size equal to the grid DimCount. Note that this value is only for the first Grid tile, as multigrid support is added, this interface will likely be changed or moved to adapt.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.41 ESMF_GridGetCoord - Get a DE-local Fortran array pointer to Grid coord data and coord bounds


INTERFACE:

        subroutine ESMF_GridGetCoord<rank><type><kind>(grid, coordDim,    &
          staggerloc, localDE, farrayPtr, datacopyflag,                  &
          exclusiveLBound, exclusiveUBound, exclusiveCount,              &
          computationalLBound, computationalUBound, computationalCount,  &
          totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
       type(ESMF_Grid),       intent(in)              :: grid
       integer,               intent(in)              :: coordDim
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc) intent(in),    optional :: staggerloc
       integer,               intent(in),    optional :: localDE
       <type> (ESMF_KIND_<kind>), pointer             :: farrayPtr(<rank>)
       type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
       integer,               intent(out),   optional :: exclusiveLBound(:)
       integer,               intent(out),   optional :: exclusiveUBound(:)
       integer,               intent(out),   optional :: exclusiveCount(:)
       integer,               intent(out),   optional :: computationalLBound(:)
       integer,               intent(out),   optional :: computationalUBound(:)
       integer,               intent(out),   optional :: computationalCount(:)
       integer,               intent(out),   optional :: totalLBound(:)
       integer,               intent(out),   optional :: totalUBound(:)
       integer,               intent(out),   optional :: totalCount(:)
       integer,               intent(out),   optional :: rc
STATUS:

DESCRIPTION:

This method gets a Fortran pointer to the piece of memory which holds the coordinate data on the local DE for the given coordinate dimension and stagger locations. This is useful, for example, for setting the coordinate values in a Grid, or for reading the coordinate values. Currently this method supports up to three coordinate dimensions, of either R4 or R8 datatype. See below for specific supported values. If the coordinates that you are trying to retrieve are of higher dimension, use the ESMF_GetCoord() interface that returns coordinate values in an ESMF_Array instead. That interface supports the retrieval of coordinates up to 7D.

Supported values for the farrayPtr argument are:

real(ESMF_KIND_R4), pointer :: farrayPtr(:)
real(ESMF_KIND_R4), pointer :: farrayPtr(:,:)
real(ESMF_KIND_R4), pointer :: farrayPtr(:,:,:)
real(ESMF_KIND_R8), pointer :: farrayPtr(:)
real(ESMF_KIND_R8), pointer :: farrayPtr(:,:)
real(ESMF_KIND_R8), pointer :: farrayPtr(:,:,:)

The arguments are:

grid
Grid to get the information from.
coordDim
The coordinate dimension to get the data from (e.g. 1=x).
[staggerloc]
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
farrayPtr
The pointer to the coordinate data.
[datacopyflag]
If not specified, default to ESMF_DATACOPY_REFERENCE, in this case farrayPtr is a reference to the data in the Grid coordinate arrays. Please see Section 54.13 for further description and a list of valid values.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to the coord dimCount.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to the coord dimCount.
[exclusiveCount]
Upon return this holds the number of items, exclusiveUBound-exclusiveLBound+1, in the exclusive region per dimension. exclusiveCount must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the stagger region. computationalLBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the upper bounds of the stagger region. exclusiveUBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalLBound]
Upon return this holds the lower bounds of the total region. totalLBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalUBound]
Upon return this holds the upper bounds of the total region. totalUBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalCount]
Upon return this holds the number of items in the total region per dimension (i.e. totalUBound-totalLBound+1). totalCount must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.42 ESMF_GridGetCoord - Get coordinates and put into an Array


INTERFACE:

   ! Private name; call using ESMF_GridGetCoord()
       subroutine ESMF_GridGetCoordIntoArray(grid, coordDim, staggerloc, &
         array, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       integer,                intent(in)            :: coordDim
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       type(ESMF_Array),       intent(out)           :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method allows the user to get access to the ESMF Array holding coordinate data at a particular stagger location. This is useful, for example, to set the coordinate values. To have an Array to access, the coordinate Arrays must have already been allocated, for example by ESMF_GridAddCoord or ESMF_GridSetCoord.

The arguments are:

grid
The grid to get the coord array from.
coordDim
The coordinate dimension to get the data from (e.g. 1=x).
[staggerloc]
The stagger location from which to get the arrays. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
array
An array into which to put the coordinate information.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.43 ESMF_GridGetCoord - Get DE-local coordinates from a specific index location in a Grid


INTERFACE:

   ! Private name; call using ESMF_GridGetCoord()
       subroutine ESMF_GridGetCoordR4(grid, staggerloc, localDE, &
         index, coord, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       integer,                intent(in),  optional :: localDE
       integer,                intent(in)            :: index(:)
       real(ESMF_KIND_R4),     intent(out)           :: coord(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Given a specific index location in a Grid, this method returns the full set of coordinates from that index location. This method should work no matter what the factorization of the Grid's coordinate components.

The arguments are:

grid
Grid to get the information from.
[staggerloc]
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
index
This array holds the index location to be queried in the Grid. This array must at least be of the size Grid rank.
coord
This array will be filled with the coordinate data. This array must at least be of the size Grid rank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.44 ESMF_GridGetCoord - Get DE-local coordinates from a specific index location in a Grid


INTERFACE:

   ! Private name; call using ESMF_GridGetCoord()
       subroutine ESMF_GridGetCoordR8(grid, staggerloc, localDE, &
         index, coord, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       integer,                intent(in),  optional :: localDE
       integer,                intent(in)            :: index(:)
       real(ESMF_KIND_R8),     intent(out)           :: coord(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Given a specific index location in a Grid, this method returns the full set of coordinates from that index location. This method should work no matter what the factorization of the Grid's coordinate components.

The arguments are:

grid
Grid to get the information from.
[staggerloc]
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
index
This array holds the index location to be queried in the Grid. This array must at least be of the size Grid rank.
coord
This array will be filled with the coordinate data. This array must at least be of the size Grid rank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.45 ESMF_GridGetCoord - Get information about the coordinates at a particular stagger location


INTERFACE:

   ! Private name; call using ESMF_GridGetCoord()
       subroutine ESMF_GridGetCoordInfo(grid, &
         staggerloc, isPresent, rc)
ARGUMENTS:
       type(ESMF_Grid),           intent(in)            :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc),    intent(in),  optional :: staggerloc
       logical,                   intent(out), optional :: isPresent
       integer,                   intent(out), optional :: rc
DESCRIPTION:

This method allows the user to get information about the coordinates on a given stagger.

The arguments are:

grid
Grid to get the information from.
[staggerloc]
The stagger location from which to get information. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[isPresent]
If .true. then coordinates have been added on this staggerloc.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.46 ESMF_GridGetCoordBounds - Get Grid coordinate bounds


INTERFACE:

       subroutine ESMF_GridGetCoordBounds(grid, coordDim,   &
         staggerloc, localDE, exclusiveLBound, exclusiveUBound, &
         exclusiveCount, computationalLBound, computationalUBound , &
         computationalCount, totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       integer,                intent(in)            :: coordDim
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       integer,                intent(in),  optional :: localDE
       integer,        target, intent(out), optional :: exclusiveLBound(:)
       integer,        target, intent(out), optional :: exclusiveUBound(:)
       integer,        target, intent(out), optional :: exclusiveCount(:)
       integer,        target, intent(out), optional :: computationalLBound(:)
       integer,        target, intent(out), optional :: computationalUBound(:)
       integer,        target, intent(out), optional :: computationalCount(:)
       integer,        target, intent(out), optional :: totalLBound(:)
       integer,        target, intent(out), optional :: totalUBound(:)
       integer,        target, intent(out), optional :: totalCount(:)
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method gets information about the range of index space which a particular piece of coordinate data occupies. In other words, this method returns the bounds of the coordinate arrays. Note that unlike the output from the Array, these values also include the undistributed dimensions and are ordered to reflect the order of the indices in the coordinate. So, for example, totalLBound and totalUBound should match the bounds of the Fortran array retrieved by ESMF_GridGetCoord.

The arguments are:

grid
Grid to get the information from.
coordDim
The coordinate dimension to get the information for (e.g. 1=x).
[staggerloc]
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveCount]
Upon return this holds the number of items, exclusiveUBound-exclusiveLBound+1, in the exclusive region per dimension. exclusiveCount must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the stagger region. computationalLBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the upper bounds of the stagger region. computationalUBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalLBound]
Upon return this holds the lower bounds of the total region. totalLBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalUBound]
Upon return this holds the upper bounds of the total region. totalUBound must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalCount]
Upon return this holds the number of items in the total region per dimension (i.e. totalUBound-totalLBound+1). totalCount must be allocated to be of size equal to the coord dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.47 ESMF_GridGetItem - Get a DE-local Fortran array pointer to Grid item data and item bounds


INTERFACE:

        subroutine ESMF_GridGetItem<rank><type><kind>(grid, itemflag,  &
          staggerloc, localDE, farrayPtr, datacopyflag,                  &
          exclusiveLBound, exclusiveUBound, exclusiveCount,              &
          computationalLBound, computationalUBound, computationalCount,  &
          totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
       type(ESMF_Grid),      intent(in)               :: grid
       type (ESMF_GridItem_Flag),intent(in)           :: itemflag
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc), intent(in),  optional  :: staggerloc
       integer,              intent(in),    optional  :: localDE
       <type> (ESMF_KIND_<kind>), pointer             :: farrayPtr(<rank>)
       type(ESMF_DataCopy_Flag),intent(in), optional  :: datacopyflag
       integer,              intent(out),   optional  :: exclusiveLBound(:)
       integer,              intent(out),   optional  :: exclusiveUBound(:)
       integer,              intent(out),   optional  :: exclusiveCount(:)
       integer,              intent(out),   optional  :: computationalLBound(:)
       integer,              intent(out),   optional  :: computationalUBound(:)
       integer,              intent(out),   optional  :: computationalCount(:)
       integer,              intent(out),   optional  :: totalLBound(:)
       integer,              intent(out),   optional  :: totalUBound(:)
       integer,              intent(out),   optional  :: totalCount(:)
       integer,              intent(out),   optional  :: rc
STATUS:

DESCRIPTION:

This method gets a Fortran pointer to the piece of memory which holds the item data on the local DE for the given stagger locations. This is useful, for example, for setting the item values in a Grid, or for reading the item values. Currently this method supports up to three grid dimensions, but is limited to the I4 datatype. See below for specific supported values. If the item values that you are trying to retrieve are of higher dimension, use the ESMF_GetItem() interface that returns coordinate values in an ESMF_Array instead. That interface supports the retrieval of coordinates up to 7D.

Supported values for the farrayPtr argument are:

integer(ESMF_KIND_I4), pointer :: farrayPtr(:)
integer(ESMF_KIND_I4), pointer :: farrayPtr(:,:)
integer(ESMF_KIND_I4), pointer :: farrayPtr(:,:,:)
real(ESMF_KIND_R4), pointer :: farrayPtr(:)
real(ESMF_KIND_R4), pointer :: farrayPtr(:,:)
real(ESMF_KIND_R4), pointer :: farrayPtr(:,:,:)
real(ESMF_KIND_R8), pointer :: farrayPtr(:)
real(ESMF_KIND_R8), pointer :: farrayPtr(:,:)
real(ESMF_KIND_R8), pointer :: farrayPtr(:,:,:)

The arguments are:

grid
Grid to get the information from.
itemflag
The item to get the information for. Please see Section 31.2.2 for a list of valid items.
[staggerloc]
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
farrayPtr
The pointer to the item data.
[datacopyflag]
If not specified, default to ESMF_DATACOPY_REFERENCE, in this case farrayPtr is a reference to the data in the Grid item arrays. Please see Section 54.13 for further description and a list of valid values.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to the grid dimCount.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to the grid dimCount.
[exclusiveCount]
Upon return this holds the number of items in the exclusive region per dimension (i.e. exclusiveUBound-exclusiveLBound+1). exclusiveCount must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the stagger region. computationalLBound must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the upper bounds of the stagger region. exclusiveUBound must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalLBound]
Upon return this holds the lower bounds of the total region. totalLBound must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalUBound]
Upon return this holds the upper bounds of the total region. totalUBound must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalCount]
Upon return this holds the number of items in the total region per dimension (i.e. totalUBound-totalLBound+1). totalCount must be allocated to be of size equal to the grid dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.48 ESMF_GridGetItem - Get a Grid item and put into an Array


INTERFACE:

   ! Private name; call using ESMF_GridGetItem()
       subroutine ESMF_GridGetItemIntoArray(grid, itemflag,  staggerloc, &
         array, rc)
ARGUMENTS:
       type(ESMF_Grid),           intent(in)            :: grid
       type (ESMF_GridItem_Flag), intent(in)            :: itemflag
       type (ESMF_StaggerLoc),    intent(in),  optional :: staggerloc
       type(ESMF_Array),          intent(out)           :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                   intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method allows the user to get access to the ESMF Array holding item data at a particular stagger location. This is useful, for example, to set the item values. To have an Array to access, the item Array must have already been allocated, for example by ESMF_GridAddItem or ESMF_GridSetItem.

The arguments are:

grid
Grid to get the information from.
itemflag
The item from which to get the arrays. Please see Section 31.2.2 for a list of valid items.
[staggerloc]
The stagger location from which to get the arrays. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
array
An array into which to put the item information.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.49 ESMF_GridGetItem - Get information about an item at a particular stagger location


INTERFACE:

   ! Private name; call using ESMF_GridGetItem()
       subroutine ESMF_GridGetItemInfo(grid, itemflag, &
         staggerloc, isPresent, rc)
ARGUMENTS:
       type(ESMF_Grid),           intent(in)            :: grid
       type (ESMF_GridItem_Flag), intent(in)            :: itemflag
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc),    intent(in),  optional :: staggerloc
       logical,                   intent(out), optional :: isPresent
       integer,                   intent(out), optional :: rc
DESCRIPTION:

This method allows the user to get information about a given item on a given stagger.

The arguments are:

grid
Grid to get the information from.
itemflag
The item for which to get information. Please see Section 31.2.2 for a list of valid items.
[staggerloc]
The stagger location for which to get information. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[isPresent]
If .true. then an item of type itemflag has been added to this staggerloc.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.50 ESMF_GridGetItemBounds - Get DE-local item bounds from a Grid


INTERFACE:

       subroutine ESMF_GridGetItemBounds(grid, itemflag,  &
         staggerloc, localDE, &
         exclusiveLBound, exclusiveUBound, exclusiveCount, &
         computationalLBound, computationalUBound, computationalCount,  &
         totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       type (ESMF_GridItem_Flag), intent(in)         :: itemflag
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       integer,                intent(in),  optional :: localDE
       integer,        target, intent(out), optional :: exclusiveLBound(:)
       integer,        target, intent(out), optional :: exclusiveUBound(:)
       integer,        target, intent(out), optional :: exclusiveCount(:)
       integer,        target, intent(out), optional :: computationalLBound(:)
       integer,        target, intent(out), optional :: computationalUBound(:)
       integer,        target, intent(out), optional :: computationalCount(:)
       integer,        target, intent(out), optional :: totalLBound(:)
       integer,        target, intent(out), optional :: totalUBound(:)
       integer,        target, intent(out), optional :: totalCount(:)
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method gets information about the range of index space which a particular piece of item data occupies. In other words, this method returns the bounds of the item arrays. Note that unlike the output from the Array, these values also include the undistributed dimensions and are ordered to reflect the order of the indices in the item. So, for example, totalLBound and totalUBound should match the bounds of the Fortran array retrieved by ESMF_GridGetItem.

The arguments are:

grid
Grid to get the information from.
itemflag
The item to get the information for. Please see Section 31.2.2 for a list of valid items.
[staggerloc]
The stagger location to get the information for. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region. exclusiveLBound must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region. exclusiveUBound must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[exclusiveCount]
Upon return this holds the number of items, exclusiveUBound-exclusiveLBound+1, in the exclusive region per dimension. exclusiveCount must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalLBound]
Upon return this holds the lower bounds of the stagger region. computationalLBound must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalUBound]
Upon return this holds the upper bounds of the stagger region. computationalUBound must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[computationalCount]
Upon return this holds the number of items in the computational region per dimension (i.e. computationalUBound-computationalLBound+1). computationalCount must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalLBound]
Upon return this holds the lower bounds of the total region. totalLBound must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalUBound]
Upon return this holds the upper bounds of the total region. totalUBound must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[totalCount]
Upon return this holds the number of items in the total region per dimension (i.e. totalUBound-totalLBound+1). totalCount must be allocated to be of size equal to the item dimCount. Please see Section 31.3.19 for a description of the regions and their associated bounds and counts.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.51 ESMF_GridIsCreated - Check whether a Grid object has been created


INTERFACE:

   function ESMF_GridIsCreated(grid, rc)
RETURN VALUE:
     logical :: ESMF_GridIsCreated
ARGUMENTS:
     type(ESMF_Grid), intent(in)            :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,             intent(out), optional :: rc
DESCRIPTION:

Return .true. if the grid has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

grid
ESMF_Grid queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.52 ESMF_GridMatch - Check if two Grid objects match


INTERFACE:

   function ESMF_GridMatch(grid1, grid2, globalflag, rc)
RETURN VALUE:
     type(ESMF_GridMatch_Flag) :: ESMF_GridMatch
ARGUMENTS:
     type(ESMF_Grid),  intent(in)              :: grid1
     type(ESMF_Grid),  intent(in)              :: grid2
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,          intent(in),   optional  :: globalflag
     integer,          intent(out),  optional  :: rc
DESCRIPTION:

Check if grid1 and grid2 match. Returns a range of values of type ESMF_GridMatch indicating how closely the Grids match. For a description of the possible return values, please see 31.2.3. Please also note that by default this call is not collective and only returns the match for the piece of the Grids on the local PET. In this case, it is possible for this call to return a different match on different PETs for the same Grids. To do a global match operation set the globalflag argument to .true.. In this case, the call becomes collective across the current VM, ensuring the same result is returned on all PETs.

The arguments are:

grid1
ESMF_Grid object.
grid2
ESMF_Grid object.
[globalflag]
By default this flag is set to false. When it's set to true, the function performs the match check globally. In this case, the method becomes collective across the current VM.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.53 ESMF_GridRedist - Redistribute the coordinates of a Grid


INTERFACE:

     subroutine ESMF_GridRedist(srcGrid, dstGrid, routehandle, rc)
ARGUMENTS:
        type(ESMF_Grid),       intent(in)              :: srcGrid
        type(ESMF_Grid),       intent(inout)           :: dstGrid
        type(ESMF_RouteHandle),intent(inout)           :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,               intent(out),  optional  :: rc
DESCRIPTION:

This call is companion to the ESMF_GridCreate() that allows the user to copy an existing ESMF Grid, but with a new distribution. The ESMF_GridRedist() allows the user to repeatedly redistribute the coordinates from srcGrid to dstGrid.

The arguments are:

srcGrid
The source grid providing the coordinates.
srcGrid
The destination grid receiving the coordinates from srcGrid.
routehandle
The ESMF_RouteHandle object returned by the companion method ESMF_GridCreate() used to create dstGrid from srcGrid.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.54 ESMF_GridSetCoord - Set coordinates using Arrays


INTERFACE:

       subroutine ESMF_GridSetCoordFromArray(grid, coordDim, staggerloc, &
         array, rc)
ARGUMENTS:
       type(ESMF_Grid),        intent(in)            :: grid
       integer,                intent(in)            :: coordDim
       type (ESMF_StaggerLoc), intent(in),  optional :: staggerloc
       type(ESMF_Array),       intent(in)            :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method sets the passed in Array as the holder of the coordinate data for stagger location staggerloc and coordinate coord. This method can be used in place of ESMF_GridAddCoord(). In fact, if the Grid location already contains an Array for this coordinate, then this one replaces it. For this method to replace ESMF_GridAddCoord() and produce a valid set of coordinates, then this method must be used to set an Array for each coordDim ranging from 1 to the dimCount of the passed in Grid.

The arguments are:

grid
The grid to set the coord in.
coordDim
The coordinate dimension to put the data in (e.g. 1=x).
[staggerloc]
The stagger location into which to copy the arrays. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
array
An array to set the grid coordinate information from.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.55 ESMF_GridSetItem - Set an item using an Array


INTERFACE:

   ! Private name; call using ESMF_GridSetItem()
       subroutine ESMF_GridSetItemFromArray(grid, itemflag,  staggerloc, &
         array, rc)
ARGUMENTS:
       type(ESMF_Grid),           intent(in)            :: grid
       type (ESMF_GridItem_Flag), intent(in)            :: itemflag
       type (ESMF_StaggerLoc),    intent(in),  optional :: staggerloc
       type(ESMF_Array),          intent(in)            :: array
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                   intent(out), optional :: rc
STATUS:

DESCRIPTION:

This method sets the passed in Array as the holder of the item data for stagger location staggerloc and item itemflag. If the location already contains an Array, then this one overwrites it. This method can be used as a replacement for ESMF_GridAddItem().

The arguments are:

grid
The grid in which to set the array.
itemflag
The item into which to copy the arrays. Please see Section 31.2.2 for a list of valid items.
[staggerloc]
The stagger location into which to copy the arrays. Please see Section 31.2.6 for a list of predefined stagger locations. If not present, defaults to ESMF_STAGGERLOC_CENTER.
array
An array to set the grid item information from.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.6.56 ESMF_GridValidate - Validate Grid internals


INTERFACE:

   subroutine ESMF_GridValidate(grid, rc)
ARGUMENTS:
     type(ESMF_Grid), intent(in)              :: grid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,         intent(out),  optional  :: rc
STATUS:

DESCRIPTION:

Validates that the Grid is internally consistent. Note that one of the checks that the Grid validate does is the Grid status. Currently, the validate will return an error if the grid is not at least ESMF_GRIDSTATUS_COMPLETE. This means that if a Grid was created with the ESMF_GridEmptyCreate method, it must also have been finished with ESMF_GridEmptyComplete() to be valid. If a Grid was created with another create call it should automatically have the correct status level to pass the status part of the validate. The Grid validate at this time doesn't check for the presence or consistency of the Grid coordinates. The method returns an error code if problems are found.

The arguments are:

grid
Specified ESMF_Grid object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.7 Class API: StaggerLoc Methods

 

31.7.1 ESMF_StaggerLocGet - Get the value of one dimension of a StaggerLoc


INTERFACE:

   ! Private name; call using ESMF_StaggerLocGet() 
       subroutine ESMF_StaggerLocGetDim(staggerloc, dim, loc, &
            rc)
ARGUMENTS:
       type (ESMF_StaggerLoc), intent(in)  :: staggerloc
       integer,                intent(in)  :: dim
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, optional,      intent(out) :: loc
       integer, optional                   :: rc
DESCRIPTION:

Gets the position of a particular dimension of a cell staggerloc The argument loc will be only be 0,1. If loc is 0 it means the position should be in the center in that dimension. If loc is +1 then for the dimension, the position should be on the positive side of the cell. Please see Section 31.3.25 for diagrams.

The arguments are:

staggerloc
Stagger location for which to get information.
dim
Dimension for which to get information (1-7).
[loc]
Output position data (should be either 0,1).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.7.2 ESMF_StaggerLocSet - Set a StaggerLoc to a particular position in the cell


INTERFACE:

   ! Private name; call using ESMF_StaggerLocSet() 
      subroutine ESMF_StaggerLocSetAllDim(staggerloc, loc, rc)
ARGUMENTS:
       type (ESMF_StaggerLoc), intent(inout) :: staggerloc
       integer,                intent(in)    :: loc(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, optional                     :: rc
STATUS:

DESCRIPTION:

Sets a custom staggerloc to a position in a cell by using the array loc. The values in the array should only be 0,1. If loc(i) is 0 it ! means the position should be in the center in that dimension. If loc(i) is 1 then for dimension i, the position should be on the side of the cell. Please see Section 31.3.25 for diagrams and further discussion of custom stagger locations.

The arguments are:

staggerloc
Grid location to be initialized
loc
Array holding position data. Each entry in loc should only be 0 or 1. note that dimensions beyond those specified are set to 0.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.7.3 ESMF_StaggerLocSet - Set one dimension of a StaggerLoc to a particular position


INTERFACE:

   ! Private name; call using ESMF_StaggerLocSet() 
        subroutine ESMF_StaggerLocSetDim(staggerloc, dim, loc, &
             rc)
ARGUMENTS:
       type (ESMF_StaggerLoc), intent(inout) :: staggerloc
       integer,                intent(in)    :: dim
       integer,                intent(in)    :: loc
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, optional                     :: rc
STATUS:

DESCRIPTION:

Sets a particular dimension of a custom staggerloc to a position in a cell by using the variable loc. The variable loc should only be 0,1. If loc is 0 it means the position should be in the center in that dimension. If loc is +1 then for the dimension, the position should be on the positive side of the cell. Please see Section 31.3.25 for diagrams and further discussion of custom stagger locations.

The arguments are:

staggerloc
Stagger location to be initialized
dim
Dimension to be changed (1-7).
loc
Position data should be either 0,1.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.7.4 ESMF_StaggerLocString - Return a StaggerLoc as a string


INTERFACE:

       subroutine ESMF_StaggerLocString(staggerloc, string, &
            rc)
ARGUMENTS:
       type(ESMF_StaggerLoc), intent(in)  :: staggerloc
       character (len = *),   intent(out) :: string
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, optional,     intent(out) :: rc
STATUS:

DESCRIPTION:

Return an ESMF_StaggerLoc as a printable string.

The arguments are:

staggerloc
The ESMF_StaggerLoc to be turned into a string.
string
Return string.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

31.7.5 ESMF_StaggerLocPrint - Print StaggerLoc information


INTERFACE:

       subroutine ESMF_StaggerLocPrint(staggerloc, rc)
ARGUMENTS:
       type (ESMF_StaggerLoc), intent(in)  :: staggerloc
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer, optional,      intent(out) :: rc
STATUS:

DESCRIPTION:

Print the internal data members of an ESMF_StaggerLoc object.

The arguments are:

staggerloc
ESMF_StaggerLoc object as the method input
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32 LocStream Class

32.1 Description

A location stream (LocStream) can be used to represent the locations of a set of data points. For example, in the data assimilation world, LocStreams can be used to represent a set of observations. The values of the data points are stored within a Field or FieldBundle created using the LocStream.

The locations are generally described using Cartesian (x, y, z), or (lat, lon, radius) coordinates. The coordinates are stored using constructs called keys. A Key is essentially a list of point descriptors, one for each data point. They may hold other information besides the coordinates - a mask, for example. They may also hold a second set of coordinates. Keys are referenced by name - see 32.2.1 and 32.2.2 for specific keynames required in regridding. Each key must contain the same number of elements as there are data points in the LocStream. While there is no assumption in the ordering of the points, the order chosen must be maintained in each of the keys.

LocStreams can be very large. Data assimilation systems might use LocStreams with up to $10^{8}$ observations, so efficiency is critical. LocStreams can be created from file, see 32.4.14.

Common operations involving LocStreams are similar to those involving Grids. For example, LocStreams allow users to:

  1. Create a Field or FieldBundle on a LocStream
  2. Regrid data in Fields defined on LocStreams
  3. Redistribute data between Fields defined on LocStreams
  4. Gather or scatter a FieldBundle defined on a LocStream from/to a root DE
  5. Extract Fortran array from Field which was defined on a LocStream

A LocStream differs from a Grid in that no topological structure is maintained between the points (e.g. the class contains no information about which point is the neighbor of which other point).

A LocStream is similar to a Mesh in that both are collections of irregularly positioned points. However, the two structures differ because a Mesh also has connectivity: each data point represents either a center or corner of a cell. There is no requirement that the points in a LocStream have connectivity, in fact there is no requirement that any two points have any particular spatial relationship at all.

32.2 Constants


32.2.1 Coordinate keyNames

DESCRIPTION:
For ESMF to be able to use coordinates specified in a LocStream key (e.g. in regridding) they need to be named with the appropriate identifiers. The particular identifiers depend on the coordinate system (i.e. coordSys argument) used to create the LocStream containing the keys. ESMF regridding expects these keys to be of type ESMF_TYPEKIND_R8.

The valid values are:
Coordinate System dimension 1 dimension 2 dimension 3 (if used)
ESMF_COORDSYS_SPH_DEG ESMF:Lon ESMF:Lat ESMF:Radius
ESMF_COORDSYS_SPH_RAD ESMF:Lon ESMF:Lat ESMF:Radius
ESMF_COORDSYS_CART ESMF:X ESMF:Y ESMF:Z


32.2.2 Masking keyName

DESCRIPTION:
Points within a LocStream can be marked and then potentially ignored during certain operations, like regridding. This masking information must be contained in a key named with the appropriate identifier. ESMF regridding expects this key to be of type ESMF_TYPEKIND_I4.

The valid value is:

ESMF:Mask

32.3 Use and Examples

32.3.1 Create a LocStream with user allocated memory

The following is an example of creating a LocStream object. After creation, key data is added, and a Field is created to hold data (temperature) at each location.

   !-------------------------------------------------------------------
   ! Get parallel information. Here petCount is the total number of 
   ! running PETs, and localPet is the number of this particular PET.
   !-------------------------------------------------------------------
   call ESMF_VMGet(vm, localPet=localPet, petCount=petCount, rc=rc)

   !-------------------------------------------------------------------
   ! Allocate and set example location information. Locations on a PET
   ! are wrapped around sphere. Each PET occupies a different latitude
   ! ranging from +50.0 to -50.0.
   !-------------------------------------------------------------------
   numLocations = 20
   allocate(lon(numLocations))
   allocate(lat(numLocations))

   do i=1,numLocations
      lon(i)=360.0*i/numLocations
      lat(i)=100*REAL(localPet,ESMF_KIND_R8)/REAL(petCount,ESMF_KIND_R8)-50.0
   enddo

   !-------------------------------------------------------------------
   ! Allocate and set example Field data
   !-------------------------------------------------------------------
   allocate(temperature(numLocations))

   do i=1,numLocations
      temperature(i)= 300 - abs(lat(i))
   enddo

   !-------------------------------------------------------------------
   ! Create the LocStream:  Allocate space for the LocStream object, 
   ! define the number and distribution of the locations. 
   !-------------------------------------------------------------------
   locstream=ESMF_LocStreamCreate(name="Temperature Measurements",   &
                                  localCount=numLocations, &
                                  coordSys=ESMF_COORDSYS_SPH_DEG,   &
                                  rc=rc)

   !-------------------------------------------------------------------
   ! Add key data, referencing a user data pointer. By changing the 
   ! datacopyflag to ESMF_DATACOPY_VALUE an internally allocated copy of the 
   ! user data may also be set.  
   !-------------------------------------------------------------------
   call ESMF_LocStreamAddKey(locstream,              &
                             keyName="ESMF:Lat",     &
                             farray=lat,             &
                             datacopyflag=ESMF_DATACOPY_REFERENCE, &
                             keyUnits="Degrees",     &
                             keyLongName="Latitude", rc=rc)

   call ESMF_LocStreamAddKey(locstream,              &
                             keyName="ESMF:Lon",     &
                             farray=lon,             &
                             datacopyflag=ESMF_DATACOPY_REFERENCE, &
                             keyUnits="Degrees",     &
                             keyLongName="Longitude", rc=rc)

   !-------------------------------------------------------------------
   ! Create a Field on the Location Stream. In this case the 
   ! Field is created from a user array, but any of the other
   ! Field create methods (e.g. from ArraySpec) would also apply.
   !-------------------------------------------------------------------       
   field_temperature=ESMF_FieldCreate(locstream,   &
                                   temperature, &
                                   name="temperature", &
                                   rc=rc)

32.3.2 Create a LocStream with internally allocated memory

The following is an example of creating a LocStream object. After creation, key data is internally allocated, the pointer is retrieved, and the data is set. A Field is also created on the LocStream to hold data (temperature) at each location.

   !-------------------------------------------------------------------
   ! Get parallel information. Here petCount is the total number of 
   ! running PETs, and localPet is the number of this particular PET.
   !-------------------------------------------------------------------
   call ESMF_VMGet(vm, localPet=localPet, petCount=petCount, rc=rc)

   numLocations = 20

   !-------------------------------------------------------------------
   ! Create the LocStream:  Allocate space for the LocStream object, 
   ! define the number and distribution of the locations. 
   !-------------------------------------------------------------------
   locstream=ESMF_LocStreamCreate(name="Temperature Measurements",   &
                                  localCount=numLocations, &
                                  coordSys=ESMF_COORDSYS_SPH_DEG,   &
                                  rc=rc)

   !-------------------------------------------------------------------
   ! Add key data (internally allocating memory).
   !-------------------------------------------------------------------
   call ESMF_LocStreamAddKey(locstream,                    &
                             keyName="ESMF:Lat",           &
                             KeyTypeKind=ESMF_TYPEKIND_R8, &
                             keyUnits="Degrees",           &
                             keyLongName="Latitude", rc=rc)

   call ESMF_LocStreamAddKey(locstream,                    &
                             keyName="ESMF:Lon",           &
                             KeyTypeKind=ESMF_TYPEKIND_R8, &
                             keyUnits="Degrees",           &
                             keyLongName="Longitude", rc=rc)

   !-------------------------------------------------------------------
   ! Get key data. 
   !-------------------------------------------------------------------
   call ESMF_LocStreamGetKey(locstream,                    &
                             keyName="ESMF:Lat",           &
                             farray=lat,                   &
                             rc=rc)

   call ESMF_LocStreamGetKey(locstream,                    &
                             keyName="ESMF:Lon",           &
                             farray=lon,                   &
                             rc=rc)

   !-------------------------------------------------------------------
   ! Set example location information. Locations on a PET are wrapped 
   ! around sphere. Each PET occupies a different latitude ranging 
   ! from +50.0 to -50.0.
   !-------------------------------------------------------------------
   do i=1,numLocations
      lon(i)=360.0*i/numLocations
      lat(i)=100*REAL(localPet,ESMF_KIND_R8)/REAL(petCount,ESMF_KIND_R8)-50.0
   enddo


   !-------------------------------------------------------------------
   ! Allocate and set example Field data
   !-------------------------------------------------------------------
   allocate(temperature(numLocations))
   do i=1,numLocations
      temperature(i)= 300 - abs(lat(i))
   enddo

   !-------------------------------------------------------------------
   ! Create a Field on the Location Stream. In this case the 
   ! Field is created from a user array, but any of the other
   ! Field create methods (e.g. from ArraySpec) would also apply.
   !-------------------------------------------------------------------    
   field_temperature=ESMF_FieldCreate(locstream,   &
                                 temperature, &
                                 name="temperature", &
                                 rc=rc)

32.3.3 Create a LocStream with a distribution based on a Grid

The following is an example of using the LocStream create from background Grid capability. Using this capability, the newly created LocStream is a copy of the old LocStream, but with a new distribution. The new LocStream is distributed such that if the coordinates of a location in the LocStream lie within a Grid cell, then that location is put on the same PET as the Grid cell.

   !-------------------------------------------------------------------
   ! Get parallel information. Here petCount is the total number of 
   ! running PETs, and localPet is the number of this particular PET.
   !-------------------------------------------------------------------
   call ESMF_VMGet(vm, localPet=localPet, petCount=petCount, rc=rc)

   !-------------------------------------------------------------------
   ! Create the LocStream:  Allocate space for the LocStream object, 
   ! define the number and distribution of the locations. 
   !-------------------------------------------------------------------
   numLocations = 20
   locstream=ESMF_LocStreamCreate(name="Temperature Measurements",   &
                                  localCount=numLocations, &
                                  coordSys=ESMF_COORDSYS_SPH_DEG,   &
                                  rc=rc)

   !-------------------------------------------------------------------
   ! Add key data (internally allocating memory).
   !-------------------------------------------------------------------
   call ESMF_LocStreamAddKey(locstream,                    &
                             keyName="ESMF:Lon",           &
                             KeyTypeKind=ESMF_TYPEKIND_R8, &
                             keyUnits="Degrees",           &
                             keyLongName="Longitude", rc=rc)

   call ESMF_LocStreamAddKey(locstream,                    &
                             keyName="ESMF:Lat",           &
                             KeyTypeKind=ESMF_TYPEKIND_R8, &
                             keyUnits="Degrees",           &
                             keyLongName="Latitude", rc=rc)

   !-------------------------------------------------------------------
   ! Get Fortran arrays which hold the key data, so that it can be set. 
   !-------------------------------------------------------------------
   call ESMF_LocStreamGetKey(locstream,                    &
                             keyName="ESMF:Lon",           &
                             farray=lon,                   &
                             rc=rc)

   call ESMF_LocStreamGetKey(locstream,                    &
                             keyName="ESMF:Lat",           &
                             farray=lat,                   &
                             rc=rc)

   !-------------------------------------------------------------------
   ! Set example location information. Locations on a PET are wrapped 
   ! around sphere. Each PET occupies a different latitude ranging 
   ! from +50.0 to -50.0.
   !-------------------------------------------------------------------
   do i=1,numLocations
      lon(i)=360.0*i/numLocations
      lat(i)=100*REAL(localPet,ESMF_KIND_R8)/REAL(petCount,ESMF_KIND_R8)-50.0
   enddo

   !-------------------------------------------------------------------
   ! Create a Grid to use as the background. The Grid is 
   ! GridLonSize by GridLatSize with the default distribution 
   ! (The first dimension split across the PETs). The coordinate range
   ! is  0 to 360 in longitude and -90 to 90 in latitude. Note that we 
   ! use indexflag=ESMF_INDEX_GLOBAL for the Grid creation. At this time 
   ! this is required for a Grid to be usable as a background Grid.
   ! Note that here the points are treated as cartesian.
   !-------------------------------------------------------------------
   grid=ESMF_GridCreateNoPeriDim(maxIndex=(/GridLonSize,GridLatSize/), &
                                 coordSys=ESMF_COORDSYS_SPH_DEG,       &
                                 indexflag=ESMF_INDEX_GLOBAL,          &
                                 rc=rc)

   !-------------------------------------------------------------------
   ! Allocate the corner stagger location in which to put the coordinates. 
   ! (The corner stagger must be used for the Grid to be usable as a
   !  background Grid.)
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid, staggerloc=ESMF_STAGGERLOC_CORNER, rc=rc)

   !-------------------------------------------------------------------
   ! Get access to the Fortran array pointers that hold the Grid 
   ! coordinate information and then set the coordinates to be uniformly 
   ! distributed around the globe. 
   !-------------------------------------------------------------------
   call ESMF_GridGetCoord(grid,                                  &
                          staggerLoc=ESMF_STAGGERLOC_CORNER,     &
                          coordDim=1, computationalLBound=clbnd, &
                          computationalUBound=cubnd,             & 
                          farrayPtr=farrayPtrLonC, rc=rc)

   call ESMF_GridGetCoord(grid,                                  &
                         staggerLoc=ESMF_STAGGERLOC_CORNER,      &
                          coordDim=2, farrayPtr=farrayPtrLatC, rc=rc)

   do i1=clbnd(1),cubnd(1)
   do i2=clbnd(2),cubnd(2)
      ! Set Grid longitude coordinates as 0 to 360
      farrayPtrLonC(i1,i2) = REAL(i1-1)*360.0/REAL(GridLonSize)

      ! Set Grid latitude coordinates as -90 to 90
      farrayPtrLatC(i1,i2) = -90. + REAL(i2-1)*180.0/REAL(GridLatSize) + &
                                      0.5*180.0/REAL(GridLatSize)
   enddo
   enddo


   !-------------------------------------------------------------------
   ! Create newLocstream on the background Grid using the 
   ! "Lon" and "Lat" keys as the coordinates for the entries in 
   ! locstream. The entries in newLocstream with coordinates (lon,lat)
   ! are on the same PET as the piece of grid which contains (lon,lat). 
   !-------------------------------------------------------------------
   newLocstream=ESMF_LocStreamCreate(locstream, &
                  background=grid, rc=rc)


   !-------------------------------------------------------------------
   ! A Field can now be created on newLocstream and 
   ! ESMF_FieldRedist() can be used to move data between Fields built 
   ! on locstream and Fields built on newLocstream.
   !-------------------------------------------------------------------

32.3.4 Regridding from a Grid to a LocStream

The following is an example of how a LocStream object can be used in regridding.

   !-------------------------------------------------------------------
   ! Create a global Grid to use as the regridding source. The Grid is 
   ! GridLonSize by GridLatSize with the default distribution 
   ! (The first dimension split across the PETs). The coordinate range
   ! is  0 to 360 in longitude and -90 to 90 in latitude. Note that we 
   ! use indexflag=ESMF_INDEX_GLOBAL for the Grid creation to calculate
   ! coordinates across PETs.
   !-------------------------------------------------------------------
   grid=ESMF_GridCreate1PeriDim(maxIndex=(/GridLonSize,GridLatSize/), &
                                coordSys=ESMF_COORDSYS_SPH_DEG,       &
                                indexflag=ESMF_INDEX_GLOBAL,          &
                                rc=rc)

   !-------------------------------------------------------------------
   ! Allocate the center stagger location in which to put the coordinates. 
   !-------------------------------------------------------------------
   call ESMF_GridAddCoord(grid, staggerloc=ESMF_STAGGERLOC_CENTER, rc=rc)

   !-------------------------------------------------------------------
   ! Get access to the Fortran array pointers that hold the Grid 
   ! coordinate information.
   !------------------------------------------------------------------- 
   ! Longitudes 
   call ESMF_GridGetCoord(grid,                                  &
                          staggerLoc=ESMF_STAGGERLOC_CENTER,     &
                          coordDim=1, computationalLBound=clbnd, &
                          computationalUBound=cubnd,             &
                          farrayPtr=farrayPtrLonC, rc=rc)

   ! Latitudes
   call ESMF_GridGetCoord(grid,                                  &
                          staggerLoc=ESMF_STAGGERLOC_CENTER,     &
                          coordDim=2, computationalLBound=clbnd, &
                          computationalUBound=cubnd,             &
                          farrayPtr=farrayPtrLatC, rc=rc)

   !-------------------------------------------------------------------
   ! Create a source Field to hold the data to be regridded to the 
   ! destination
   !-------------------------------------------------------------------
   srcField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_R8,   &
                               staggerloc=ESMF_STAGGERLOC_CENTER, &
                               name="source", rc=rc)

   !-------------------------------------------------------------------
   ! Set the Grid coordinates to be uniformly distributed around the globe. 
   !-------------------------------------------------------------------
   do i1=clbnd(1),cubnd(1)
   do i2=clbnd(2),cubnd(2)
      ! Set Grid longitude coordinates as 0 to 360
      farrayPtrLonC(i1,i2) = REAL(i1-1)*360.0/REAL(GridLonSize)

      ! Set Grid latitude coordinates as -90 to 90
      farrayPtrLatC(i1,i2) = -90. + REAL(i2-1)*180.0/REAL(GridLatSize) + &
                                       0.5*180.0/REAL(GridLatSize)

   enddo
   enddo

   !-------------------------------------------------------------------
   ! Set the number of points the destination LocStream will have
   ! depending on the PET. 
   !-------------------------------------------------------------------
   if (petCount .eq. 1) then
     numLocationsOnThisPet=7
   else
     if (localpet .eq. 0) then
       numLocationsOnThisPet=2
     else if (localpet .eq. 1) then
       numLocationsOnThisPet=2
     else if (localpet .eq. 2) then
       numLocationsOnThisPet=2
     else if (localpet .eq. 3) then
       numLocationsOnThisPet=1
     endif
   endif

   !-------------------------------------------------------------------
   ! Create the LocStream:  Allocate space for the LocStream object,
   ! define the number of locations on this PET. 
   !-------------------------------------------------------------------
   locstream=ESMF_LocStreamCreate(name="Test Data",                 &
                                  localCount=numLocationsOnThisPet, &
                                  coordSys=ESMF_COORDSYS_SPH_DEG,   &
                                  rc=rc)

   !-------------------------------------------------------------------
   ! Add key data to LocStream(internally allocating memory).
   !-------------------------------------------------------------------
   call ESMF_LocStreamAddKey(locstream,                    &
                             keyName="ESMF:Lat",           &
                             KeyTypeKind=ESMF_TYPEKIND_R8, &
                             keyUnits="degrees",           &
                             keyLongName="Latitude", rc=rc)

   call ESMF_LocStreamAddKey(locstream,                    &
                             keyName="ESMF:Lon",           &
                             KeyTypeKind=ESMF_TYPEKIND_R8, &
                             keyUnits="degrees",           &
                             keyLongName="Longitude", rc=rc)

   !-------------------------------------------------------------------
   ! Get access to the Fortran array pointers that hold the key data.
   !-------------------------------------------------------------------
   ! Longitudes
   call ESMF_LocStreamGetKey(locstream,           &
                             keyName="ESMF:Lon",  &
                             farray=lonArray,     &
                             rc=rc)

   ! Latitudes
   call ESMF_LocStreamGetKey(locstream,           &
                             keyName="ESMF:Lat",  &
                             farray=latArray,     &
                             rc=rc)

   !-------------------------------------------------------------------
   ! Set coordinates in key arrays depending on the PET.
   ! For this example use an arbitrary set of points around globe.  
   !-------------------------------------------------------------------
   if (petCount .eq. 1) then
     latArray = (/-87.75, -56.25, -26.5, 0.0, 26.5, 56.25, 87.75 /)
     lonArray = (/51.4, 102.8, 154.2, 205.6, 257.0, 308.4, 359.8 /)
   else
     if (localpet .eq. 0) then
       latArray = (/ -87.75, -56.25 /)
       lonArray = (/ 51.4, 102.8 /)
     else if (localpet .eq.1) then
       latArray = (/ -26.5, 0.0 /)
       lonArray = (/ 154.2, 205.6 /)
     else if (localpet .eq.2) then
       latArray = (/ 26.5, 56.25 /)
       lonArray = (/ 257.0, 308.4 /)
     else if (localpet .eq.3) then
       latArray = (/ 87.75 /)
       lonArray = (/ 359.8 /)
     endif
   endif

   !-------------------------------------------------------------------
   ! Create the destination Field on the LocStream to hold the 
   ! result of the regridding. 
   !-------------------------------------------------------------------
   dstField = ESMF_FieldCreate(locstream, typekind=ESMF_TYPEKIND_R8, &
                               name="dest", rc=rc)

   !-------------------------------------------------------------------
   ! Calculate the RouteHandle that represents the regridding from 
   ! the source to destination Field using the Bilinear regridding method.
   !-------------------------------------------------------------------
   call ESMF_FieldRegridStore( srcField=srcField,                       &
                               dstField=dstField,                       &
                               routeHandle=routeHandle,                 &
                               regridmethod=ESMF_REGRIDMETHOD_BILINEAR, &
                               rc=rc)

   !-------------------------------------------------------------------
   ! Regrid from srcField to dstField
   !-------------------------------------------------------------------
   ! Can loop here regridding from srcField to dstField as src data changes
   ! do i=1,...

        ! (Put data into srcField)

        !-------------------------------------------------------------------
        ! Use the RouteHandle to regrid data from srcField to dstField.
        !-------------------------------------------------------------------
        call ESMF_FieldRegrid(srcField, dstField, routeHandle, rc=rc)

        ! (Can now use the data in dstField)

   ! enddo

   !-------------------------------------------------------------------
   ! Now that we are done, release the RouteHandle freeing its memory. 
   !-------------------------------------------------------------------
   call ESMF_FieldRegridRelease(routeHandle, rc=rc)

32.4 Class API

32.4.1 ESMF_LocStreamAssignment(=) - LocStream assignment


INTERFACE:

     interface assignment(=)
     locstream1 = locstream2
ARGUMENTS:
     type(ESMF_LocStream) :: locstream1
     type(ESMF_LocStream) :: locstream2
STATUS:

DESCRIPTION:

Assign locstream1 as an alias to the same ESMF LocStream object in memory as locstream2. If locstream2 is invalid, then locstream1 will be equally invalid after the assignment.

The arguments are:

locstream1
The ESMF_LocStream object on the left hand side of the assignment.
locstream2
The ESMF_LocStream object on the right hand side of the assignment.

32.4.2 ESMF_LocStreamOperator(==) - LocStream equality operator


INTERFACE:

   interface operator(==)
     if (locstream1 == locstream2) then ... endif
               OR
     result = (locstream1 == locstream2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_LocStream), intent(in) :: locstream1
     type(ESMF_LocStream), intent(in) :: locstream2
STATUS:

DESCRIPTION:

Test whether locstream1 and locstream2 are valid aliases to the same ESMF LocStream object in memory. For a more general comparison of two ESMF LocStreams, going beyond the simple alias test, the ESMF_LocStreamMatch() function (not yet implemented) must be used.

The arguments are:

locstream1
The ESMF_LocStream object on the left hand side of the equality operation.
locstream2
The ESMF_LocStream object on the right hand side of the equality operation.

32.4.3 ESMF_LocStreamOperator(/=) - LocStream not equal operator


INTERFACE:

   interface operator(/=)
     if (locstream1 /= locstream2) then ... endif
               OR
     result = (locstream1 /= locstream2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_LocStream), intent(in) :: locstream1
     type(ESMF_LocStream), intent(in) :: locstream2
STATUS:

DESCRIPTION:

Test whether locstream1 and locstream2 are not valid aliases to the same ESMF LocStream object in memory. For a more general comparison of two ESMF LocStreams, going beyond the simple alias test, the ESMF_LocStreamMatch() function (not yet implemented) must be used.

The arguments are:

locstream1
The ESMF_LocStream object on the left hand side of the non-equality operation.
locstream2
The ESMF_LocStream object on the right hand side of the non-equality operation.

32.4.4 ESMF_LocStreamAddKey - Add a key Array and allocate the internal memory


INTERFACE:

   ! Private name; call using ESMF_LocStreamAddKey()
   subroutine ESMF_LocStreamAddKeyAlloc(locstream, keyName, &
                keyTypeKind, keyUnits, keyLongName, rc)
ARGUMENTS:
     type(ESMF_Locstream),     intent(in)            :: locstream
     character (len=*),        intent(in)            :: keyName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_TypeKind_Flag), intent(in),  optional :: keyTypeKind
     character (len=*),        intent(in),  optional :: keyUnits 
     character (len=*),        intent(in),  optional :: keyLongName 
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Add a key to a locstream with a required keyName. Once a key has been added, a pointer to its internally allocated memory can be retrieved and used to set key values.

The arguments are:

locstream
The ESMF_LocStream object to add key to.
keyName
The name of the key to add.
[keyTypeKind]
The type/kind of the key data. If not specified then the type/kind will default to 8 byte reals.
[keyUnits]
The units of the key data. If not specified, then the item remains blank.
[keyLongName]
The long name of the key data. If not specified, then the item remains blank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.5 ESMF_LocStreamAddKey - Add a key Array


INTERFACE:

   ! Private name; call using ESMF_LocStreamAddKey()
   subroutine ESMF_LocStreamAddKeyArray(locstream, keyName, keyArray, &
                destroyKey, keyUnits, keyLongName, rc)
ARGUMENTS:
     type(ESMF_Locstream), intent(in)             :: locstream
     character (len=*),    intent(in)             :: keyName
     type(ESMF_Array),     intent(in)             :: keyArray
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,              intent(in),  optional  :: destroyKey
     character (len=*),    intent(in),  optional  :: keyUnits 
     character (len=*),    intent(in),  optional  :: keyLongName 
     integer,              intent(out), optional  :: rc
DESCRIPTION:

Add a key to a locstream with a required keyName and a required ESMF_Array. The user is responsible for the creation of the ESMF_Array that will hold the key values.

The arguments are:

locstream
The ESMF_LocStream object to add key to.
keyName
The name of the key to add.
keyArray
An ESMF Array which contains the key data
[destroyKey]
if .true. destroy this key array when the locstream is destroyed. Defaults to .false.
[keyUnits]
The units of the key data. If not specified, then the item remains blank.
[keyLongName]
The long name of the key data. If not specified, then the item remains blank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.6 ESMF_LocStreamAddKey - Add a key Array created around user memory


INTERFACE:

   ! Private name; call using ESMF_LocStreamAddKey()
    subroutine ESMF_LocStreamAddKeyI4(locstream, keyName, farray, &
                 datacopyflag, keyUnits, keyLongName, rc)
ARGUMENTS:
      type(ESMF_Locstream), intent(in) :: locstream
      character (len=*), intent(in) :: keyName
      <farray>
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
      type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
      character (len=*), intent(in), optional :: keyUnits
      character (len=*), intent(in), optional :: keyLongName
      integer, intent(out), optional :: rc
DESCRIPTION:

Add a key to a locstream with a required keyName and a required Fortran array. The user is responsible for the creation of the Fortran array that will hold the key values, including the maintenance of any allocated memory.

Supported values for <farray> are:

integer(ESMF_KIND_I4), intent(in) :: farray(:)
real(ESMF_KIND_R4), intent(in) :: farray(:)
real(ESMF_KIND_R8), intent(in) :: farray(:)

The arguments are:

locstream
The ESMF_LocStream object to add key to.
keyName
The name of the key to add.
farray
Valid native Fortran array, i.e. memory must be associated with the actual argument. The type/kind/rank information of farray will be used to set the key Array's properties accordingly.
[datacopyflag]
Specifies whether the Array object will reference the memory allocation provided by farray directly or will copy the data from farray into a new memory allocation. Valid options are ! ESMF_DATACOPY_REFERENCE (default) or ESMF_DATACOPY_VALUE. Depending on the specific situation the ESMF_DATACOPY_REFERENCE option may be unsafe when specifying an array slice for farray.
[keyUnits]
The units of the key data. If not specified, then the item remains blank.
[keyLongName]
The long name of the key data. If not specified, then the item remains blank.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.7 ESMF_LocStreamCreate - Create a new LocStream by projecting onto a Grid


INTERFACE:

       ! Private name; call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateByBkgGrid(locstream, &
                  background, maskValues, &
                  unmappedaction, name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateByBkgGrid
ARGUMENTS:
       type(ESMF_LocStream),           intent(in)            :: locstream
       type(ESMF_Grid),                intent(in)            :: background
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer(ESMF_KIND_I4),          intent(in),  optional :: maskValues(:)
       type(ESMF_UnmappedAction_Flag), intent(in),  optional :: unmappedaction
       character (len=*),              intent(in),  optional :: name
       integer,                        intent(out), optional :: rc
DESCRIPTION:

Create an location stream from an existing one in accordance with the distribution of the background Grid. The entries in the new location stream are redistributed, so that they lie on the same PET as the piece of Grid which contains the coordinates of the entries. The coordinates of the entries are the data in the keys named ESMF:Lon, ESMF:Lat, ESMF:Radius in the case of a spherical system and ESMF:X, ESMF:Y, ESMF:Z for cartesian. To copy data in Fields or FieldBundles built on locstream to the new one simply use ESMF_FieldRedist() or ESMF_FieldBundleRedist().

The arguments are:

locstream
Location stream from which the new location stream is to be created
background
Background Grid which determines the distribution of the entries in the new location stream. The background Grid Note also that this subroutine uses the corner stagger location in the Grid for determining where a point lies, because this is the stagger location which fully contains the cell. A Grid must have coordinate data in this stagger location to be used in this subroutine. For a 2D Grid this stagger location is ESMF_STAGGERLOC_CORNER for a 3D Grid this stagger location is ESMF_STAGGERLOC_CORNER_VFACE. Note that currently the background Grid also needs to have been created with indexflag=ESMF_INDEX_GLOBAL to be usable here.
[maskValues]
List of values that indicate a background grid point should be masked out. If not specified, no masking will occur.
[unmappedaction]
Specifies what should happen if there are destination points that can't be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR. [NOTE: the unmappedaction=ESMF_UNMAPPEDACTION_IGNORE option is currently not implemented.]
[name]
Name of the resulting location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.8 ESMF_LocStreamCreate - Create a new LocStream by projecting onto a Mesh


INTERFACE:

       ! Private name; call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateByBkgMesh(locstream, &
                  background, unmappedaction, name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateByBkgMesh
ARGUMENTS:
       type(ESMF_LocStream),           intent(in)           :: locstream
       type(ESMF_Mesh),                intent(in)           :: background
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_UnmappedAction_Flag), intent(in), optional :: unmappedaction
       character (len=*),              intent(in), optional :: name
       integer,                        intent(out),optional :: rc
DESCRIPTION:

Create an location stream from an existing one in accordance with the distribution of the background Mesh. The entries in the new location stream are redistributed, so that they lie on the same PET as the piece of Mesh which contains the coordinates of the entries. The coordinates of the entries are the data in the keys named ESMF:Lon, ESMF:Lat, ESMF:Radius in the case of a spherical system and ESMF:X, ESMF:Y, ESMF:Z for cartesian. To copy data in Fields or FieldBundles built on locstream to the new one simply use ESMF_FieldRedist() or ESMF_FieldBundleRedist().

The arguments are:

locstream
Location stream from which the new location stream is to be created
background
Background Mesh which determines the distribution of entries in the new locatiion stream.
[unmappedaction]
Specifies what should happen if there are destination points that can't be mapped to a source cell. Please see Section 54.62 for a list of valid options. If not specified, unmappedaction defaults to ESMF_UNMAPPEDACTION_ERROR. [NOTE: the unmappedaction=ESMF_UNMAPPEDACTION_IGNORE option is currently not implemented.]
[name]
Name of the resulting location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.9 ESMF_LocStreamCreate - Create a new LocStream from a distgrid


INTERFACE:

       ! Private name: call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateFromDG(distgrid, &
         indexflag, coordSys, name, vm, rc )
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateFromDG
ARGUMENTS:
       type(ESMF_DistGrid),      intent(in)            :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_Index_Flag),    intent(in),  optional :: indexflag    
       type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
       character (len=*),        intent(in),  optional :: name
       type(ESMF_VM),            intent(in),  optional :: vm
       integer,                  intent(out), optional :: rc
DESCRIPTION:

Allocates memory for a new ESMF_LocStream object, constructs its internal derived types.

The arguments are:

distgrid
Distgrid specifying size and distribution. Only 1D distgrids are allowed.
[indexflag]
Flag that indicates how the DE-local indices are to be defined. Defaults to ESMF_INDEX_DELOCAL, which indicates that the index range on each DE starts at 1. See Section 54.29 for the full range of options.
[coordSys]
The coordinate system of the location stream coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[name]
Name of the location stream
[vm]
If present, the LocStream object is created on the specified ESMF_VM object. The default is to create on the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.10 ESMF_LocStreamCreate - Create a new LocStream from an irregular distribution


INTERFACE:

       ! Private name: call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateIrreg(minIndex, countsPerDE, &
                   indexflag, coordSys, name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateIrreg
ARGUMENTS:
       integer, intent(in), optional                   :: minIndex
       integer, intent(in)                             :: countsPerDE(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_Index_Flag), intent(in), optional     :: indexflag
       type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
       character (len=*), intent(in), optional         :: name
       integer, intent(out), optional                  :: rc
DESCRIPTION:

Allocates memory for a new ESMF_LocStream object, constructs its internal derived types. The ESMF_DistGrid is set up, indicating how the LocStream is distributed.

The arguments are:

[minIndex]
If indexflag=ESMF_INDEX_DELOCAL, this setting is used to indicate the number to start the index ranges at. If not present, defaults to 1.
countsPerDE
This array has an element for each DE, specifying the number of locations for that DE.
[indexflag]
Flag that indicates how the DE-local indices are to be defined. Defaults to ESMF_INDEX_DELOCAL, which indicates that the index range on each DE starts at 1. See Section 54.29 for the full range of options.
[coordSys]
The coordinate system of the location stream coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[name]
Name of the location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.11 ESMF_LocStreamCreate - Create a new LocStream from a local count


INTERFACE:

       ! Private name: call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateFromLocal(localCount, &
                   indexflag, coordSys, name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateFromLocal
ARGUMENTS:
       integer, intent(in)                             :: localCount
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_Index_Flag), intent(in), optional     :: indexflag
       type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
       character (len=*), intent(in), optional         :: name
       integer, intent(out), optional                  :: rc
DESCRIPTION:

Allocates memory for a new ESMF_LocStream object, constructs its internal derived types. The ESMF_DistGrid is set up, indicating how the LocStream is distributed. The assumed layout is one DE per PET.

The arguments are:

localCount
Number of grid cells to be distributed to this DE/PET.
[indexflag]
Flag that indicates how the DE-local indices are to be defined. Defaults to ESMF_INDEX_DELOCAL, which indicates that the index range on each DE starts at 1. See Section 54.29 for the full range of options.
[coordSys]
The coordinate system of the location stream coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[name]
Name of the location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.12 ESMF_LocStreamCreate - Create a new LocStream from an old one and a distgrid


INTERFACE:

       ! Private name: call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateFromNewDG(locstream, distgrid, &
            name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateFromNewDG
ARGUMENTS:
       type(ESMF_LocStream), intent(in)                :: locstream
       type(ESMF_DistGrid),  intent(in)                :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       character (len=*),    intent(in), optional      :: name
       integer,              intent(out), optional     :: rc
DESCRIPTION:

Create a new location stream that is a copy of an old one, but with a new distribution. The new distribution is given by a distgrid passed into the call. Key and other class information is copied from the old locstream to the new one. Information contained in Fields build on the location streams can be copied over by using the Field redistribution calls (e.g. ESMF_FieldRedistStore() and ESMF_FieldRedist()).

The arguments are:

locstream
Location stream from which the new location stream is to be created
distgrid
Distgrid for new distgrid
[name]
Name of the resulting location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.13 ESMF_LocStreamCreate - Create a new LocStream using a regular distribution


INTERFACE:

       ! Private name: call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateReg(regDecomp, decompFlag, &
                     minIndex, maxIndex, &
                     coordSys, indexflag, name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateReg
ARGUMENTS:
       integer,                  intent(in),  optional  :: regDecomp
       type(ESMF_Decomp_Flag),   intent(in),  optional  :: decompflag
       integer,                  intent(in),  optional  :: minIndex
       integer,                  intent(in)             :: maxIndex
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_CoordSys_Flag), intent(in),  optional  :: coordSys
       type(ESMF_Index_Flag),    intent(in),  optional  :: indexflag
       character (len=*),        intent(in),  optional  :: name
       integer,                  intent(out), optional  :: rc
DESCRIPTION:

Allocates memory for a new ESMF_LocStream object, constructs its internal derived types. The ESMF_DistGrid is set up, indicating how the LocStream is distributed.

The arguments are:

[regDecomp]
Specify into how many chunks to divide the locations. If not specified, defaults to the number of PETs.
[decompFlag]
Specify what to do with leftover locations after division. If not specified, defaults to ESMF_DECOMP_BALANCED. Please see Section 54.14 for a full description of the possible options.
[minIndex]
If indexflag=ESMF_INDEX_DELOCAL, this setting is used to indicate the number to start the index ranges at. If not present, defaults to 1.
maxIndex
The maximum index across all PETs.
[coordSys]
The coordinate system of the location stream coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[indexflag]
Flag that indicates how the DE-local indices are to be defined. Defaults to ESMF_INDEX_DELOCAL, which indicates that the index range on each DE starts at 1. See Section 54.29 for the full range of options.
[name]
Name of the location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.14 ESMF_LocStreamCreate - Create a new LocStream from a grid file


INTERFACE:

       ! Private name: call using ESMF_LocStreamCreate()
       function ESMF_LocStreamCreateFromFile(filename, &
            fileformat, varname, indexflag, centerflag, name, rc)
RETURN VALUE:
       type(ESMF_LocStream) :: ESMF_LocStreamCreateFromFile
ARGUMENTS:
       character (len=*),          intent(in)           :: filename
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       type(ESMF_FileFormat_Flag), intent(in), optional :: fileformat
       character(len=*),           intent(in), optional :: varname
       type(ESMF_Index_Flag),      intent(in), optional :: indexflag
       logical,                    intent(in), optional :: centerflag
       character (len=*),          intent(in), optional :: name
       integer,                    intent(out),optional :: rc
DESCRIPTION:

Create a new ESMF_LocStream object and add the coordinate keys and mask key to the LocStream using the coordinates defined in a grid file. Currently, it supports the SCRIP format, the ESMF unstructured grid format and the UGRID format. For a 2D or 3D grid in ESMF or UGRID format, it can construct the LocStream using either the center coordinates or the corner coordinates. For a SCRIP format grid file, the LocStream can only be constructed using the center coordinates. In addition, it supports 1D network topology in UGRID format. When construction a LocStream using a 1D UGRID, it always uses node coordinates (i.e., corner coordinates).

The arguments are:

filename
Name of grid file to be used to create the location stream.
[fileformat]
The file format. The valid options are ESMF_FILEFORMAT_SCRIP, ESMF_FILEFORMAT_ESMFMESH, and ESMF_FILEFORMAT_UGRID. Please see section 54.20 for a detailed description of the options. If not specified, the default is ESMF_FILEFORMAT_SCRIP.
[varname]
An optional variable name stored in the UGRID file to be used to generate the mask using the missing value of the data value of this variable. The first two dimensions of the variable has to be the the longitude and the latitude dimension and the mask is derived from the first 2D values of this variable even if this data is 3D, or 4D array. If not specified, no mask is used for a UGRID file.
[indexflag]
Flag that indicates how the DE-local indices are to be defined. Defaults to ESMF_INDEX_DELOCAL, which indicates that the index range on each DE starts at 1. See Section 54.29 for the full range of options.
[centerflag]
Flag that indicates whether to use the center coordinates to construct the location stream. If true, use center coordinates, otherwise, use the corner coordinates. If not specified, use center coordinates as default. For SCRIP files, only center coordinate is supported.
[name]
Name of the location stream
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.15 ESMF_LocStreamDestroy - Release resources associated with a LocStream


INTERFACE:

       subroutine ESMF_LocStreamDestroy(locstream, noGarbage, rc)
ARGUMENTS:
       type(ESMF_LocStream), intent(inout)          :: locstream 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       logical,              intent(in),   optional :: noGarbage
       integer,              intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Deallocate an ESMF_LocStream object and appropriate internal structures.

The arguments are:

locstream
locstream to destroy
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.16 ESMF_LocStreamGet - Return object-wide information from a LocStream


INTERFACE:

   subroutine ESMF_LocStreamGet(locstream, &
        distgrid, keyCount, keyNames, localDECount, indexflag, &
        coordSys, name, rc)
ARGUMENTS:
     type(ESMF_Locstream),         intent(in)            :: locstream
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DistGrid),          intent(out), optional :: distgrid
     integer,                      intent(out), optional :: keyCount
     character(len=ESMF_MAXSTR),                optional :: keyNames(:) 
     integer,                      intent(out), optional :: localDECount
     type(ESMF_Index_Flag),        intent(out), optional :: indexflag
     type(ESMF_CoordSys_Flag),     intent(out), optional :: coordSys
     character(len=*),             intent(out), optional :: name
     integer,                      intent(out), optional :: rc
DESCRIPTION:

Query an ESMF_LocStream for various information. All arguments after the locstream are optional.

The arguments are:

locstream
The ESMF_LocStream object to query.
[distgrid]
The ESMF_DistGrid object that describes
[keyCount]
Number of keys in the locstream.
[keyNames]
The names of the keys in the locstream. Keynames should be an array of character strings. The character strings should be of length ESMF_MAXSTR and the array's length should be at least keyCount.
[localDECount]
Number of DEs on this PET in the locstream.
[indexflag]
The indexflag for this indexflag.
[coordSys]
The coordinate system for this location stream.
[name]
Name of queried item.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.17 ESMF_LocStreamGetBounds - Get DE-local bounds of a LocStream


INTERFACE:

       subroutine ESMF_LocStreamGetBounds(locstream,   &
           localDE, exclusiveLBound, exclusiveUBound, exclusiveCount,   &
           computationalLBound, computationalUBound, computationalCount,&
           rc)
ARGUMENTS:
       type(ESMF_LocStream),   intent(in) :: locstream
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,                intent(in),  optional :: localDE
       integer,                intent(out), optional :: exclusiveLBound
       integer,                intent(out), optional :: exclusiveUBound
       integer,                intent(out), optional :: exclusiveCount
       integer,                intent(out), optional :: computationalLBound
       integer,                intent(out), optional :: computationalUBound
       integer,                intent(out), optional :: computationalCount
       integer, intent(out), optional :: rc
DESCRIPTION:

This method gets the bounds of a localDE for a locstream.

The arguments are:

locstream
LocStream to get the information from.
localDE
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region.
[exclusiveCount]
! Upon return this holds the number of items in the exclusive region
(i.e. exclusiveUBound-exclusiveLBound+1). exclusiveCount.
[computationalLBound]
Upon return this holds the lower bounds of the computational region.
[computationalUBound]
Upon return this holds the upper bounds of the computational region.
[computationalCount]
Upon return this holds the number of items in the computational region
(i.e. computationalUBound-computationalLBound+1). computationalCount.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.18 ESMF_LocStreamGetKey - Get an Array associated with a key


INTERFACE:

   ! Private name; call using ESMF_LocStreamGetKey()
   subroutine ESMF_LocStreamGetKeyArray(locstream, keyName, keyArray, &
        rc)
ARGUMENTS:
     type(ESMF_Locstream), intent(in)            :: locstream
     character (len=*),    intent(in)            :: keyName
     type(ESMF_Array),     intent(out)           :: keyArray
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(out), optional :: rc
DESCRIPTION:

Get ESMF Array associated with key.

The arguments are:

locstream
The ESMF_LocStream object to get key from.
keyName
The name of the key to get.
keyArray
Array associated with key.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.19 ESMF_LocStreamGetKey - Get info associated with a key


INTERFACE:

   ! Private name; call using ESMF_LocStreamGetKey()
   subroutine ESMF_LocStreamGetKeyInfo(locstream, keyName, &
        keyUnits, keyLongName, typekind, isPresent, rc)
ARGUMENTS:
     type(ESMF_Locstream),     intent(in)            :: locstream
     character (len=*),        intent(in)            :: keyName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character (len=*),        intent(out), optional :: keyUnits 
     character (len=*),        intent(out), optional :: keyLongName 
     type(ESMF_TypeKind_Flag), intent(out), optional :: typekind
     logical,                  intent(out), optional :: isPresent
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Get ESMF Array associated with key.

The arguments are:

locstream
The ESMF_LocStream object to get key from.
keyName
The name of the key to get.
[keyUnits]
The units of the key data. If not specified, then the item remains blank.
[keyLongName]
The long name of the key data. If not specified, then the item remains blank.
[typekind]
The typekind of the key data
[isPresent]
Whether or not the keyname is present
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.20 ESMF_LocStreamGetKey - Get a DE-local Fortran array pointer to key values


INTERFACE:

   ! Private name; call using ESMF_LocStreamGetKey()
        subroutine ESMF_LocStreamGetKey(locstream, keyName, &
            localDE, exclusiveLBound, exclusiveUBound, exclusiveCount,       &
            computationalLBound, computationalUBound, computationalCount,    &
            totalLBound, totalUBound, totalCount,                            &
            farray, datacopyflag, rc)
ARGUMENTS:
        type(ESMF_LocStream),   intent(in)            :: locstream
        character (len=*),      intent(in)            :: keyName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
        integer,                intent(in),  optional :: localDE
        integer,                intent(out), optional :: exclusiveLBound
        integer,                intent(out), optional :: exclusiveUBound
        integer,                intent(out), optional :: exclusiveCount
        integer,                intent(out), optional :: computationalLBound
        integer,                intent(out), optional :: computationalUBound
        integer,                intent(out), optional :: computationalCount
        integer,                intent(out), optional :: totalLBound
        integer,                intent(out), optional :: totalUBound
        integer,                intent(out), optional :: totalCount
        <farray>
        type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
        integer, intent(out), optional :: rc
DESCRIPTION:

This method gets a Fortran pointer to the piece of memory which holds the key data for a particular key on the given local DE. This is useful, for example, for setting the key values in a LocStream, or for reading the values.

Supported values for <farray> are:

integer(ESMF_KIND_I4), pointer :: farray(:)
real(ESMF_KIND_R4), pointer :: farray(:)
real(ESMF_KIND_R8), pointer :: farray(:)

The arguments are:

locstream
LocStream to get the information from.
keyName
The key to get the information from.
[localDE]
The local DE for which information is requested. [0,..,localDECount-1]. For localDECount==1 the localDE argument may be omitted, in which case it will default to localDE=0.
[exclusiveLBound]
Upon return this holds the lower bounds of the exclusive region.
[exclusiveUBound]
Upon return this holds the upper bounds of the exclusive region.
[exclusiveCount]
Upon return this holds the number of items in the exclusive region
(i.e. exclusiveUBound-exclusiveLBound+1). exclusiveCount.
[computationalLBound]
Upon return this holds the lower bounds of the computational region.
[computationalUBound]
Upon return this holds the upper bounds of the computational region.
[computationalCount]
Upon return this holds the number of items in the computational region
(i.e. computationalUBound-computationalLBound+1).
[totalLBound]
Upon return this holds the lower bounds of the total region.
[totalUBound]
Upon return this holds the upper bounds of the total region.
[totalCount]
Upon return this holds the number of items in the total region (i.e. totalUBound-totalLBound+1).
farray
The pointer to the coordinate data.
[datacopyflag]
If not specified, default to ESMF_DATACOPY_REFERENCE, in this case farray is a reference to the data in the Grid coordinate arrays. Please see Section 54.13 for further description and a list of valid values.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.21 ESMF_LocStreamIsCreated - Check whether a LocStream object has been created


INTERFACE:

   function ESMF_LocStreamIsCreated(locstream, rc)
RETURN VALUE:
     logical :: ESMF_LocStreamIsCreated
ARGUMENTS:
     type(ESMF_LocStream), intent(in)            :: locstream
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,             intent(out), optional :: rc
DESCRIPTION:

Return .true. if the locstream has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

locstream
ESMF_LocStream queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.22 ESMF_LocStreamPrint - Print the contents of a LocStream


INTERFACE:

       subroutine ESMF_LocStreamPrint(locstream, options, rc)
ARGUMENTS:
       type(ESMF_LocStream), intent(in)             :: locstream 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       character (len = *),  intent(in),   optional :: options
       integer,              intent(out),  optional :: rc
DESCRIPTION:

Prints information about the locstream to stdout. This subroutine goes through the internal data members of a locstream ! data type and prints information of each data member.

The arguments are:

locstream
[options]
Print options are not yet supported.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

32.4.23 ESMF_LocStreamValidate - Check validity of a LocStream


INTERFACE:

       subroutine ESMF_LocStreamValidate(locstream, rc)
ARGUMENTS:
       type(ESMF_LocStream), intent(in)            :: locstream 
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
       integer,              intent(out), optional :: rc
DESCRIPTION:

Validates that the locstream is internally consistent. Currently this method determines if the locstream is uninitialized or already destroyed.

The method returns an error code if problems are found.

The arguments are:

locstream
ESMF_LocStream to validate.
[rc]
Return code; equals ESMF_SUCCESS if the locstream is valid.

33 Mesh Class

33.1 Description

Unstructured grids are commonly used in the computational solution of partial differential equations. These are especially useful for problems that involve complex geometry, where using the less flexible structured grids can result in grid representation of regions where no computation is needed. Finite element and finite volume methods map naturally to unstructured grids and are used commonly in hydrology, ocean modeling, and many other applications.

In order to provide support for application codes using unstructured grids, the ESMF library provides a class for representing unstructured grids called the Mesh. Fields can be created on a Mesh to hold data. Fields created on a Mesh can also be used as either the source or destination or both of an interpolation (i.e. an ESMF_FieldRegridStore() call) which allows data to be moved between unstructured grids. This section describes the Mesh and how to create and use them in ESMF.


33.1.1 Mesh representation in ESMF

A Mesh in ESMF is constructed of nodes and elements.

A node, also known as a vertex or corner, is a part of a Mesh which represents a single point. Coordinate information is set in a node.

An element, also known as a cell, is a part of a mesh which represents a small region of space. Elements are described in terms of a connected set of nodes which represent locations along their boundaries.

Field data may be located on either the nodes or elements of a Mesh.


The dimension of a Mesh in ESMF is specified with two parameters: the parametric dimension and the spatial dimension.

The parametric dimension of a Mesh is the dimension of the topology of the Mesh. This can be thought of as the dimension of the elements which make up the Mesh. For example, a Mesh composed of triangles would have a parametric dimension of 2, whereas a Mesh composed of tetrahedra would have a parametric dimension of 3.

The spatial dimension of a Mesh is the dimension of the space the Mesh is embedded in. In other words, it is the number of coordinate dimensions needed to describe the location of the nodes making up the Mesh.

For example, a Mesh constructed of squares on a plane would have a parametric dimension of 2 and a spatial dimension of 2. If that same Mesh were used to represent the 2D surface of a sphere, then the Mesh would still have a parametric dimension of 2, but now its spatial dimension would be 3.

33.1.2 Supported Meshes

The range of Meshes supported by ESMF are defined by several factors: dimension, element types, and distribution.

ESMF currently only supports Meshes whose number of coordinate dimensions (spatial dimension) is 2 or 3. The dimension of the elements in a Mesh (parametric dimension) must be less than or equal to the spatial dimension, but also must be either 2 or 3. This means that a Mesh may be either 2D elements in 2D space, 3D elements in 3D space, or a manifold constructed of 2D elements embedded in 3D space.

ESMF supports a range of elements for each Mesh parametric dimension. For a parametric dimension of 2, the native supported element types are triangles and quadrilaterals. In addition to these, ESMF supports 2D polygons with any number of sides. Internally these are represented as sets of triangles, but to the user should behave like any other element. For a parametric dimension of 3, the supported element types are tetrahedrons and hexahedrons. See Section 33.2.1 for diagrams of these. The Mesh supports any combination of element types within a particular dimension, but types from different dimensions may not be mixed. For example, a Mesh cannot be constructed of both quadrilaterals and tetrahedra.

ESMF currently only supports distributions where every node on a PET must be a part of an element on that PET. In other words, there must not be nodes without a corresponding element on any PET.

33.2 Constants


33.2.1 ESMF_MESHELEMTYPE

DESCRIPTION:
An ESMF Mesh can be constructed from a combination of different elements. The type of elements that can be used in a Mesh depends on the Mesh's parameteric dimension, which is set during Mesh creation. The following are the valid Mesh element types for each valid Mesh parametric dimension (2D or 3D) .


                     3                          4 ---------- 3
                    / \                         |            |  
                   /   \                        |            |
                  /     \                       |            |
                 /       \                      |            |
                /         \                     |            |
               1 --------- 2                    1 ---------- 2

           ESMF_MESHELEMTYPE_TRI            ESMF_MESHELEMTYPE_QUAD

     2D element types (numbers are the order for elementConn during Mesh create)

For a Mesh with parametric dimension of 2 ESMF supports two native element types (illustrated above), but also supports polygons with more sides. Internally these polygons are represented as a set of triangles, but to the user should behave like other elements. To specify the non-native polygons in the elementType argument use the number of corners of the polygon (e.g. for a pentagon use 5). The connectivity for a polygon should be specified in counterclockwise order. The following table summarizes this information:

Element Type Number of Nodes Description
ESMF_MESHELEMTYPE_TRI 3 A triangle
ESMF_MESHELEMTYPE_QUAD 4 A quadrilateral (e.g. a rectangle)
N N An N-gon (e.g. if N=5 a pentagon)



                                            
                 3                               8---------------7
                /|\                             /|              /|
               / | \                           / |             / |
              /  |  \                         /  |            /  |
             /   |   \                       /   |           /   |
            /    |    \                     5---------------6    |
           4-----|-----2                    |    |          |    |
            \    |    /                     |    4----------|----3
             \   |   /                      |   /           |   /
              \  |  /                       |  /            |  /
               \ | /                        | /             | /
                \|/                         |/              |/
                 1                          1---------------2

       ESMF_MESHELEMTYPE_TETRA             ESMF_MESHELEMTYPE_HEX  

  3D element types (numbers are the order for elementConn during Mesh create)

For a Mesh with parametric dimension of 3 the valid element types (illustrated above) are:

Element Type Number of Nodes Description
ESMF_MESHELEMTYPE_TETRA 4 A tetrahedron (NOT VALID IN BILINEAR OR PATCH REGRID)
ESMF_MESHELEMTYPE_HEX 8 A hexahedron (e.g. a cube)

33.3 Use and Examples

This section describes the use of the ESMF Mesh class. It starts with an explanation and examples of creating a Mesh and then goes through other Mesh methods. This set of sections covers the use of the Mesh class interfaces. For further detail which applies to creating a Field on a Mesh, please see Section 26.3.19.


33.3.1 Mesh creation

To create a Mesh we need to set some properties of the Mesh as a whole, some properties of each node in the mesh and then some properties of each element which connects the nodes (for a definition of node and element please see Section 33.1.1).

For the Mesh as a whole we set its parametric dimension (parametricDim) and spatial dimension (spatialDim). A Mesh's parametric dimension can be thought of as the dimension of the elements which make up the Mesh. A Mesh's spatial dimension, on the other hand, is the is the number of coordinate dimensions needed to describe the location of the nodes making up the Mesh. (For a fuller definition of these terms please see Section 33.1.1.)

The structure of the per node and element information used to create a Mesh is influenced by the Mesh distribution strategy. The Mesh class is distributed by elements. This means that a node must be present on any PET that contains an element associated with that node, but not on any other PET (a node can't be on a PET without an element "home"). Since a node may be used by two or more elements located on different PETs, a node may be duplicated on multiple PETs. When a node is duplicated in this manner, one and only one of the PETs that contain the node must "own" the node. The user sets this ownership when they define the nodes during Mesh creation. When a Field is created on a Mesh (i.e. on the Mesh nodes), on each PET the Field is only created on the nodes which are owned by that PET. This means that the size of the Field memory on the PET can be smaller than the number of nodes used to create the Mesh on that PET. Please see Section 26.3.19 in Field for further explanation and examples of this issue and others in working with Fields on Meshes.

For each node in the Mesh we set three properties: the global id of the node (nodeIds), node coordinates (nodeCoords), and which PET owns the node (nodeOwners). The node id is a unique (across all PETs) integer attached to the particular node. It is used to indicate which nodes are the same when connecting together pieces of the Mesh on different processors. The node coordinates indicate the location of a node in space and are used in the ESMF_FieldRegrid() functionality when interpolating. The node owner indicates which PET is in charge of the node. This is used when creating a Field on the Mesh to indicate which PET should contain a Field location for the data.

For each element in the Mesh we set three properties: the global id of the element (elementIds), the topology type of the element (elementTypes), and which nodes are connected together to form the element (elementConn). The element id is a unique (across all PETs) integer attached to the particular element. The element type describes the topology of the element (e.g. a triangle vs. a quadrilateral). The range of choices for the topology of the elements in a Mesh are restricted by the Mesh's parametric dimension (e.g. a Mesh can't contain a 2D element like a triangle, when its parametric dimension is 3D), but it can contain any combination of elements appropriate to its dimension. In particular, in 2D ESMF supports two native element types triangle and quadrilateral, but also provides support for polygons with any number of sides. These polygons are represented internally as sets of triangles, but to the user should behave like other elements. To specify a polygon with more than four sides, the element type should be set to the number of corners of the polygon (e.g. element type=6 for a hexagon). The element connectivity indicates which nodes are to be connected together to form the element. The number of nodes connected together for each element is implied by the elements topology type (elementTypes). It is IMPORTANT to note, that the entries in this list are NOT the global ids of the nodes, but are indices into the PET local lists of node info used in the Mesh Create. In other words, the element connectivity isn't specified in terms of the global list of nodes, but instead is specified in terms of the locally described node info. One other important point about connectivities is that the order of the nodes in the connectivity list of an element is important. Please see Section 33.2.1 for diagrams illustrating the correct order of nodes in an element. In general, when specifying an element with parametric dimension 2, the nodes should be given in counterclockwise order around the element.

Mesh creation may either be performed as a one step process using the full ESMF_MeshCreate() call, or may be done in three steps. The three step process starts with a more minimal ESMF_MeshCreate() call. It is then followed by the ESMF_MeshAddNodes() to specify nodes, and then the ESMF_MeshAddElements() call to specify elements. This three step sequence is useful to conserve memory because the node arrays being used for the ESMF_MeshAddNodes() call can be deallocated before creating the arrays to be used in the ESMF_MeshAddElements() call.


33.3.2 Create a small single PET Mesh in one step

  
   
    2.0   7 ------- 8 ------- 9
          |         |         |
          |    4    |    5    |
          |         |         |
    1.0   4 ------- 5 ------- 6
          |         |  \   3  |
          |    1    |    \    |
          |         |  2   \  |
    0.0   1 ------- 2 ------- 3
  
         0.0       1.0        2.0 
   
          Node Id labels at corners
         Element Id labels in centers
         (Everything owned by PET 0)

This example is intended to illustrate the creation of a small Mesh on one PET. The reason for starting with a single PET case is so that the user can start to familiarize themselves with the concepts of Mesh creation without the added complication of multiple processors. Later examples illustrate the multiple processor case. This example creates the small 2D Mesh which can be seen in the figure above. Note that this Mesh consists of 9 nodes and 5 elements, where the elements are a mixture of quadrilaterals and triangles. The coordinates of the nodes in the Mesh range from 0.0 to 2.0 in both dimensions. The node ids are in the corners of the elements whereas the element ids are in the centers. The following section of code illustrates the creation of this Mesh.

  ! Set number of nodes
  numNodes=9

  ! Allocate and fill the node id array.
  allocate(nodeIds(numNodes))
  nodeIds=(/1,2,3,4,5,6,7,8,9/) 

  ! Allocate and fill node coordinate array.
  ! Since this is a 2D Mesh the size is 2x the
  ! number of nodes.
  allocate(nodeCoords(2*numNodes))
  nodeCoords=(/0.0,0.0, & ! node id 1
               1.0,0.0, & ! node id 2
               2.0,0.0, & ! node id 3
               0.0,1.0, & ! node id 4
               1.0,1.0, & ! node id 5
               2.0,1.0, & ! node id 6
               0.0,2.0, & ! node id 7
               1.0,2.0, & ! node id 8
               2.0,2.0 /) ! node id 9

  ! Allocate and fill the node owner array.
  ! Since this Mesh is all on PET 0, it's just set to all 0.
  allocate(nodeOwners(numNodes))
  nodeOwners=0 ! everything on PET 0


  ! Set the number of each type of element, plus the total number.
  numQuadElems=3
  numTriElems=2
  numTotElems=numQuadElems+numTriElems


  ! Allocate and fill the element id array.
  allocate(elemIds(numTotElems))
  elemIds=(/1,2,3,4,5/) 


  ! Allocate and fill the element topology type array.
  allocate(elemTypes(numTotElems))
  elemTypes=(/ESMF_MESHELEMTYPE_QUAD, & ! elem id 1
              ESMF_MESHELEMTYPE_TRI,  & ! elem id 2
              ESMF_MESHELEMTYPE_TRI,  & ! elem id 3
              ESMF_MESHELEMTYPE_QUAD, & ! elem id 4
              ESMF_MESHELEMTYPE_QUAD/)  ! elem id 5


  ! Allocate and fill the element connection type array.
  ! Note that entries in this array refer to the 
  ! positions in the nodeIds, etc. arrays and that
  ! the order and number of entries for each element
  ! reflects that given in the Mesh options 
  ! section for the corresponding entry
  ! in the elemTypes array. The number of 
  ! entries in this elemConn array is the
  ! number of nodes in a quad. (4) times the 
  ! number of quad. elements plus the number
  ! of nodes in a triangle (3) times the number
  ! of triangle elements. 
  allocate(elemConn(4*numQuadElems+3*numTriElems))
  elemConn=(/1,2,5,4, &  ! elem id 1
             2,3,5,   &  ! elem id 2
             3,6,5,   &  ! elem id 3
             4,5,8,7, &  ! elem id 4
             5,6,9,8/)   ! elem id 5


  ! Create Mesh structure in 1 step
  mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
         coordSys=ESMF_COORDSYS_CART, &
         nodeIds=nodeIds, nodeCoords=nodeCoords, &
         nodeOwners=nodeOwners, elementIds=elemIds,&
         elementTypes=elemTypes, elementConn=elemConn, &
         rc=localrc)


  ! After the creation we are through with the arrays, so they may be
  ! deallocated.
  deallocate(nodeIds)
  deallocate(nodeCoords)
  deallocate(nodeOwners)
  deallocate(elemIds)
  deallocate(elemTypes)
  deallocate(elemConn)


  ! At this point the mesh is ready to use. For example, as is 
  ! illustrated here, to have a field created on it. Note that 
  ! the Field only contains data for nodes owned by the current PET.
  ! Please see Section "Create a Field from a Mesh" under Field
  ! for more information on creating a Field on a Mesh. 
  field = ESMF_FieldCreate(mesh, ESMF_TYPEKIND_R8,  rc=localrc)


33.3.3 Create a small single PET Mesh in three steps

This example is intended to illustrate the creation of a small Mesh in three steps on one PET. The Mesh being created is exactly the same one as in the last example (Section 33.3.2), but the three step process allows the creation to occur in a more memory efficient manner.

  ! Create the mesh structure setting the dimensions
  ! and coordinate system
  mesh = ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
                         coordSys=ESMF_COORDSYS_CART, &
                         rc=localrc)

  ! Set number of nodes
  numNodes=9

  ! Allocate and fill the node id array.
  allocate(nodeIds(numNodes))
  nodeIds=(/1,2,3,4,5,6,7,8,9/) 

  ! Allocate and fill node coordinate array.
  ! Since this is a 2D Mesh the size is 2x the
  ! number of nodes.
  allocate(nodeCoords(2*numNodes))
  nodeCoords=(/0.0,0.0, & ! node id 1
               1.0,0.0, & ! node id 2
               2.0,0.0, & ! node id 3
               0.0,1.0, & ! node id 4
               1.0,1.0, & ! node id 5
               2.0,1.0, & ! node id 6
               0.0,2.0, & ! node id 7
               1.0,2.0, & ! node id 8
               2.0,2.0 /) ! node id 9

  ! Allocate and fill the node owner array.
  ! Since this Mesh is all on PET 0, it's just set to all 0.
  allocate(nodeOwners(numNodes))
  nodeOwners=0 ! everything on PET 0

  ! Add the nodes to the Mesh
  call ESMF_MeshAddNodes(mesh, nodeIds=nodeIds, &
         nodeCoords=nodeCoords, nodeOwners=nodeOwners, rc=localrc)

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  ! HERE IS THE POINT OF THE THREE STEP METHOD
  ! WE CAN DELETE THESE NODE ARRAYS BEFORE 
  ! ALLOCATING THE ELEMENT ARRAYS, THEREBY
  ! REDUCING THE AMOUNT OF MEMORY NEEDED 
  ! AT ONE TIME. 
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  deallocate(nodeIds)
  deallocate(nodeCoords)
  deallocate(nodeOwners)


  ! Set the number of each type of element, plus the total number.
  numQuadElems=3
  numTriElems=2
  numTotElems=numQuadElems+numTriElems

  ! Allocate and fill the element id array.
  allocate(elemIds(numTotElems))
  elemIds=(/1,2,3,4,5/) 

  ! Allocate and fill the element topology type array.
  allocate(elemTypes(numTotElems))
  elemTypes=(/ESMF_MESHELEMTYPE_QUAD, & ! elem id 1
              ESMF_MESHELEMTYPE_TRI,  & ! elem id 2
              ESMF_MESHELEMTYPE_TRI,  & ! elem id 3
              ESMF_MESHELEMTYPE_QUAD, & ! elem id 4
              ESMF_MESHELEMTYPE_QUAD/)  ! elem id 5


  ! Allocate and fill the element connection type array.
  ! Note that entries in this array refer to the 
  ! positions in the nodeIds, etc. arrays and that
  ! the order and number of entries for each element
  ! reflects that given in the Mesh options 
  ! section for the corresponding entry
  ! in the elemTypes array. The number of 
  ! entries in this elemConn array is the
  ! number of nodes in a quad. (4) times the 
  ! number of quad. elements plus the number
  ! of nodes in a triangle (3) times the number
  ! of triangle elements. 
  allocate(elemConn(4*numQuadElems+3*numTriElems))
  elemConn=(/1,2,5,4, &  ! elem id 1
             2,3,5,   &  ! elem id 2
             3,6,5,   &  ! elem id 3
             4,5,8,7, &  ! elem id 4
             5,6,9,8/)   ! elem id 5


  ! Finish the creation of the Mesh by adding the elements
  call ESMF_MeshAddElements(mesh, elementIds=elemIds,&
         elementTypes=elemTypes, elementConn=elemConn, &
         rc=localrc)

  ! After the creation we are through with the arrays, so they may be
  ! deallocated.
  deallocate(elemIds)
  deallocate(elemTypes)
  deallocate(elemConn)


  ! At this point the mesh is ready to use. For example, as is 
  ! illustrated here, to have a field created on it. Note that 
  ! the Field only contains data for nodes owned by the current PET.
  ! Please see Section "Create a Field from a Mesh" under Field
  ! for more information on creating a Field on a Mesh. 
  field = ESMF_FieldCreate(mesh, ESMF_TYPEKIND_R8,  rc=localrc)


33.3.4 Create a small Mesh on 4 PETs in one step

  
    2.0   7 ------- 8        [8] ------ 9          
          |         |         |         |
          |    4    |         |    5    |
          |         |         |         |
    1.0  [4] ----- [5]       [5] ----- [6]
          
         0.0       1.0       1.0       2.0
  
             PET 2               PET 3
  
  
    1.0   4 ------- 5        [5] ------ 6
          |         |         |  \   3  |
          |    1    |         |    \    |
          |         |         | 2    \  |
    0.0   1 ------- 2        [2] ------ 3
  
         0.0       1.0       1.0      2.0 
   
             PET 0               PET 1
  
                 Node Id labels at corners
                Element Id labels in centers

This example is intended to illustrate the creation of a small Mesh on multiple PETs. This example creates the same small 2D Mesh as the previous two examples (See Section 33.3.2 for a diagram), however, in this case the Mesh is broken up across 4 PETs. The figure above illustrates the distribution of the Mesh across the PETs. As in the previous diagram, the node ids are in the corners of the elements and the element ids are in the centers. In this figure '[' and ']' around a character indicate a node which is owned by another PET. The nodeOwner parameter indicates which PET owns the node. Note that the three step creation illustrated in Section 33.3.3 could also be used in a parallel Mesh creation such as this by simply interleaving the three calls in the appropriate places between the node and element array definitions.

 ! Break up what's being set by PET
 if (localPET .eq. 0) then !!! This part only for PET 0
    ! Set number of nodes
     numNodes=4

    ! Allocate and fill the node id array.
    allocate(nodeIds(numNodes))
    nodeIds=(/1,2,4,5/) 

    ! Allocate and fill node coordinate array.
    ! Since this is a 2D Mesh the size is 2x the
    ! number of nodes.
    allocate(nodeCoords(2*numNodes))
    nodeCoords=(/0.0,0.0, & ! node id 1
                 1.0,0.0, & ! node id 2
                 0.0,1.0, & ! node id 4
                 1.0,1.0 /) ! node id 5

    ! Allocate and fill the node owner array.
    allocate(nodeOwners(numNodes))
    nodeOwners=(/0, & ! node id 1
                 0, & ! node id 2
                 0, & ! node id 4
                 0/)  ! node id 5

    ! Set the number of each type of element, plus the total number.
    numQuadElems=1
    numTriElems=0
    numTotElems=numQuadElems+numTriElems

    ! Allocate and fill the element id array.
    allocate(elemIds(numTotElems))
    elemIds=(/1/) 

    ! Allocate and fill the element topology type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_QUAD/) ! elem id 1

    ! Allocate and fill the element connection type array.
    ! Note that entry are local indices
    allocate(elemConn(4*numQuadElems+3*numTriElems))
    elemConn=(/1,2,4,3/) ! elem id 1

  else if (localPET .eq. 1) then !!! This part only for PET 1
    ! Set number of nodes
     numNodes=4

    ! Allocate and fill the node id array.
    allocate(nodeIds(numNodes))
    nodeIds=(/2,3,5,6/) 

    ! Allocate and fill node coordinate array.
    ! Since this is a 2D Mesh the size is 2x the
    ! number of nodes.
    allocate(nodeCoords(2*numNodes))
    nodeCoords=(/1.0,0.0, & ! node id 2
                 2.0,0.0, & ! node id 3
                 1.0,1.0, & ! node id 5
                 2.0,1.0 /) ! node id 6

    ! Allocate and fill the node owner array.
    allocate(nodeOwners(numNodes))
    nodeOwners=(/0, & ! node id 2
                 1, & ! node id 3
                 0, & ! node id 5
                 1/)  ! node id 6

    ! Set the number of each type of element, plus the total number.
    numQuadElems=0
    numTriElems=2
    numTotElems=numQuadElems+numTriElems

    ! Allocate and fill the element id array.
    allocate(elemIds(numTotElems))
    elemIds=(/2,3/) 

    ! Allocate and fill the element topology type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_TRI, & ! elem id 2
                ESMF_MESHELEMTYPE_TRI/)  ! elem id 3

    ! Allocate and fill the element connection type array.
    allocate(elemConn(4*numQuadElems+3*numTriElems))
    elemConn=(/1,2,3, & ! elem id 2
               2,4,3/)  ! elem id 3

  else if (localPET .eq. 2) then !!! This part only for PET 2
    ! Set number of nodes
     numNodes=4

    ! Allocate and fill the node id array.
    allocate(nodeIds(numNodes))
    nodeIds=(/4,5,7,8/) 

    ! Allocate and fill node coordinate array.
    ! Since this is a 2D Mesh the size is 2x the
    ! number of nodes.
    allocate(nodeCoords(2*numNodes))
    nodeCoords=(/0.0,1.0, & ! node id 4
                 1.0,1.0, & ! node id 5
                 0.0,2.0, & ! node id 7
                 1.0,2.0 /) ! node id 8

    ! Allocate and fill the node owner array.
    ! Since this Mesh is all on PET 0, it's just set to all 0.
    allocate(nodeOwners(numNodes))
    nodeOwners=(/0, & ! node id 4
                 0, & ! node id 5
                 2, & ! node id 7
                 2/)  ! node id 8

    ! Set the number of each type of element, plus the total number.
    numQuadElems=1
    numTriElems=0
    numTotElems=numQuadElems+numTriElems

    ! Allocate and fill the element id array.
    allocate(elemIds(numTotElems))
    elemIds=(/4/) 

    ! Allocate and fill the element topology type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_QUAD/) ! elem id 4

    ! Allocate and fill the element connection type array.
    allocate(elemConn(4*numQuadElems+3*numTriElems))
    elemConn=(/1,2,4,3/) ! elem id 4

  else if (localPET .eq. 3) then !!! This part only for PET 3
    ! Set number of nodes
     numNodes=4

    ! Allocate and fill the node id array.
    allocate(nodeIds(numNodes))
    nodeIds=(/5,6,8,9/) 

    ! Allocate and fill node coordinate array.
    ! Since this is a 2D Mesh the size is 2x the
    ! number of nodes.
    allocate(nodeCoords(2*numNodes))
    nodeCoords=(/1.0,1.0, &  ! node id 5
                 2.0,1.0, &  ! node id 6
                 1.0,2.0, &  ! node id 8
                 2.0,2.0 /)  ! node id 9

    ! Allocate and fill the node owner array.
    allocate(nodeOwners(numNodes))
    nodeOwners=(/0, & ! node id 5
                 1, & ! node id 6
                 2, & ! node id 8
                 3/)  ! node id 9

    ! Set the number of each type of element, plus the total number.
    numQuadElems=1
    numTriElems=0
    numTotElems=numQuadElems+numTriElems

    ! Allocate and fill the element id array.
    allocate(elemIds(numTotElems))
    elemIds=(/5/)  

    ! Allocate and fill the element topology type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_QUAD/) ! elem id 5

    ! Allocate and fill the element connection type array.
    allocate(elemConn(4*numQuadElems+3*numTriElems))
    elemConn=(/1,2,4,3/) ! elem id 5
  endif

  
  ! Create Mesh structure in 1 step
  mesh=ESMF_MeshCreate(parametricDim=2, spatialDim=2, &
         coordSys=ESMF_COORDSYS_CART, &
         nodeIds=nodeIds, nodeCoords=nodeCoords, &
         nodeOwners=nodeOwners, elementIds=elemIds,&
         elementTypes=elemTypes, elementConn=elemConn, &
         rc=localrc)


  ! After the creation we are through with the arrays, so they may be
  ! deallocated.
  deallocate(nodeIds)
  deallocate(nodeCoords)
  deallocate(nodeOwners)
  deallocate(elemIds)
  deallocate(elemTypes)
  deallocate(elemConn)


  ! At this point the mesh is ready to use. For example, as is 
  ! illustrated here, to have a field created on it. Note that 
  ! the Field only contains data for nodes owned by the current PET.
  ! Please see Section "Create a Field from a Mesh" under Field
  ! for more information on creating a Field on a Mesh. 
  field = ESMF_FieldCreate(mesh, ESMF_TYPEKIND_R8,  rc=localrc)

33.3.5 Create a copy of a Mesh with a new distribution

  
    2.0   7 -------[8]               8 ------- 9          
          |         |                |         |
          |    4    |                |    5    |
          |         |                |         |
    1.0   4 ------ [5]               5 ------- 6
          
         0.0       1.0              1.0       2.0
  
             PET 1                      PET 0
  
  
    1.0  [4] ----- [5]              [5] ----- [6]
          |         |  \                \      |
          |    1    | 2  \                \  3 |
          |         |      \                \  |
    0.0   1 ------- 2 -----[3]                 3
  
         0.0       1.0               1.0      2.0 
   
             PET 2                      PET 3
  
                 Node Id labels at corners
                Element Id labels in centers

This example demonstrates the creation of a new Mesh which is a copy of an existing Mesh with a new distribution of the original Mesh's nodes and elements. To create the new Mesh in this manner the user needs two DistGrids. One to describe the new distribution of the nodes. The other to describe the new distribution of the elements. In this example we create new DistGrids from a list of indices. The DistGrids are then used in the redistribution Mesh create interface which is overloaded to ESMF_MeshCreate(). In this example we redistribute the Mesh created in the previous example (Section 33.3.4) to the distribution pictured above. Note that for simplicity's sake, the position of the Mesh in the diagram is basically the same, but the PET positions and node owners have been changed.

  ! Setup the new location of nodes and elements depending on the processor
  if (localPet .eq. 0) then !!! This part only for PET 0
     allocate(elemIds(1))
     elemIds=(/5/)
     
     allocate(nodeIds(4))
     nodeIds=(/5,6,8,9/)
     
  else if (localPet .eq. 1) then !!! This part only for PET 1
     allocate(elemIds(1))
     elemIds=(/4/)
     
     allocate(nodeIds(2))
     nodeIds=(/7,4/)        
     
  else if (localPet .eq. 2) then !!! This part only for PET 2
     allocate(elemIds(2))
     elemIds=(/1,2/)
     
     allocate(nodeIds(2))
     nodeIds=(/1,2/)
     
  else if (localPet .eq. 3) then !!! This part only for PET 3
     allocate(elemIds(1))
     elemIds=(/3/)
     
     allocate(nodeIds(1))
     nodeIds=(/3/)
     
  endif


  ! Create new node DistGrid
  nodedistgrid=ESMF_DistGridCreate(nodeIds, rc=localrc)
  if (localrc .ne. ESMF_SUCCESS) rc=ESMF_FAILURE

 
  ! Create new element DistGrid
  elemdistgrid=ESMF_DistGridCreate(elemIds, rc=localrc)
  if (localrc .ne. ESMF_SUCCESS) rc=ESMF_FAILURE


  ! Can now deallocate distribution lists
  deallocate(elemIds)
  deallocate(nodeIds)


  ! Create new redisted Mesh based on DistGrids
  mesh2=ESMF_MeshCreate(mesh,                         &
                        nodalDistgrid=nodedistgrid,   & 
                        elementDistgrid=elemdistgrid, &
                        rc=localrc)
  if (localrc .ne. ESMF_SUCCESS) rc=ESMF_FAILURE


  ! At this point the mesh is ready to use. For example, as is 
  ! illustrated here, to have a field created on it. Note that 
  ! the Field only contains data for nodes owned by the current PET.
  ! Please see Section "Create a Field from a Mesh" under Field
  ! for more information on creating a Field on a Mesh. 
  field = ESMF_FieldCreate(mesh2, ESMF_TYPEKIND_R8,  rc=localrc)


33.3.6 Create a small Mesh of all one element type on 4 PETs using easy element method

  
    2.0   * ------- *         * ------- *          
          |         |         |         |
          |    3    |         |    4    |
          |         |         |         |
    1.0   * ------- *         * ------- *
          
         0.0       1.0       1.0       2.0
  
             PET 2               PET 3
  
  
    1.0   * ------- *         * ------- *
          |         |         |         |
          |    1    |         |    2    |
          |         |         |         |
    0.0   * ------- *         * ------- *
  
         0.0       1.0       1.0      2.0 
   
             PET 0               PET 1
  
             Element Id labels in centers

This example is intended to illustrate the creation of a small Mesh on multiple PETs using the easy element creation interface. Here the Mesh consists of only one type of element, so we can use a slightly more convenient interface. In this interface the user only needs to specify the element type once and the elementCornerCoords argument has three dimensions. This means that the corners for all elements are not collapsed into a 1D list as happens with the next example.

The figure above shows the Mesh to be created and it's distribution across the PETs. As in the previous diagrams, the element ids are in the centers. Note that in the example code below the user doesn't specify the element ids. In this case, they are assigned sequentially through the local elements on each PET starting with 1 for the first element on PET 0. (It isn't shown in the example below, but there is an optional argument that enables the user to set the element ids if they wish.) Unlike some of the previous examples of Mesh creation, here the user doesn't specify node ids or ownership, so that information is shown by a "*" in the diagram.

 ! Break up what's being set by PET
 if (localPET .eq. 0) then !!! This part only for PET 0

    ! Set the number of elements on this PET
    numTotElems=1

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords3(2,4,numTotElems))
    elemCornerCoords3(:,1,1)=(/0.0,0.0/) ! elem id 1 corner 1 
    elemCornerCoords3(:,2,1)=(/1.0,0.0/) ! elem id 1 corner 2
    elemCornerCoords3(:,3,1)=(/1.0,1.0/) ! elem id 1 corner 3
    elemCornerCoords3(:,4,1)=(/0.0,1.0/) ! elem id 1 corner 4

  else if (localPET .eq. 1) then !!! This part only for PET 1

    ! Set the number of elements on this PET
    numTotElems=1

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords3(2,4,numTotElems))
    elemCornerCoords3(:,1,1)=(/1.0,0.0/) ! elem id 2 corner 1 
    elemCornerCoords3(:,2,1)=(/2.0,0.0/) ! elem id 2 corner 2
    elemCornerCoords3(:,3,1)=(/2.0,1.0/) ! elem id 2 corner 3
    elemCornerCoords3(:,4,1)=(/1.0,1.0/) ! elem id 2 corner 4 


  else if (localPET .eq. 2) then !!! This part only for PET 2

    ! Set the number of elements on this PET
    numTotElems=1

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords3(2,4,numTotElems))
    elemCornerCoords3(:,1,1)=(/0.0,1.0/) ! elem id 3 corner 1 
    elemCornerCoords3(:,2,1)=(/1.0,1.0/) ! elem id 3 corner 2
    elemCornerCoords3(:,3,1)=(/1.0,2.0/) ! elem id 3 corner 3
    elemCornerCoords3(:,4,1)=(/0.0,2.0/) ! elem id 3 corner 4


  else if (localPET .eq. 3) then !!! This part only for PET 3

    ! Set the number of elements on this PET
    numTotElems=1

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords3(2,4,numTotElems))
    elemCornerCoords3(:,1,1)=(/1.0,1.0/) ! elem id 4 corner 1 
    elemCornerCoords3(:,2,1)=(/2.0,1.0/) ! elem id 4 corner 2
    elemCornerCoords3(:,3,1)=(/2.0,2.0/) ! elem id 4 corner 3
    elemCornerCoords3(:,4,1)=(/1.0,2.0/) ! elem id 4 corner 4

  endif
  
  ! Create Mesh structure in 1 step
  mesh=ESMF_MeshCreate(parametricDim=2, &
         coordSys=ESMF_COORDSYS_CART,   &
         elementType=ESMF_MESHELEMTYPE_QUAD, &
         elementCornerCoords=elemCornerCoords3, &
         rc=localrc)


  ! After the creation we are through with the arrays, so they may be
  ! deallocated.
  deallocate(elemCornerCoords3)

  ! At this point the mesh is ready to use. For example, as is 
  ! illustrated here, to have a field created on it. Note that 
  ! the Field only contains data for elements owned by the current PET.
  ! Please see Section "Create a Field from a Mesh" under Field
  ! for more information on creating a Field on a Mesh. 
  field = ESMF_FieldCreate(mesh, ESMF_TYPEKIND_R8, &
       meshloc=ESMF_MESHLOC_ELEMENT, rc=localrc)


33.3.7 Create a small Mesh of multiple element types on 4 PETs using easy element method

  
    2.0   * ------- *         * ------- *          
          |         |         |         |
          |    4    |         |    5    |
          |         |         |         |
    1.0   * ------- *         * ------- *
          
         0.0       1.0       1.0       2.0
  
             PET 2               PET 3
  
  
    1.0   * ------- *         * ------- *
          |         |         |  \   3  |
          |    1    |         |    \    |
          |         |         | 2    \  |
    0.0   * ------- *         * ------- *
  
         0.0       1.0       1.0      2.0 
   
             PET 0               PET 1
  
             Element Id labels in centers

This example is intended to illustrate the creation of a small Mesh on multiple PETs using the easy element creation interface. In this example, the Mesh being created contains elements of multiple types. To support the specification of a set of elements containing different types and thus different numbers of corners, the elementCornerCoords argument has the corner and element dimensions collapsed together into one dimension.

The figure above shows the Mesh to be created and it's distribution across the PETs. As in the previous diagrams, the element ids are in the centers. Note that in the example code below the user doesn't specify the element ids. In this case, they are assigned sequentially through the local elements on each PET starting with 1 for the first element on PET 0. (It isn't shown in the example below, but there is an optional argument that enables the user to set the element ids if they wish.) Unlike some of the previous examples of Mesh creation, here the user doesn't specify node ids or ownership, so that information is shown by a "*" in the diagram.

 ! Break up what's being set by PET
 if (localPET .eq. 0) then !!! This part only for PET 0

    ! Set the number of each type of element, plus the total number.
    numQuadElems=1
    numTriElems=0
    numTotElems=numQuadElems+numTriElems
    numElemCorners=4*numQuadElems+3*numTriElems

    ! Allocate and fill the element type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_QUAD/) ! elem id 1

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords2(2,numElemCorners))
    elemCornerCoords2(:,1)=(/0.0,0.0/) ! elem id 1 corner 1 
    elemCornerCoords2(:,2)=(/1.0,0.0/) ! elem id 1 corner 2
    elemCornerCoords2(:,3)=(/1.0,1.0/) ! elem id 1 corner 3
    elemCornerCoords2(:,4)=(/0.0,1.0/) ! elem id 1 corner 4

  else if (localPET .eq. 1) then !!! This part only for PET 1

    ! Set the number of each type of element, plus the total number.
    numQuadElems=0
    numTriElems=2
    numTotElems=numQuadElems+numTriElems
    numElemCorners=4*numQuadElems+3*numTriElems

    ! Allocate and fill the element type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_TRI, & ! elem id 2
                ESMF_MESHELEMTYPE_TRI/)  ! elem id 3

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords2(2,numElemCorners))
    elemCornerCoords2(:,1)=(/1.0,0.0/) ! elem id 2 corner 1 
    elemCornerCoords2(:,2)=(/2.0,0.0/) ! elem id 2 corner 2
    elemCornerCoords2(:,3)=(/1.0,1.0/) ! elem id 2 corner 3
    elemCornerCoords2(:,4)=(/2.0,0.0/) ! elem id 3 corner 1 
    elemCornerCoords2(:,5)=(/2.0,1.0/) ! elem id 3 corner 2
    elemCornerCoords2(:,6)=(/1.0,1.0/) ! elem id 3 corner 3

  else if (localPET .eq. 2) then !!! This part only for PET 2

    ! Set the number of each type of element, plus the total number.
    numQuadElems=1
    numTriElems=0
    numTotElems=numQuadElems+numTriElems
    numElemCorners=4*numQuadElems+3*numTriElems


    ! Allocate and fill the element type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_QUAD/) ! elem id 4

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords2(2,numElemCorners))
    elemCornerCoords2(:,1)=(/0.0,1.0/) ! elem id 4 corner 1 
    elemCornerCoords2(:,2)=(/1.0,1.0/) ! elem id 4 corner 2
    elemCornerCoords2(:,3)=(/1.0,2.0/) ! elem id 4 corner 3
    elemCornerCoords2(:,4)=(/0.0,2.0/) ! elem id 4 corner 4


  else if (localPET .eq. 3) then !!! This part only for PET 3

    ! Set the number of each type of element, plus the total number.
    numQuadElems=1
    numTriElems=0
    numTotElems=numQuadElems+numTriElems
    numElemCorners=4*numQuadElems+3*numTriElems

    ! Allocate and fill the element type array.
    allocate(elemTypes(numTotElems))
    elemTypes=(/ESMF_MESHELEMTYPE_QUAD/) ! elem id 5

    ! Allocate and fill element corner coordinate array.
    allocate(elemCornerCoords2(2,numElemCorners))
    elemCornerCoords2(:,1)=(/1.0,1.0/) ! elem id 5 corner 1 
    elemCornerCoords2(:,2)=(/2.0,1.0/) ! elem id 5 corner 2
    elemCornerCoords2(:,3)=(/2.0,2.0/) ! elem id 5 corner 3
    elemCornerCoords2(:,4)=(/1.0,2.0/) ! elem id 5 corner 4

  endif
  
  ! Create Mesh structure in 1 step
  mesh=ESMF_MeshCreate(parametricDim=2, &
         coordSys=ESMF_COORDSYS_CART,   &
         elementTypes=elemTypes, &
         elementCornerCoords=elemCornerCoords2, &
         rc=localrc)


  ! After the creation we are through with the arrays, so they may be
  ! deallocated.
  deallocate(elemTypes)
  deallocate(elemCornerCoords2)

  ! At this point the mesh is ready to use. For example, as is 
  ! illustrated here, to have a field created on it. Note that 
  ! the Field only contains data for elements owned by the current PET.
  ! Please see Section "Create a Field from a Mesh" under Field
  ! for more information on creating a Field on a Mesh. 
  field = ESMF_FieldCreate(mesh, ESMF_TYPEKIND_R8, &
       meshloc=ESMF_MESHLOC_ELEMENT, rc=localrc)


33.3.8 Create a Mesh from an unstructured grid file

ESMF supports the creation of a Mesh from three grid file formats: the SCRIP format 12.8.1, the ESMF format 12.8.2 or the proposed CF unstructured grid UGRID format 12.8.4. All three of these grid file formats are NetCDF files.

When creating a Mesh from a SCRIP format file, there are a number of options to control the output Mesh. The data is located at the center of the grid cell in a SCRIP grid; whereas the data is located at the corner of a cell in an ESMF Mesh object. Therefore, we create a Mesh object by default by constructing a "dual" mesh using the coordinates in the file. If the user wishes to not construct the dual mesh, the optional argument convertToDual may be used to control this behavior. When comvertToDual is set to .false. the Mesh constructed from the file will not be the dual. This is necessary when using the Mesh as part of a conservative regridding operation in the ESMF_FieldRegridStore() call, so the weights are properly generated for the cell centers in the file.

The following example code depicts how to create a Mesh using a SCRIP file. Note that you have to set the fileformat to ESMF_FILEFORMAT_SCRIP.

   mesh = ESMF_MeshCreate(filename="data/ne4np4-pentagons.nc", &
        fileformat=ESMF_FILEFORMAT_SCRIP, rc=localrc)

As mentioned above ESMF also supports creating Meshes from the ESMF format. The ESMF format works better with the methods used to create an ESMF Mesh object, so less conversion needs to be done to create a Mesh, and thus this format is more efficient than SCRIP to use within ESMF. The ESMF format is also more general than the SCRIP format because it supports higher dimension coordinates and more general topologies. Currently, ESMF_MeshCreate() does not support conversion to a dual mesh for this format. All regrid methods are supported on Meshes in this format.

Here is an example of creating a Mesh from an ESMF unstructured grid file. Note that you have to set the fileformat to ESMF_FILEFORMAT_ESMFMESH.

   mesh = ESMF_MeshCreate(filename="data/ne4np4-esmf.nc", &
            fileformat=ESMF_FILEFORMAT_ESMFMESH, &
            rc=localrc)


33.3.9 Create a Mesh representation of a cubed sphere grid

This example demostrates how to create a ESMF_Mesh object representing a cubed sphere grid with identical regular decomposition for every tile. In this example, the tile resolution is 45, so there will be a total 45x45x6=12150 elements in the mesh. nx and ny are the regular decomposition of each tile. The total number of DEs is nx x ny x 6. If the number of PETs are less than the total number of DEs, the DEs will be distributed to the PETs using the default cyclic distribution.

   ! Decompose each tile into 2 x 1 blocks
   nx=2
   ny=1

   ! Create Mesh
   mesh = ESMF_MeshCreateCubedSphere(tileSize=45, nx=nx,ny=ny, rc=localrc)

33.3.10 Remove Mesh memory

There are two different levels that the memory in a Mesh can be removed. The first of these is the standard destroy call, ESMF_MeshDestroy(). As with other classes, this call removes all memory associated with the object, and afterwards the object can not be used further (i.e. should not be used in any methods). The second, which is unique to Mesh, is the ESMF_MeshFreeMemory() call. This call removes the connection and coordinate information associated with the Mesh, but leaves the distgrid information. The coordinate and connection information held in the Mesh can consume a large amount of memory for a big Mesh, so using this call can very significantly reduce the amount of memory used. However, once this method has been used on a Mesh there are some restriction on what may be done with it. Once a Mesh has had its memory freed using this method, any Field built on the Mesh can no longer be used as part of an ESMF_FieldRegridStore() call. However, because the distgrid information is still part of the Mesh, Fields built on such a Mesh can still be part of an ESMF_FieldRegrid() call (where the routehandle was generated previous to the ESMF_MeshFreeMemory() operation). Fields may also still be created on these Meshes. The following short piece of code illustrates the use of this call.

   ! Here a Field built on a mesh may be used
   ! as part of a ESMF_FieldRegridStore() call

   ! This call removes connection and coordinate 
   ! information, significantly reducing the memory used by
   ! mesh, but limiting what can be done with it.
   call ESMF_MeshFreeMemory(mesh, rc=localrc)

   ! Here a new Field may be built on mesh, or
   ! a field built on a mesh may be used as part
   ! of an ESMF_FieldRegrid() call

   ! Destroy the mesh
   call ESMF_MeshDestroy(mesh, rc=localrc)

   ! Here mesh can't be used for anything


33.3.11 Mesh Masking

There are two types of masking available in Mesh: node masking and element masking. These both work in a similar manner, but vary slightly in the details of setting the mask information during mesh creation.

For node masking, the mask information is set using the nodeMask argument to either ESMF_MeshCreate() or ESMF_MeshAddNodes(). When a regrid store method is called (e.g. ESMF_FieldRegridStore()) the mask values arguments (srcMaskValues and dstMaskValues) can then be used to indicate which particular values set in the nodeMask array indicate that the node should be masked. For example, when calling ESMF_FieldRegridStore() if dstMaskValues has been set to 1, then any node in the destination Mesh whose corresponding nodeMask value is 1 will be masked out (a node with any other value than 1 will not be masked).

For element masking, the mask information is set using the elementMask argument to either ESMF_MeshCreate() or ESMF_MeshAddElements(). In a similar manner to node masking, when a regrid store method is called (e.g. ESMF_FieldRegridStore()) the mask values arguments (srcMaskValues and dstMaskValues) can then be used to indicate which particular values set in the elementMask array indicate that the element should be masked. For example, when calling ESMF_FieldRegridStore() if dstMaskValues has been set to 1, then any element in the destination Mesh whose corresponding elementMask value is 1 will be masked out (an element with any other value than 1 will not be masked).


33.3.12 Mesh Halo Communication

  
    2.0   7 ------- 8        [8] ------ 9          
          |         |         |         |
          |    4    |         |    5    |
          |         |         |         |
    1.0  [4] ----- [5]       [5] ----- [6]
          
         0.0       1.0       1.0       2.0
  
             PET 2               PET 3
  
  
    1.0   4 ------- 5        [5] ------ 6
          |         |         |  \   3  |
          |    1    |         |    \    |
          |         |         | 2    \  |
    0.0   1 ------- 2        [2] ------ 3
  
         0.0       1.0       1.0      2.0 
   
             PET 0               PET 1
  
              Node Id labels at corners
             Element Id labels in centers

This section illustrates the process of setting up halo communication for a Field built on the nodes of a Mesh. The Mesh used in this example is the one that was created in section 33.3.4. The diagram for that Mesh is repeated above for convenience's sake. The halo method used here is the one described in section 28.2.16, but made more specific to the case of a Mesh. This example shows how to set up haloing for nodes which are owned by another processor (e.g. the node with id 5 on PET 1 above). However, it could be expanded to halo other nodes simply by including them in the halo arrays below on the PET where their values are needed.

The first step in setting up the halo communication is to create arrays containing the ids of the haloed nodes on the PETs where they are needed.

The following illustrates that for the Mesh diagramed above.

  ! Create halo lists based on PET id.
  if (localPET .eq. 0) then !!! This part only for PET 0

    ! Allocate and fill the halo list.
    allocate(haloSeqIndexList(0))  ! There are no haloed points on PET 0

  else if (localPET .eq. 1) then !!! This part only for PET 1

    ! Allocate and fill the halo list.
    allocate(haloSeqIndexList(2)) 
    haloSeqIndexList=(/2,5/)

  else if (localPET .eq. 2) then !!! This part only for PET 2

    ! Allocate and fill the halo list.
    allocate(haloSeqIndexList(2)) 
    haloSeqIndexList=(/4,5/)

  else if (localPET .eq. 3) then !!! This part only for PET 3

    ! Allocate and fill the halo list.
    allocate(haloSeqIndexList(3)) 
    haloSeqIndexList=(/5,6,8/)

  endif

The next step is to create an ESMF Array with a halo region to hold the data being haloed.

  ! Get node DistGrid from the Mesh. 
  call ESMF_MeshGet(mesh, nodalDistgrid=nodeDistgrid, rc=localrc)

  ! Create an ESMF Array with a halo region from a node DistGrid.
  array=ESMF_ArrayCreate(nodeDistgrid, typekind=ESMF_TYPEKIND_R8, &
       haloSeqIndexList=haloSeqIndexList, rc=localrc)

Note that currently the halo data is stored at the end of the Array data on each PET in the order specified by the haloSeqIndexList argument (e.g. for PET 3 the halo information will be in the order 5,6,8 at the end of the piece of array on PET 3). This means that if the halo information needs to be in the order of nodes specified when you create the Mesh, then the nodes owned by another processor need to be at the end of the node information when the Mesh is created (e.g. when creating the piece of the Mesh on PET 3, then nodes 5,6,8 would need to be at the end of the node information lists).

At this point haloing could be done on the ESMF Array by using the ESMF_ArrayHaloStore() call followed by ESMF_ArrayHalo(). However, in this example we wrap the Array in an ESMF Field. This allows it to be used in Field specific calls (e.g. ESMF_FieldRegridStore()) as well as for haloing.

  ! Wrap the ESMF Array in a Field created on the nodes of the Mesh. 
  field=ESMF_FieldCreate(mesh, array=array, &
       meshLoc=ESMF_MESHLOC_NODE, rc=localrc)

We can now proceed with haloing the Field by using the ESMF_FieldHaloStore() call to create a RouteHandle, and then the ESMF_FieldHalo() call to apply the RouteHandle. Note that once the RouteHandle has been created it can be applied repeatedly to redo the halo communication as data changes in the Field.

  ! Create the RouteHandle for the halo communication.
  call ESMF_FieldHaloStore(field, routehandle=haloHandle, rc=localrc)

  ! Can repeatedly do halo as data in field changes.
  ! do t=...
 
    ! Data set in non-halo field locations. 

    ! Do the halo communication.
    call ESMF_FieldHalo(field, routehandle=haloHandle, rc=localrc)

    ! Halo locations now filled in field.

  ! enddo 

  ! After its last use the RouteHandle can be released.
  call ESMF_FieldHaloRelease(haloHandle, rc=localrc)

  ! The Field can now be destroyed.
  call ESMF_FieldDestroy(field, rc=localrc)

  ! The Array can now be destroyed.
  call ESMF_ArrayDestroy(array, rc=localrc)

33.4 Class API

33.4.1 ESMF_MeshAssignment(=) - Mesh assignment


INTERFACE:

     interface assignment(=)
     mesh1 = mesh2
ARGUMENTS:
     type(ESMF_Mesh) :: mesh1
     type(ESMF_Mesh) :: mesh2
STATUS:

DESCRIPTION:

Assign mesh1 as an alias to the same ESMF Mesh object in memory as mesh2. If mesh2 is invalid, then mesh1 will be equally invalid after the assignment.

The arguments are:

mesh1
The ESMF_Mesh object on the left hand side of the assignment.
mesh2
The ESMF_Mesh object on the right hand side of the assignment.

33.4.2 ESMF_MeshOperator(==) - Mesh equality operator


INTERFACE:

    interface operator(==)
     if (mesh1 == mesh2) then ... endif
               OR
     result = (mesh1 == mesh2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_Mesh), intent(in) :: mesh1
     type(ESMF_Mesh), intent(in) :: mesh2
STATUS:

DESCRIPTION:

Test whether mesh1 and mesh2 are valid aliases to the same ESMF Mesh object in memory. For a more general comparison of two ESMF Meshes, going beyond the simple alias test, the ESMF_MeshMatch() function (not yet implemented) must be used.

The arguments are:

mesh1
The ESMF_Mesh object on the left hand side of the equality operation.
mesh2
The ESMF_Mesh object on the right hand side of the equality operation.

33.4.3 ESMF_MeshOperator(/=) - Mesh not equal operator


INTERFACE:

   interface operator(/=)
     if (mesh1 /= mesh2) then ... endif
               OR
     result = (mesh1 /= mesh2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_Mesh), intent(in) :: mesh1
     type(ESMF_Mesh), intent(in) :: mesh2
STATUS:

DESCRIPTION:

Test whether mesh1 and mesh2 are not valid aliases to the same ESMF Mesh object in memory. For a more general comparison of two ESMF Meshes, going beyond the simple alias test, the ESMF_MeshMatch() function (not yet implemented) must be used.

The arguments are:

mesh1
The ESMF_Mesh object on the left hand side of the non-equality operation.
mesh2
The ESMF_Mesh object on the right hand side of the non-equality operation.


33.4.4 ESMF_MeshAddElements - Add elements to a Mesh


INTERFACE:

     subroutine ESMF_MeshAddElements(mesh, elementIds, elementTypes, &
                  elementConn, elementMask, elementArea, elementCoords, &
                  elementDistgrid, rc)
ARGUMENTS:
     type(ESMF_Mesh),    intent(inout)         :: mesh
     integer,            intent(in)            :: elementIds(:)
     integer,            intent(in)            :: elementTypes(:)
     integer,            intent(in)            :: elementConn(:)
     integer,            intent(in),  optional :: elementMask(:)
     real(ESMF_KIND_R8), intent(in),  optional :: elementArea(:)
     real(ESMF_KIND_R8), intent(in),  optional :: elementCoords(:)
     type(ESMF_DistGrid), intent(in), optional :: elementDistgrid
     integer,            intent(out), optional :: rc
DESCRIPTION:

This call is the third and last part of the three part mesh create sequence and should be called after the mesh is created with ESMF_MeshCreate() (33.4.6) and after the nodes are added with ESMF_MeshAddNodes() (33.4.5). This call adds the elements to the mesh and finalizes the create. After this call the Mesh is usable, for example a Field may be built on the created Mesh object and this Field may be used in a ESMF_FieldRegridStore() call.

The parameters to this call elementIds, elementTypes, and elementConn describe the elements to be created. The description for a particular element lies at the same index location in elementIds and elementTypes. Each entry in elementConn consists of the list of nodes used to create that element, so the connections for element $e$ in the elementIds array will start at $number\_of\_nodes\_in\_element(1) + number\_of\_nodes\_in\_element(2) +
\cdots + number\_of\_nodes\_in\_element(e-1) + 1$ in elementConn.

This call is collective across the current VM.

elementIds
An array containing the global ids of the elements to be created on this PET. This input consists of a 1D array the size of the number of elements on this PET. Each element id must be a number equal to or greater than 1. An id should be unique in the sense that different elements must have different ids (the same element that appears on different processors must have the same id). There may be gaps in the sequence of ids, but if these gaps are the same scale as the length of the sequence it can lead to inefficiencies when the Mesh is used (e.g. in ESMF_FieldRegridStore()).
elementTypes
An array containing the types of the elements to be created on this PET. The types used must be appropriate for the parametric dimension of the Mesh. Please see Section 33.2.1 for the list of options. This input consists of a 1D array the size of the number of elements on this PET.
elementConn
An array containing the indexes of the sets of nodes to be connected together to form the elements to be created on this PET. The entries in this list are NOT node global ids, but rather each entry is a local index (1 based) into the list of nodes which were created on this PET by the previous ESMF_MeshAddNodes() call. In other words, an entry of 1 indicates that this element contains the node described by nodeIds(1), nodeCoords(1), etc. passed into the ESMF_MeshAddNodes() call on this PET. It is also important to note that the order of the nodes in an element connectivity list matters. Please see Section 33.2.1 for diagrams illustrating the correct order of nodes in a element. This input consists of a 1D array with a total size equal to the sum of the number of nodes in each element on this PET. The number of nodes in each element is implied by its element type in elementTypes. The nodes for each element are in sequence in this array (e.g. the nodes for element 1 are elementConn(1), elementConn(2), etc.).
[elementMask]
An array containing values which can be used for element masking. Which values indicate masking are chosen via the srcMaskValues or dstMaskValues arguments to ESMF_FieldRegridStore() call. This input consists of a 1D array the size of the number of elements on this PET.
[elementArea]
An array containing element areas. If not specified, the element areas are internally calculated. This input consists of a 1D array the size of the number of elements on this PET. NOTE: ESMF doesn't currently do unit conversion on areas. If these areas are going to be used in a process that also involves the areas of another Grid or Mesh (e.g. conservative regridding), then it is the user's responsibility to make sure that the area units are consistent between the two sides. If ESMF calculates an area on the surface of a sphere, then it is in units of square radians. If it calculates the area for a Cartesian grid, then it is in the same units as the coordinates, but squared.
[elementCoords]
An array containing the physical coordinates of the elements to be created on this PET. This input consists of a 1D array the size of the number of elements on this PET times the Mesh's spatial dimension. The coordinates in this array are ordered so that the coordinates for an element lie in sequence in memory. (e.g. for a Mesh with spatial dimension 2, the coordinates for element 1 are in elementCoords(1) and elementCoords(2), the coordinates for element 2 are in elementCoords(3) and elementCoords(4), etc.).
[elementDistgrid]
If present, use this as the element Distgrid for the Mesh. The passed in Distgrid needs to contain a local set of sequence indices matching the set of local element ids (i.e. those in elementIds). However, specifying an externally created Distgrid gives the user more control over aspects of the Distgrid containing those sequence indices (e.g. how they are broken into DEs). If not present, a 1D Distgrid will be created internally consisting of one DE per PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.


33.4.5 ESMF_MeshAddNodes - Add nodes to a Mesh


INTERFACE:

     subroutine ESMF_MeshAddNodes(mesh, nodeIds, nodeCoords, nodeOwners, &
                                  nodeMask, nodalDistgrid, rc)
ARGUMENTS:
     type(ESMF_Mesh),    intent(inout)         :: mesh
     integer,            intent(in)            :: nodeIds(:)
     real(ESMF_KIND_R8), intent(in)            :: nodeCoords(:)
     integer,            intent(in),  optional :: nodeOwners(:)
     integer,            intent(in),  optional :: nodeMask(:)
     type(ESMF_DistGrid), intent(in), optional :: nodalDistgrid
     integer,            intent(out), optional :: rc
DESCRIPTION:

This call is the second part of the three part mesh create sequence and should be called after the mesh's dimensions are set using ESMF_MeshCreate() (33.4.6). This call adds the nodes to the mesh. The next step is to call ESMF_MeshAddElements() (33.4.4).

The parameters to this call nodeIds, nodeCoords, and nodeOwners describe the nodes to be created on this PET. The description for a particular node lies at the same index location in nodeIds and nodeOwners. Each entry in nodeCoords consists of spatial dimension coordinates, so the coordinates for node $n$ in the nodeIds array will start at $(n-1)*spatialDim+1$.

nodeIds
An array containing the global ids of the nodes to be created on this PET. This input consists of a 1D array the size of the number of nodes on this PET. Each node id must be a number equal to or greater than 1. An id should be unique in the sense that different nodes must have different ids (the same node that appears on different processors must have the same id). There may be gaps in the sequence of ids, but if these gaps are the same scale as the length of the sequence it can lead to inefficiencies when the Mesh is used (e.g. in ESMF_FieldRegridStore()).
nodeCoords
An array containing the physical coordinates of the nodes to be created on this PET. This input consists of a 1D array the size of the number of nodes on this PET times the Mesh's spatial dimension (spatialDim). The coordinates in this array are ordered so that the coordinates for a node lie in sequence in memory. (e.g. for a Mesh with spatial dimension 2, the coordinates for node 1 are in nodeCoords(1) and nodeCoords(2), the coordinates for node 2 are in nodeCoords(3) and nodeCoords(4), etc.).
[nodeOwners]
An array containing the PETs that own the nodes to be created on this PET. If the node is shared with another PET, the value may be a PET other than the current one. Only nodes owned by this PET will have PET local entries in a Field created on the Mesh. This input consists of a 1D array the size of the number of nodes on this PET. If not provided by the user, then ESMF will calculate node ownership.
[nodeMask]
An array containing values which can be used for node masking. Which values indicate masking are chosen via the srcMaskValues or dstMaskValues arguments to ESMF_FieldRegridStore() call. This input consists of a 1D array the size of the number of nodes on this PET.
[nodalDistgrid]
If present, use this as the node Distgrid for the Mesh. The passed in Distgrid needs to contain a local set of sequence indices matching the set of local node ids (i.e. the ids in nodeIds with nodeOwners equal to the current PET). However, specifying an externally created Distgrid gives the user more control over aspects of the Distgrid containing those sequence indices (e.g. how they are broken into DEs). If not present, a 1D Distgrid will be created internally consisting of one DE per PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.


33.4.6 ESMF_MeshCreate - Create a Mesh as a 3 step process


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function ESMF_MeshCreate3Part(parametricDim, spatialDim, coordSys, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)                                 :: ESMF_MeshCreate3Part
ARGUMENTS:
     integer,                  intent(in)            :: parametricDim
     integer,                  intent(in)            :: spatialDim
     type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
     character(len=*),         intent(in),  optional :: name
     integer,                  intent(out), optional :: rc
DESCRIPTION:

This call is the first part of the three part mesh create sequence. This call sets the dimension of the elements in the mesh (parametricDim) and the number of coordinate dimensions in the mesh (spatialDim). The next step is to call ESMF_MeshAddNodes() (33.4.5) to add the nodes and then ESMF_MeshAddElements() (33.4.4) to add the elements and finalize the mesh.

This call is collective across the current VM.

parametricDim
Dimension of the topology of the Mesh. (E.g. a mesh constructed of squares would have a parametric dimension of 2, whereas a Mesh constructed of cubes would have one of 3.)
spatialDim
The number of coordinate dimensions needed to describe the locations of the nodes making up the Mesh. For a manifold, the spatial dimension can be larger than the parametric dim (e.g. the 2D surface of a sphere in 3D space), but it can't be smaller.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.7 ESMF_MeshCreate - Create a Mesh all at once


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function ESMF_MeshCreate1Part(parametricDim, spatialDim, &
                    nodeIds, nodeCoords, nodeOwners, nodeMask, nodalDistgrid, &
                    elementIds, elementTypes, elementConn, &
                    elementMask, elementArea, elementCoords, &
                    elementDistgrid, coordSys, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)                                 :: ESMF_MeshCreate1Part
ARGUMENTS:
     integer,                  intent(in)            :: parametricDim
     integer,                  intent(in)            :: spatialDim
     integer,                  intent(in)            :: nodeIds(:)
     real(ESMF_KIND_R8),       intent(in)            :: nodeCoords(:)
     integer,                  intent(in),  optional :: nodeOwners(:)
     integer,                  intent(in),  optional :: nodeMask(:)
     type(ESMF_DistGrid),      intent(in),  optional :: nodalDistgrid
     integer,                  intent(in)            :: elementIds(:)
     integer,                  intent(in)            :: elementTypes(:)
     integer,                  intent(in)            :: elementConn(:)
     integer,                  intent(in),  optional :: elementMask(:)
     real(ESMF_KIND_R8),       intent(in),  optional :: elementArea(:)
     real(ESMF_KIND_R8),       intent(in),  optional :: elementCoords(:)
     type(ESMF_DistGrid),      intent(in),  optional :: elementDistgrid
     type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
     character(len=*),         intent(in),  optional :: name
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Create a Mesh object in one step. After this call the Mesh is usable, for example, a Field may be built on the created Mesh object and this Field may be used in a ESMF_FieldRegridStore() call.

This call sets the dimension of the elements in the mesh (parametricDim) and the number of coordinate dimensions in the mesh (spatialDim). It then creates the nodes, and then creates the elements by connecting together the nodes.

The parameters to this call nodeIds, nodeCoords, and nodeOwners describe the nodes to be created on this PET. The description for a particular node lies at the same index location in nodeIds and nodeOwners. Each entry in nodeCoords consists of spatial dimension coordinates, so the coordinates for node $n$ in the nodeIds array will start at $(n-1)*spatialDim+1$.

The parameters to this call elementIds, elementTypes, and elementConn describe the elements to be created. The description for a particular element lies at the same index location in elementIds and elementTypes. Each entry in elementConn consists of the list of nodes used to create that element, so the connections for element $e$ in the elementIds array will start at $number\_of\_nodes\_in\_element(1) + number\_of\_nodes\_in\_element(2) +
\cdots + number\_of\_nodes\_in\_element(e-1) + 1$ in elementConn.

This call is collective across the current VM.

parametricDim
Dimension of the topology of the Mesh. (E.g. a mesh constructed of squares would have a parametric dimension of 2, whereas a Mesh constructed of cubes would have one of 3.)
spatialDim
The number of coordinate dimensions needed to describe the locations of the nodes making up the Mesh. For a manifold, the spatial dimension can be larger than the parametric dim (e.g. the 2D surface of a sphere in 3D space), but it can't be smaller.
nodeIds
An array containing the global ids of the nodes to be created on this PET. This input consists of a 1D array the size of the number of nodes on this PET. Each node id must be a number equal to or greater than 1. An id should be unique in the sense that different nodes must have different ids (the same node that appears on different processors must have the same id). There may be gaps in the sequence of ids, but if these gaps are the same scale as the length of the sequence it can lead to inefficiencies when the Mesh is used (e.g. in ESMF_FieldRegridStore()).
nodeCoords
An array containing the physical coordinates of the nodes to be created on this PET. This input consists of a 1D array the size of the number of nodes on this PET times the Mesh's spatial dimension (spatialDim). The coordinates in this array are ordered so that the coordinates for a node lie in sequence in memory. (e.g. for a Mesh with spatial dimension 2, the coordinates for node 1 are in nodeCoords(1) and nodeCoords(2), the coordinates for node 2 are in nodeCoords(3) and nodeCoords(4), etc.).
[nodeOwners]
An array containing the PETs that own the nodes to be created on this PET. If the node is shared with another PET, the value may be a PET other than the current one. Only nodes owned by this PET will have PET local entries in a Field created on the Mesh. This input consists of a 1D array the size of the number of nodes on this PET. If not provided by the user, then ESMF will calculate node ownership.
[nodeMask]
An array containing values which can be used for node masking. Which values indicate masking are chosen via the srcMaskValues or dstMaskValues arguments to ESMF_FieldRegridStore() call. This input consists of a 1D array the size of the number of nodes on this PET.
[nodalDistgrid]
If present, use this as the node Distgrid for the Mesh. The passed in Distgrid needs to contain a local set of sequence indices matching the set of local node ids (i.e. the ids in nodeIds with nodeOwners equal to the current PET). However, specifying an externally created Distgrid gives the user more control over aspects of the Distgrid containing those sequence indices (e.g. how they are broken into DEs). If not present, a 1D Distgrid will be created internally consisting of one DE per PET.
elementIds
An array containing the global ids of the elements to be created on this PET. This input consists of a 1D array the size of the number of elements on this PET. Each element id must be a number equal to or greater than 1. An id should be unique in the sense that different elements must have different ids (the same element that appears on different processors must have the same id). There may be gaps in the sequence of ids, but if these gaps are the same scale as the length of the sequence it can lead to inefficiencies when the Mesh is used (e.g. in ESMF_FieldRegridStore()).
elementTypes
An array containing the types of the elements to be created on this PET. The types used must be appropriate for the parametric dimension of the Mesh. Please see Section 33.2.1 for the list of options. This input consists of a 1D array the size of the number of elements on this PET.
elementConn
An array containing the indexes of the sets of nodes to be connected together to form the elements to be created on this PET. The entries in this list are NOT node global ids, but rather each entry is a local index (1 based) into the list of nodes to be created on this PET by this call. In other words, an entry of 1 indicates that this element contains the node described by nodeIds(1), nodeCoords(1), etc. on this PET. It is also important to note that the order of the nodes in an element connectivity list matters. Please see Section 33.2.1 for diagrams illustrating the correct order of nodes in a element. This input consists of a 1D array with a total size equal to the sum of the number of nodes contained in each element on this PET. The number of nodes in each element is implied by its element type in elementTypes. The nodes for each element are in sequence in this array (e.g. the nodes for element 1 are elementConn(1), elementConn(2), etc.).
[elementMask]
An array containing values which can be used for element masking. Which values indicate masking are chosen via the srcMaskValues or dstMaskValues arguments to ESMF_FieldRegridStore() call. This input consists of a 1D array the size of the number of elements on this PET.
[elementArea]
An array containing element areas. If not specified, the element areas are internally calculated. This input consists of a 1D array the size of the number of elements on this PET. NOTE: ESMF doesn't currently do unit conversion on areas. If these areas are going to be used in a process that also involves the areas of another Grid or Mesh (e.g. conservative regridding), then it is the user's responsibility to make sure that the area units are consistent between the two sides. If ESMF calculates an area on the surface of a sphere, then it is in units of square radians. If it calculates the area for a Cartesian grid, then it is in the same units as the coordinates, but squared.
[elementCoords]
An array containing the physical coordinates of the elements to be created on this PET. This input consists of a 1D array the size of the number of elements on this PET times the Mesh's spatial dimension (spatialDim). The coordinates in this array are ordered so that the coordinates for an element lie in sequence in memory. (e.g. for a Mesh with spatial dimension 2, the coordinates for element 1 are in elementCoords(1) and elementCoords(2), the coordinates for element 2 are in elementCoords(3) and elementCoords(4), etc.).
[elementDistgrid]
If present, use this as the element Distgrid for the Mesh. The passed in Distgrid needs to contain a local set of sequence indices matching the set of local element ids (i.e. those in elementIds). However, specifying an externally created Distgrid gives the user more control over aspects of the Distgrid containing those sequence indices (e.g. how they are broken into DEs). If not present, a 1D Distgrid will be created internally consisting of one DE per PET.
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.8 ESMF_MeshCreate - Create a Mesh from a Grid


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function ESMF_MeshCreateFromGrid(grid, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)         :: ESMF_MeshCreateFromGrid
ARGUMENTS:
     type(ESMF_Grid),        intent(in)            :: grid
     character(len=*),       intent(in),  optional :: name
     integer,                intent(out), optional :: rc
DESCRIPTION:

Create an ESMF Mesh from an ESMF Grid. This method creates the elements of the Mesh from the cells of the Grid, and the nodes of the Mesh from the corners of the Grid. Corresponding locations in the Grid and new Mesh will have the same coordinates, sequence indices, masking, and area information.

This method currently only works for 2D Grids. In addition, this method requires the input Grid to have coordinates in the corner stagger location.

grid
The ESMF Grid from which to create the Mesh.
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.9 ESMF_MeshCreate - Create a Mesh from a file


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function ESMF_MeshCreateFromFile(filename, fileformat, &
                  convertToDual, addUserArea, maskFlag, varname, &
                  nodalDistgrid, elementDistgrid, &
                  coordSys, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)         :: ESMF_MeshCreateFromFile
ARGUMENTS:
     character(len=*),           intent(in)            :: filename
     type(ESMF_FileFormat_Flag), intent(in)            :: fileformat
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                    intent(in),  optional :: convertToDual
     logical,                    intent(in),  optional :: addUserArea
     type(ESMF_MeshLoc),         intent(in),  optional :: maskFlag
     character(len=*),           intent(in),  optional :: varname
     type(ESMF_DistGrid),        intent(in),  optional :: nodalDistgrid
     type(ESMF_DistGrid),        intent(in),  optional :: elementDistgrid
     type(ESMF_CoordSys_Flag),   intent(in),  optional :: coordSys
     character(len=*),           intent(in),  optional :: name
     integer,                    intent(out), optional :: rc
DESCRIPTION:

Create a Mesh from a file. Provides options to convert to 3D and in the case of SCRIP format files, allows the dual of the mesh to be created.

This call is collective across the current VM.

filename
The name of the grid file
fileformat
The file format. The valid options are ESMF_FILEFORMAT_SCRIP, ESMF_FILEFORMAT_ESMFMESH and ESMF_FILEFORMAT_UGRID. Please see Section 54.20 for a detailed description of the options.
[convertToDual]
if .true., the mesh will be converted to its dual. If not specified, defaults to .false..
[addUserArea]
if .true., the cell area will be read in from the GRID file. This feature is only supported when the grid file is in the SCRIP or ESMF format. If not specified, defaults to .false..
[maskFlag]
If maskFlag is present, generate the mask using the missing_value attribute defined in 'varname' This flag is only supported when the grid file is in the UGRID format. The value could be either ESMF_MESHLOC_NODE or ESMF_MESHLOC_ELEMENT. If the value is ESMF_MESHLOC_NODE, the node mask will be generated and the variable has to be defined on the "node" (specified by its location attribute). If the value is ESMF_MESHLOC_ELEMENT, the element mask will be generated and the variable has to be defined on the "face" of the mesh. If the variable is not defined on the right location, no mask will be generated. If not specified, no mask will be generated.
[varname]
If maskFlag is present, provide a variable name stored in the UGRID file and the mask will be generated using the missing value of the data value of this variable. The first two dimensions of the variable has to be the the longitude and the latitude dimension and the mask is derived from the first 2D values of this variable even if this data is 3D, or 4D array. If not specified, defaults to empty string.
[nodalDistgrid]
A Distgrid describing the user-specified distribution of the nodes across the PETs.
[elementDistgrid]
A Distgrid describing the user-specified distribution of the elements across the PETs.
[coordSys]
The coordinate system in which to store the mesh coordinate data. If this setting doesn't match the coordinate system in the file, then the coordinates in the file will be converted to this system during mesh creation. It is currently an error to convert Cartesian file coordinates into a spherical coordinate system. For a full list of options, please see Section 54.11. If not specified, then defaults to the coordinate system in the file.
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.10 ESMF_MeshCreate - Create a copy of a Mesh with a new distribution


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function ESMF_MeshCreateRedist(mesh, nodalDistgrid, &
       elementDistgrid, vm, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)                            :: ESMF_MeshCreateRedist
ARGUMENTS:
     type(ESMF_Mesh),          intent(in)            :: mesh
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DistGrid),      intent(in),  optional :: nodalDistgrid
     type(ESMF_DistGrid),      intent(in),  optional :: elementDistgrid
     type(ESMF_VM),            intent(in),  optional :: vm
     character(len=*),         intent(in),  optional :: name
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Create a copy of an existing Mesh with a new distribution. Information in the Mesh such as connections, coordinates, areas, masks, etc. are automatically redistributed to the new Mesh. To redistribute data in Fields built on the original Mesh create a Field on the new Mesh and then use the Field redistribution functionality (ESMF_FieldRedistStore(), etc.). The equivalent methods can also be used for data in FieldBundles.

mesh
The source Mesh to be redistributed.
[nodalDistgrid]
A Distgrid describing the new distribution of the nodes across the PETs.
[elementDistgrid]
A Distgrid describing the new distribution of the elements across the PETs.
[vm]
If present, the Mesh object is created on the specified ESMF_VM object. The default is to create on the VM of the current context.
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.11 ESMF_MeshCreate - Create a Mesh of just one element type using corner coordinates


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function  ESMF_MeshCreateEasyElems1Type(parametricDim, coordSys, &
                    elementIds, elementType, elementCornerCoords, &
                    elementMask, elementArea, elementCoords, &
                    elementDistgrid, rc)
RETURN VALUE:
     type(ESMF_Mesh) :: ESMF_MeshCreateEasyElems1Type
ARGUMENTS:
     integer,                  intent(in)            :: parametricDim
     type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
     integer,                  intent(in),  optional :: elementIds(:)
     integer,                  intent(in)            :: elementType
     real(ESMF_KIND_R8),       intent(in)            :: elementCornerCoords(:,:,:)
     integer,                  intent(in),  optional :: elementMask(:)
     real(ESMF_KIND_R8),       intent(in),  optional :: elementArea(:)
     real(ESMF_KIND_R8),       intent(in),  optional :: elementCoords(:,:)
     type(ESMF_DistGrid),      intent(in),  optional :: elementDistgrid
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Create a Mesh object in one step by just specifying the corner coordinates of each element. Internally these corners are turned into nodes forming the outside edges of the elements. This call assumes that each element is the same type to make the specification of the elements a bit easier. After this call the Mesh is usable, for example, a Field may be built on the created Mesh object and this Field may be used in ESMF_FieldRegridStore(). However, the Mesh created by this call consists of a set of disconnected elements, and so shouldn't be used in a situation where connections between elements are necessary (e.g. bilinear regridding on element centers, patch regridding, or second-order conservative regridding).

This call sets the dimension of the elements in the Mesh via parametricDim and the number of coordinate dimensions in the mesh is determined from the first dimension of elementCornerCoords.

The parameters to this call elementIds, elementTypes, and elementCornerCoords describe the elements to be created. The description for a particular element lies at the same index location in elementIds and elementTypes. The argument elementCornerCoords contains the coordinates of the corners used to create each element. The first dimension of this argument are across the coordinate dimensions. The second dimension of this argument is across the corners of a particular element. The last dimension of this argument is across the list of elements on this PET, so the coordinates of corner c in element e on this PET would be in elementCornerCoords(:,c,e).

This call is collective across the current VM.

parametricDim
Dimension of the topology of the Mesh. (E.g. a mesh constructed of squares would have a parametric dimension of 2, whereas a Mesh constructed of cubes would have one of 3.)
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[elementIds]
An array containing the global ids of the elements to be created on this PET. This input consists of a 1D array the size of the number of elements on this PET. Each element id must be a number equal to or greater than 1. An id should be unique in the sense that different elements must have different ids (the same element that appears on different processors must have the same id). There may be gaps in the sequence of ids, but if these gaps are the same scale as the length of the sequence it can lead to inefficiencies when the Mesh is used (e.g. in ESMF_FieldRegridStore()). If not specified, then elements are numbered in sequence starting with the first element on PET 0.
elementType
An variable containing the type of the elements to be created in this Mesh. The type used must be appropriate for the parametric dimension of the Mesh. Please see Section 33.2.1 for the list of options.
elementCornerCoords
A 3D array containing the coordinates of the corners of the elements to be created on this PET. The first dimension of this array is for the coordinates and should be of size 2 or 3. The size of this dimension will be used to determine the spatialDim of the Mesh. The second dimension is the number of corners for an element. The 3rd dimension is a list of all the elements on this PET.
[elementMask]
An array containing values which can be used for element masking. Which values indicate masking are chosen via the srcMaskValues or dstMaskValues arguments to ESMF_FieldRegridStore() call. This input consists of a 1D array the size of the number of elements on this PET.
[elementArea]
An array containing element areas. If not specified, the element areas are internally calculated. This input consists of a 1D array the size of the number of elements on this PET. NOTE: ESMF doesn't currently do unit conversion on areas. If these areas are going to be used in a process that also involves the areas of another Grid or Mesh (e.g. conservative regridding), then it is the user's responsibility to make sure that the area units are consistent between the two sides. If ESMF calculates an area on the surface of a sphere, then it is in units of square radians. If it calculates the area for a Cartesian grid, then it is in the same units as the coordinates, but squared.
[elementCoords]
An array containing the physical coordinates of the elements to be created on this PET. This input consists of a 2D array with the first dimension that same size as the first dimension of elementCornerCoords. The second dimension should be the same size as the elementTypes argument.
[elementDistgrid]
If present, use this as the element Distgrid for the Mesh. The passed in Distgrid needs to contain a local set of sequence indices matching the set of local element ids (i.e. those in elementIds). However, specifying an externally created Distgrid gives the user more control over aspects of the Distgrid containing those sequence indices (e.g. how they are broken into DEs). If not present, a 1D Distgrid will be created internally consisting of one DE per PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.12 ESMF_MeshCreate - Create a Mesh using element corner coordinates


INTERFACE:

   ! Private name; call using ESMF_MeshCreate()
     function  ESMF_MeshCreateEasyElemsGen(parametricDim, coordSys, &
                    elementIds, elementTypes, elementCornerCoords, &
                    elementMask, elementArea, elementCoords, &
                    elementDistgrid, rc)
RETURN VALUE:
     type(ESMF_Mesh) :: ESMF_MeshCreateEasyElemsGen
ARGUMENTS:
     integer,                  intent(in)            :: parametricDim
     type(ESMF_CoordSys_Flag), intent(in),  optional :: coordSys
     integer,                  intent(in),  optional :: elementIds(:)
     integer,                  intent(in)            :: elementTypes(:)
     real(ESMF_KIND_R8),       intent(in)            :: elementCornerCoords(:,:)
     integer,                  intent(in),  optional :: elementMask(:)
     real(ESMF_KIND_R8),       intent(in),  optional :: elementArea(:)
     real(ESMF_KIND_R8),       intent(in),  optional :: elementCoords(:,:)
     type(ESMF_DistGrid),      intent(in),  optional :: elementDistgrid
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Create a Mesh object in one step by just specifying the corner coordinates of each element. Internally these corners are turned into nodes forming the outside edges of the elements. After this call the Mesh is usable, for example, a Field may be built on the created Mesh object and this Field may be used in ESMF_FieldRegridStore(). However, the Mesh created by this call consists of a set of disconnected elements, and so shouldn't be used in a situation where connections between elements are necessary (e.g. bilinear regridding on element centers, patch regridding, or second-order conservative regridding).

This call sets the dimension of the elements in the Mesh via parametricDim and the number of coordinate dimensions in the mesh is determined from the first dimension of elementCornerCoords.

The parameters to this call elementIds, elementTypes, and elementCornerCoords describe the elements to be created. The description for a particular element lies at the same index location in elementIds and elementTypes. The argument elementCornerCoords consists of a list of all the corners used to create all the elements, so the corners for element $e$ in the elementTypes array will start at $number\_of\_corners\_in\_element(1)
+ number\_of\_corners\_in\_element(2) +
\cdots + number\_of\_corners\_in\_element(e-1) + 1$ in elementCornerCoords.

This call is collective across the current VM.

parametricDim
Dimension of the topology of the Mesh. (E.g. a mesh constructed of squares would have a parametric dimension of 2, whereas a Mesh constructed of cubes would have one of 3.)
[coordSys]
The coordinate system of the grid coordinate data. For a full list of options, please see Section 54.11. If not specified then defaults to ESMF_COORDSYS_SPH_DEG.
[elementIds]
An array containing the global ids of the elements to be created on this PET. This input consists of a 1D array the size of the number of elements on this PET. Each element id must be a number equal to or greater than 1. An id should be unique in the sense that different elements must have different ids (the same element that appears on different processors must have the same id). There may be gaps in the sequence of ids, but if these gaps are the same scale as the length of the sequence it can lead to inefficiencies when the Mesh is used (e.g. in ESMF_FieldRegridStore()). If not specified, then elements are numbered in sequence starting with the first element on PET 0.
elementTypes
An array containing the types of the elements to be created on this PET. The types used must be appropriate for the parametric dimension of the Mesh. Please see Section 33.2.1 for the list of options. This input consists of a 1D array the size of the number of elements on this PET.
elementCornerCoords
A 2D array containing the coordinates of the corners of the elements to be created on this PET. The first dimension of this array is for the coordinates and should be of size 2 or 3. The size of this dimension will be used to determine the spatialDim of the Mesh. The second dimension is a collapsed list of all the corners in all the elements. The list of corners has been collapsed to 1D to enable elements with different number of corners to be supported in the same list without wasting space. The number of corners in each element is implied by its element type in elementTypes. The corners for each element are in sequence in this array (e.g. If element 1 has 3 corners then they are in elementCornerCoords(:,1), elementCornerCoords(:,2), elementCornerCoords(:,3) and the corners for the next element start in elementCornerCoords(:,4)).
[elementMask]
An array containing values which can be used for element masking. Which values indicate masking are chosen via the srcMaskValues or dstMaskValues arguments to ESMF_FieldRegridStore() call. This input consists of a 1D array the size of the number of elements on this PET.
[elementArea]
An array containing element areas. If not specified, the element areas are internally calculated. This input consists of a 1D array the size of the number of elements on this PET. NOTE: ESMF doesn't currently do unit conversion on areas. If these areas are going to be used in a process that also involves the areas of another Grid or Mesh (e.g. conservative regridding), then it is the user's responsibility to make sure that the area units are consistent between the two sides. If ESMF calculates an area on the surface of a sphere, then it is in units of square radians. If it calculates the area for a Cartesian grid, then it is in the same units as the coordinates, but squared.
[elementCoords]
An array containing the physical coordinates of the elements to be created on this PET. This input consists of a 2D array with the first dimension that same size as the first dimension of elementCornerCoords. The second dimension should be the same size as the elementTypes argument.
[elementDistgrid]
If present, use this as the element Distgrid for the Mesh. The passed in Distgrid needs to contain a local set of sequence indices matching the set of local element ids (i.e. those in elementIds). However, specifying an externally created Distgrid gives the user more control over aspects of the Distgrid containing those sequence indices (e.g. how they are broken into DEs). If not present, a 1D Distgrid will be created internally consisting of one DE per PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.13 ESMF_MeshCreateCubedSphere - Create a Mesh representation of a cubed sphere grid


INTERFACE:

 function ESMF_MeshCreateCubedSphere(tileSize, nx, ny, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)         :: ESMF_MeshCreateCubedSphere
ARGUMENTS:
     integer,                  intent(in)            :: tileSize
     integer,                  intent(in)            :: nx
     integer,                  intent(in)            :: ny
     character(len=*),         intent(in),  optional :: name
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Create a ESMF_Mesh object for a cubed sphere grid using identical regular decomposition for every tile. The grid coordinates are generated based on the algorithm used by GEOS-5, The tile resolution is defined by tileSize. Each tile is decomposed into nx x ny blocks and the total number of DEs used is nx x ny x 6. If the total PET is not equal to the number of DEs, the DEs are distributed into PETs in the default cyclic distribution. Internally, the nodes and the elements from multiple DEs are collapsed into a 1D array. Therefore, the nodal distgrid or the element distgrid attached to the Mesh object is always a one DE arbitrarily distributed distgrid. The sequential indices of the nodes and the elements are derived based on the location of the point in the Cubed Sphere grid. If an element is located at (x, y) of tile n. Its sequential index would be (n-1)*tileSize*tileSize+(y-1)*tileSize+x. If it is a node, its sequential index would be (n-1)*(tileSize+1)*(tileSize+1)+(y-1)*(tileSize+1)+x.

The arguments are:

tilesize
The number of elements on each side of the tile of the Cubed Sphere grid
nx
The number of blocks on the horizontal size of each tile
ny
The number of blocks on the vertical size of each tile
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.14 ESMF_MeshDestroy - Release resources associated with a Mesh


INTERFACE:

       subroutine ESMF_MeshDestroy(mesh, noGarbage, rc)
RETURN VALUE:
ARGUMENTS:
     type(ESMF_Mesh), intent(inout)          :: mesh
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,         intent(in),   optional :: noGarbage
     integer,         intent(out),  optional :: rc
STATUS:

DESCRIPTION:

This call removes internal memory associated with mesh. After this call mesh will no longer be usable. ! ! The arguments are:

mesh
Mesh object to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.15 ESMF_MeshEmptyCreate - Create a Mesh to hold Distgrid information


INTERFACE:

     function ESMF_MeshEmptyCreate(nodalDistgrid, elementDistgrid, name, rc)
RETURN VALUE:
     type(ESMF_Mesh)         :: ESMF_MeshEmptyCreate
ARGUMENTS:
     type(ESMF_DistGrid),      intent(in),  optional :: elementdistgrid
     type(ESMF_DistGrid),      intent(in),  optional :: nodalDistgrid
     character(len=*),         intent(in),  optional :: name
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Create a Mesh to hold distribution information (i.e. Distgrids). Such a mesh will have no coordinate or connectivity information stored. Aside from holding distgrids the Mesh created by this call can't be used in other ESMF functionality (e.g. it can't be used to create a Field or in regridding).

[nodalDistgrid]
The nodal distgrid.
[elementDistgrid]
The elemental distgrid.
[name]
The name of the Mesh.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.16 ESMF_MeshFreeMemory - Remove a Mesh and its memory


INTERFACE:

       subroutine ESMF_MeshFreeMemory(mesh, rc)
RETURN VALUE:
ARGUMENTS:
     type(ESMF_Mesh), intent(inout)        :: mesh
     integer,        intent(out), optional :: rc
DESCRIPTION:

This call removes the portions of mesh which contain connection and coordinate information. After this call, Fields build on mesh will no longer be usable as part of an ESMF_FieldRegridStore() operation. However, after this call Fields built on mesh can still be used in an ESMF_FieldRegrid() operation if the routehandle was generated beforehand. New Fields may also be built on mesh after this call.

The arguments are:

mesh
Mesh object whose memory is to be freed.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.17 ESMF_MeshGet - Get Mesh information


INTERFACE:

       subroutine ESMF_MeshGet(mesh, parametricDim, spatialDim, &
                    nodeCount, nodeIds, nodeCoords, nodeOwners, &
                    nodeMaskIsPresent, nodeMask,&
                    elementCount, elementIds, elementTypes, &
                    elementConnCount, elementConn, &
                    elementMaskIsPresent,elementMask, &
                    elementAreaIsPresent, elementArea, &
                    elementCoordsIsPresent, elementCoords, &
                    nodalDistgridIsPresent, nodalDistgrid, &
                    elementDistgridIsPresent, elementDistgrid, &
                    numOwnedNodes, ownedNodeCoords, &
                    numOwnedElements, ownedElemCoords, &
                    elemMaskArray, elemAreaArray, &
                    isMemFreed, coordSys, status, name, rc)
RETURN VALUE:
ARGUMENTS:
     type(ESMF_Mesh),          intent(in)            :: mesh
     integer,                  intent(out), optional :: parametricDim
     integer,                  intent(out), optional :: spatialDim
     integer,                  intent(out), optional :: nodeCount
     integer,                  intent(out), optional :: nodeIds(:)
     real(ESMF_KIND_R8),       intent(out), optional :: nodeCoords(:)
     integer,                  intent(out), optional :: nodeOwners(:)
     logical,                  intent(out), optional :: nodeMaskIsPresent
     integer,                  intent(out), optional :: nodeMask(:)
     integer,                  intent(out), optional :: elementCount
     integer,                  intent(out), optional :: elementIds(:)
     integer,                  intent(out), optional :: elementTypes(:)
     integer,                  intent(out), optional :: elementConnCount
     integer,                  intent(out), optional :: elementConn(:)
     logical,                  intent(out), optional :: elementMaskIsPresent
     integer,                  intent(out), optional :: elementMask(:)
     logical,                  intent(out), optional :: elementAreaIsPresent
     real(ESMF_KIND_R8),       intent(out), optional :: elementArea(:)
     logical,                  intent(out), optional :: elementCoordsIsPresent
     real(ESMF_KIND_R8),       intent(out), optional :: elementCoords(:)
     logical,                  intent(out), optional :: nodalDistgridIsPresent
     type(ESMF_DistGrid),      intent(out), optional :: nodalDistgrid
     logical,                  intent(out), optional :: elementDistgridIsPresent
     type(ESMF_DistGrid),      intent(out), optional :: elementDistgrid
     integer,                  intent(out), optional :: numOwnedNodes
     real(ESMF_KIND_R8),       intent(out), optional :: ownedNodeCoords(:)
     integer,                  intent(out), optional :: numOwnedElements
     real(ESMF_KIND_R8),       intent(out), optional :: ownedElemCoords(:)
     logical,                  intent(out), optional :: isMemFreed
     type(ESMF_Array),         intent(inout), optional :: elemMaskArray
     type(ESMF_Array),         intent(inout), optional :: elemAreaArray
     type(ESMF_CoordSys_Flag), intent(out), optional :: coordSys
     type(ESMF_MeshStatus_Flag),intent(out), optional :: status
     character(len=*),         intent(out), optional :: name
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Get various information from a mesh.

The arguments are:

mesh
Mesh object to retrieve information from.
[parametricDim]
Dimension of the topology of the Mesh. (E.g. a mesh constructed of squares would have a parametric dimension of 2, whereas a Mesh constructed of cubes would have one of 3.)
[spatialDim]
The number of coordinate dimensions needed to describe the locations of the nodes making up the Mesh. For a manifold, the spatial dimension can be larger than the parametric dim (e.g. the 2D surface of a sphere in 3D space), but it can't be smaller.
[nodeCount]
The number of local nodes in the mesh (both owned and shared with another PET).
[nodeIds]
An array of ids for each local node in the mesh. The nodeIds array should be of size nodeCount.
[nodeCoords]
An array of coordinates for each local node in the mesh. The nodeCoords array should be of size (spatialDim*nodeCount).
[nodeOwners]
An array of the PET numbers that own each local node in the mesh. The nodeOwners array should be of size nodeCount.
[nodeMaskIsPresent]
.true. if node masking was set in mesh, .false. otherwise.
[nodeMask]
An array of mask values for each local node in the mesh. The nodeOwners array should be of size nodeCount.
[elementCount]
The number of local elements in the mesh (both owned and shared with another PET).
[elementIds]
An array of ids for each local element in the mesh. The elementIds array should be of size elementCount.
[elementTypes]
An array of types for each local element in the mesh. Please see section 33.2.1 for the list of options. The elementTypes array should be of size elementCount.
[elementConnCount]
The number of entries elementConn array. Provided as a convenience.
elementConn
An array containing the indexes of the sets of nodes to be connected together to form the elements to be created on this PET. The entries in this list are NOT node global ids, but rather each entry is a local index (1 based) into the list of nodes to be created on this PET by this call. In other words, an entry of 1 indicates that this element contains the node described by nodeIds(1), nodeCoords(1), etc. on this PET. It is also important to note that the order of the nodes in an element connectivity list matters. Please see Section 33.2.1 for diagrams illustrating the correct order of nodes in a element. This input consists of a 1D array with a total size equal to the sum of the number of nodes contained in each element on this PET (also provided by elementConnCount). The number of nodes in each element is implied by its element type in elementTypes. The nodes for each element are in sequence in this array (e.g. the nodes for element 1 are elementConn(1), elementConn(2), etc.).
[elementMaskIsPresent]
.true. if element masking was set in mesh, .false. otherwise.
[elementMask]
An array of mask values for each local element in the mesh. The elementMask array should be of size elementCount.
[elementAreaIsPresent]
.true. if element areas were set in mesh, .false. otherwise.
[elementArea]
An array of area values for each local element in the mesh. The elementArea array should be of size elementCount.
[elementCoordsIsPresent]
.true. if element coordinates were set in mesh, .false. otherwise.
[elementCoords]
An array of coordinate values for each local element in the mesh. The elementCoord array should be of size (spatialDim*elementCount).
[nodalDistgridIsPresent]
.true. if nodalDistgrid was set in Mesh object, .false. otherwise.
[nodalDistgrid]
A Distgrid describing the distribution of the nodes across the PETs. Note that on each PET the distgrid will only contain entries for nodes owned by that PET. This is the DistGrid that would be used to construct the Array in a Field that is constructed on mesh.
[elementDistgridIsPresent]
.true. if elementDistgrid was set in Mesh object, .false. otherwise.
[elementDistgrid]
A Distgrid describing the distribution of elements across the PETs. Note that on each PET the distgrid will only contain entries for elements owned by that PET.
[numOwnedNodes]
The number of local nodes which are owned by this PET. This is the number of PET local entries in the nodalDistgrid.
[ownedNodeCoords]
The coordinates for the local nodes. These coordinates will be in the proper order to correspond with the nodes in the nodalDistgrid returned by this call, and hence with a Field built on mesh. The size of the input array should be the spatial dim of mesh times numOwnedNodes.
[numOwnedElements]
The number of local elements which are owned by this PET. Note that every element is owned by the PET it resides on, so unlike for nodes, numOwnedElements is identical to the number of elements on the PET. It is also the number of PET local entries in the elementDistgrid.
[ownedElemCoords]
The center coordinates for the local elements. These coordinates will be in the proper order to correspond with the elements in the elementDistgrid returned by this call, and hence with a Field built on the center of mesh. The size of the input array should be the spatial dim of mesh times numOwnedElements.
[elemMaskArray]
The mask information for elements put into an ESMF Array. The ESMF Array must be build on a DistGrid which matches the elementDistgrid.
[elemAreaArray]
The area information for elements put into an ESMF Array. The ESMF Array must be build on a DistGrid which matches the elementDistgrid.
[isMemFreed]
Indicates if the coordinate and connection memory been freed from mesh. If so, it can no longer be used as part of an ESMF_FieldRegridStore() call.
[coordSys]
The coordinate system of the grid coordinate data.
[status]
Flag indicating the status of the Mesh. Please see Section 54.43 for the list of options.
[name]
Name of the Mesh object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.18 ESMF_MeshGetMOAB - Check on status of using MOAB library internally.


INTERFACE:

    subroutine ESMF_MeshGetMOAB(moabOn, rc)
ARGUMENTS:
     logical, intent(out)                      :: moabOn
     integer, intent(out), optional            :: rc
DESCRIPTION:

This method is only temporary. It was created to enable testing during the stage in ESMF development while we have two internal mesh implementations. At some point it will be removed.

This method can be used to check whether the MOAB library is being used to hold the internal structure of the Mesh. When set to .true. the following Mesh create calls create a Mesh using MOAB internally. When set to .false. the following Mesh create calls use the ESMF native internal mesh respresentation. Note that ESMF Meshes created on MOAB are only supported in a limited set of operations and should be used with caution as they haven't yet been tested as thoroughly as the native version. Also, operations that use a pair of Meshes (e.g. regrid weight generation) are only supported between meshes of the same type (e.g. you can regrid between two MOAB meshes, but not between a MOAB and a native mesh).

moabOn
Output variable which indicates current state of MOAB.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.19 ESMF_MeshIsCreated - Check whether a Mesh object has been created


INTERFACE:

   function ESMF_MeshIsCreated(mesh, rc)
RETURN VALUE:
     logical :: ESMF_MeshIsCreated
ARGUMENTS:
     type(ESMF_Mesh), intent(in)            :: mesh
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,             intent(out), optional :: rc
DESCRIPTION:

Return .true. if the mesh has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

mesh
ESMF_Mesh queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.20 ESMF_MeshSet - Set some Mesh information


INTERFACE:

       subroutine ESMF_MeshSet(mesh, &
            elementMask, elementArea, rc)
RETURN VALUE:
ARGUMENTS:
     type(ESMF_Mesh),          intent(in)            :: mesh
     integer,                  intent(in), optional :: elementMask(:)
     real(ESMF_KIND_R8),       intent(in), optional :: elementArea(:)
     integer,                  intent(out), optional :: rc
DESCRIPTION:

This call allows the user to change the set of information that it's legal to alter after a mesh has been created. Currently, this call requires that the information has already been added to the mesh during creation. For example, you can only change the element mask information, if the mesh was initially created with element masking.

The arguments are:

mesh
[elementMask]
An array of mask values for each local element in the mesh. The elementMask array should be of size elementCount.
[elementArea]
An array of area values for each local element in the mesh. The elementArea array should be of size elementCount.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

33.4.21 ESMF_MeshSetMOAB - Toggle using the MOAB library internally.


INTERFACE:

    subroutine ESMF_MeshSetMOAB(moabOn, rc)
ARGUMENTS:
     logical, intent(in)                        :: moabOn
     integer, intent(out) , optional            :: rc
DESCRIPTION:

This method is only temporary. It was created to enable testing during the stage in ESMF development while we have two internal mesh implementations. At some point it will be removed.

This method can be employed to turn on or off using the MOAB library to hold the internal structure of the Mesh. When set to .true. the following Mesh create calls create a Mesh using MOAB internally. When set to .false. the following Mesh create calls use the ESMF native internal mesh respresentation. Note that ESMF Meshes created on MOAB are only supported in a limited set of operations and should be used with caution as they haven't yet been tested as thoroughly as the native version. Also, operations that use a pair of Meshes (e.g. regrid weight generation) are only supported between meshes of the same type (e.g. you can regrid between two MOAB meshes, but not between a MOAB and a native mesh).

moabOn
Variable used to turn MOAB on or off
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

34 XGrid Class

34.1 Description

An exchange grid represents the 2D boundary layer usually between the atmosphere on one side and ocean and land on the other in an Earth system model. There are dynamical and thermodynamical processes on either side of the boundary layer and on the boundary layer itself. The boundary layer exchanges fluxes from either side and adjusts boundary conditions for the model components involved. For climate modeling, it is critical that the fluxes transferred by the boundary layer are conservative.

The ESMF exchange grid is implemented as the ESMF_XGrid class. Internally it's represented by a collection of the intersected cells between atmosphere and ocean/land[17] grids. These polygonal cells can have irregular shapes and can be broken down into triangles facilitating a finite element approach.

There are two ways to create an ESMF_XGrid object from user supplied information. The first way to create an ESMF_XGrid takes two lists of ESMF_Grid or ESMF_Mesh that represent the model component grids on either side of the exchange grid. From the two lists of ESMF_Grid or ESMF_Mesh, information required for flux exchange calculation between any pair of the model components from either side of the exchange grid is computed. In addition, the internal representation of the ESMF_XGrid is computed and can be optionally stored as an ESMF_Mesh. This internal representation is the collection of the intersected polygonal cells as a result of merged ESMF_Meshes from both sides of the exchange grid. ESMF_Field can be created on the ESMF_XGrid and used for weight generation and regridding as the internal representation in the ESMF_XGrid has a complete geometrical description of the exchange grid.

The second way to create an ESMF_XGrid requires users to supply all necessary information to compute communication routehandle. A later call to ESMF_FieldRegridStore() with the xgrid and source and destination ESMF_Fields computes the ESMF_Routehandle object for matrix multiply operation used in model remapping.

ESMF_XGrid deals with 2 distinctive kinds of fraction for each Grid or Mesh cell involved in its creation. The following description applies to both ESMF_Grid and ESMF_Mesh involved in the ESMF_XGrid creation process. The first fraction quantity $f_1$ is the same as defined in direct Field regrid between a source and destination ESMF_Field pair, namely the fraction of a total Grid cell area $A$ that is used in weight generation. The second fraction quantity $f_2$ is a result of the Grid merging process when multiple ESMF_Grids or model components exist on one side of the exchange grid. To compute XGrid, the multiple ESMF_Grids are first merged together to form a super mesh. During the merging process, Grids that are of a higher priority clips into lower priority Grids, creating fractional cells in the lower priority Grids. Priority is a mechanism to resolve the claim of a surface region by multiple Grids. To conserve flux, any surface area can only be claimed by a unique Grid. This is a typical practice in earth system modelling, e.g. to handle land and ocean boundary.

In addition to the matrix multiply communication routehandle, ESMF_XGrid exports both $f_1$ and $f_2$ to the user through the ESMF_FieldRegridStore() method because each remapping pair has different $f_1$ and $f_2$ associated with it. $f_2$ from source Grid is folded directly in the calculated weight matrices since its used to calculate destination point flux density $F$. The global source flux is defined as $\sum_{g=1}^{g=n\_srcgrid}\sum_{s=1}^{s=n\_srccell}{ f_{1s} f_{2s} A_s F_s }$. The global destination flux is defined as: $\sum_{g=1}^{g=n\_dstgrid}\sum_{d=1}^{d=n\_dstcell}{ \sum_{s=1}^{s=n\_intersect}(w_{sd} F_s) f_{2d} A_d}$, $w_{sd}$ is the $f_2$ modified weight intersecting s-th source Grid cell with d-th destination Grid cell. It can be proved that this formulation of the fractions and weight calculation ensures first order global conservation of flux $\mathcal{F}$ transferred from source grids to exchange grid, and from exchange grid to destination grids.

34.2 Constants


34.2.1 ESMF_XGRIDSIDE

DESCRIPTION:
Specify which side of the ESMF_XGrid the current operation is taking place.

The type of this flag is:

type(ESMF_XGridSide_Flag)

The valid values are:

ESMF_XGRIDSIDE_A
A side of the eXchange Grid, corresponding to the A side of the Grids used to create an XGrid.
ESMF_XGRIDSIDE_B
B side of the eXchange Grid, corresponding to the B side of the Grids used to create an XGrid.
ESMF_XGRIDSIDE_BALANCED
The internally generated balanced side of the eXchange Grid in the middle.

34.3 Use and Examples


34.3.1 Create an XGrid from Grids then use it for regridding

An ESMF_XGrid object can be created from Grids on either side of the exchange grid. Internally the weight matrices and index mapping are computed and stored in the XGrid, along with other necessary information for flux exchange calculation between any pair of model components used for the XGrid creation.

In this example, we create an XGrid from overlapping Grids on either side of the XGrid. Then we perform a flux exchange from one side to the other side of the XGrid.

We start by creating the Grids on both sides and associate coordinates with the Grids on the corner stagger. The Grids use global indexing and padding for coordinates on the corner stagger.

For details of Grid creation and coordinate use, please refer to Grid class documentation: 31.3.2.

    ! First Grid on side A
    sideA(1) = ESMF_GridCreateNoPeriDim(maxIndex=(/20, 20/), &
      indexflag=ESMF_INDEX_GLOBAL, &
      gridEdgeLWidth=(/0,0/), gridEdgeUWidth=(/1,1/), &
      name='source Grid 1 on side A', rc=localrc)

    ! Second Grid on side A
    sideA(2) = ESMF_GridCreateNoPeriDim(maxIndex=(/20, 10/), &
      indexflag=ESMF_INDEX_GLOBAL, &
      gridEdgeLWidth=(/0,0/), gridEdgeUWidth=(/1,1/), &
      name='source Grid 2 on side A', rc=localrc)

    ! Allocate coordinates for Grid corner stagger
    do i = 1, 2
      call ESMF_GridAddCoord(sideA(i), staggerloc=ESMF_STAGGERLOC_CORNER, &
          rc=localrc)

    enddo

Assign coordinate for the Grids on sideA at corner stagger.

    ! SideA first grid spans (0-20, 0-20) with 1.0x1.0 degree resolution
    ! X corner
    call ESMF_GridGetCoord(sideA(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CORNER, coordDim=1, &
        farrayPtr=coordX, rc=localrc)

    ! Y corner
    call ESMF_GridGetCoord(sideA(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CORNER, coordDim=2, &
        farrayPtr=coordY, rc=localrc)

    do i = lbound(coordX,1), ubound(coordX,1)
      do j = lbound(coordX, 2), ubound(coordX, 2)
        coordX(i,j) = (i-1)*1.0
        coordY(i,j) = (j-1)*1.0
      enddo
    enddo

    ! SideA second grid spans (14.3-24.3, 14.2-24.2) with 0.5x1.0 degree
    ! resolution X corner
    call ESMF_GridGetCoord(sideA(2), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CORNER, coordDim=1, &
        farrayPtr=coordX, rc=localrc)

    ! Y corner
    call ESMF_GridGetCoord(sideA(2), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CORNER, coordDim=2, &
        farrayPtr=coordY, rc=localrc)

    do i = lbound(coordX,1), ubound(coordX,1)
      do j = lbound(coordX, 2), ubound(coordX, 2)
        coordX(i,j) = 14.3+(i-1)*0.5
        coordY(i,j) = 14.2+(j-1)*1.0
      enddo
    enddo

Create the destination grid on side B, only one Grid exists on side B. Also associate coordinate with the Grid:

    sideB(1) = ESMF_GridCreateNoPeriDim(maxIndex=(/30, 30/), &
      indexflag=ESMF_INDEX_GLOBAL, &
      gridEdgeLWidth=(/0,0/), gridEdgeUWidth=(/1,1/), &
      name='source Grid 1 on side B', rc=localrc)

    do i = 1, 1
      call ESMF_GridAddCoord(sideB(i), staggerloc=ESMF_STAGGERLOC_CORNER, &
          rc=localrc)

    enddo

    ! SideB grid spans (0-30, 0-30) with 1.0x1.0 degree resolution
    ! X corner
    call ESMF_GridGetCoord(sideB(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CORNER, coordDim=1, &
        farrayPtr=coordX, rc=localrc)

    ! Y corner
    call ESMF_GridGetCoord(sideB(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CORNER, coordDim=2, &
        farrayPtr=coordY, rc=localrc)

    do i = lbound(coordX,1), ubound(coordX,1)
      do j = lbound(coordX, 2), ubound(coordX, 2)
        coordX(i,j) = (i-1)*1.0
        coordY(i,j) = (j-1)*1.0
      enddo
    enddo

Create an ESMF_XGrid object from the two lists of Grids on side A and B. In this example both Grids on side A overlaps with the Grid on side B. It's an error to have a Grid on either side that is spatially disjoint with the XGrid. Neither of the Grid on side A is identical to the Grid on side B. Calling the ESMF_XGridCreate() method is straightforward:

    xgrid = ESMF_XGridCreate(sideAGrid=sideA, sideBGrid=sideB, rc=localrc)

Create an ESMF_Field on the XGrid:

    field = ESMF_FieldCreate(xgrid, typekind=ESMF_TYPEKIND_R8, &
                rc=localrc)

Query the Field for its Fortran data pointer and its exclusive bounds:

    call ESMF_FieldGet(field, farrayPtr=xfarrayPtr, &
        exclusiveLBound=xlb, exclusiveUBound=xub, rc=localrc)

Create src and dst Fields on side A and side B Grids.

    do i = 1, 2
        srcField(i) = ESMF_FieldCreate(sideA(i), &
                typekind=ESMF_TYPEKIND_R8, rc=localrc)

    enddo
    do i = 1, 1
        dstField(i) = ESMF_FieldCreate(sideB(i), &
                typekind=ESMF_TYPEKIND_R8, rc=localrc)

    enddo

The current implementation requires that Grids used to generate the XGrid must not match, i.e. they are different either topologically or geometrically or both. In this example, the first source Grid is topologically identical to the destination Grid but their geometric coordinates are different.

First we compute the regrid routehandles, these routehandles can be used repeatedly afterwards. Then we initialize the values in the Fields. Finally we execute the Regrid.

    ! Compute regrid routehandles. The routehandles can be used
    ! repeatedly afterwards.
    ! From A -> X
    do i = 1, 2
      call ESMF_FieldRegridStore(xgrid, srcField(i), field, &
        routehandle=rh_src2xgrid(i), rc = localrc)

    enddo
    ! from X -> B, retrieve the destination fraction Fields.
    do i = 1, 1
      call ESMF_FieldRegridStore(xgrid, field, dstField(i), &
        dstFracField=dstFrac, dstMergeFracField=dstFrac2, &
        routehandle=rh_xgrid2dst(i), rc = localrc)

    enddo

    ! Initialize values in the source Fields on side A
    do i = 1, 2
      call ESMF_FieldGet(srcField(i), farrayPtr=farrayPtr, rc=localrc)

      farrayPtr = i
    enddo
    ! Initialize values in the destination Field on XGrid
    xfarrayPtr = 0.0
    ! Initialize values in the destination Field on Side B
    do i = 1, 1
      call ESMF_FieldGet(dstField(i), farrayPtr=farrayPtr, rc=localrc)

      farrayPtr = 0.0
    enddo

First we regrid from the Fields on side A to the Field on the XGrid:

    ! Execute regrid from A -> X
    do i = 1, 2
      call ESMF_FieldRegrid(srcField(i), field, &
        routehandle=rh_src2xgrid(i), &
        zeroregion=ESMF_REGION_SELECT, rc = localrc)

    enddo

Next we regrid from the Field on XGrid to the destination Field on side B:

    ! Execute the regrid store
    do i = 1, 1
      call ESMF_FieldRegrid(field, dstField(i), &
        routehandle=rh_xgrid2dst(i), &
        rc = localrc)

    enddo

After the regridding calls, the routehandle can be released by calling the ESMF_FieldRegridRelease() method.

    do i = 1, 2
      call ESMF_FieldRegridRelease(routehandle=rh_src2xgrid(i), rc=localrc)

    enddo
    call ESMF_FieldRegridRelease(routehandle=rh_xgrid2dst(1), rc=localrc)

In the above example, we first set up all the required parameters to create an XGrid from user supplied input. Then we create Fields on the XGrid and the Grids on either side. Finally we use the ESMF_FieldRegrid() interface to perform a flux exchange from the source side to the destination side.


34.3.2 Using XGrid in Earth System modeling

A typical application in Earth System Modeling is to calculate flux exchange through the planetary boundary layer that can be represented by ESMF_XGrid. Atmosphere is above the planetary boundary layer while land and ocean are below the boundary layer. To create an XGrid, the land and ocean Grids that are usually different in resolution need to be merged first to create a super Mesh. This merging process is enabled through the support of masking.

The global land and ocean Grids need to be created with masking enabled. In practice, each Grid cell has an integer masking value attached to it. For examples using masking in ESMF_Grid please refer to section 31.3.17.

When calling the ESMF_XGridCreate() method, user can supply the optional arguments sideAMaskValues and sideBMaskValues. These arguments are one dimensional Fortran integer arrays. If any of the sideAMaskValues entry matches the masking value used in sideA Grid, the sideA Grid cell is masked out, vice versa for sideB. Thus by specifying different regions of a land and ocean Grids to be masked out, the two global Grids can be merged into a new global Mesh covering the entire Earth.

The following call shows how to use the ESMF_XGridCreate() method with the optional arguments sideAMaskValues and sideBMaskValues.

    xgrid = ESMF_XGridCreate(sideAGrid=sideA, sideBGrid=sideB, &
      sideAMaskValues=(/2/), sideBMaskValues=(/3,4/), rc=localrc)


34.3.3 Create an XGrid from user input data then use it for regridding

Alternatively, XGrid can be created from Grids on either side, area and centroid information of XGrid cells, sparse matrix matmul information. The functionalities provided by the XGrid object is constrained by the user supplied input during its creation time.

In this example, we will set up a simple XGrid from overlapping Grids on either side of the XGrid. Then we perform a flux exchange from one side to the other side of the XGrid. The Grids are laid out in the following figure:

Figure 20: Grid layout for simple XGrid creation example. Overlapping of 3 Grids (Green 2x2, Red 2x1, Blue 2x2). Green and red Grids on side A, blue Grid on side B, black indicates the resulting XGrid. Color coded sequence indices are shown. Physical coordinates are the tuples in parenthesis, e.g. at the four corners of rectangular computational domain.
\scalebox{0.6}{\includegraphics{XGridEx1}}

We start by creating the Grids on both sides and associate coordinates with the Grids. For details of Grid creation and coordinate use, please refer to Grid class documentation.

    sideA(1) = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/2,2/), &
        coordDep1=(/1/), &
        coordDep2=(/2/), &
        name='source Grid 1 on side A', rc=localrc)

    sideA(2) = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/2,1/), &
        coordDep1=(/1/), &
        coordDep2=(/2/), &
        name='source Grid 2 on side A', rc=localrc)

    do i = 1, 2
        call ESMF_GridAddCoord(sideA(i), staggerloc=ESMF_STAGGERLOC_CENTER, &
            rc=localrc)

    enddo

Coordinate for the Grids on sideA, refer to the Grid layout diagram for the interpretation of the coordinate values:

    ! SideA first grid
    centroidA1X=(/0.5, 1.5/)
    centroidA1Y=(/0.5, 1.5/)
    call ESMF_GridGetCoord(sideA(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CENTER, coordDim=1, &
        farrayPtr=coordX, rc=localrc)

    coordX = centroidA1X
    call ESMF_GridGetCoord(sideA(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CENTER, coordDim=2, &
        farrayPtr=coordY, rc=localrc)

    coordY = centroidA1Y

    ! SideA second grid
    centroidA2X=(/0.5, 1.5/)
    centroidA2Y=(/2.5/)
    call ESMF_GridGetCoord(sideA(2), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CENTER, coordDim=1, &
        farrayPtr=coordX, rc=localrc)

    coordX = centroidA2X
    call ESMF_GridGetCoord(sideA(2), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CENTER, coordDim=2, &
        farrayPtr=coordY, rc=localrc)

    coordY = centroidA2Y

Create the destination grid on side B, only one Grid exists on side B. Also associate coordinate with the Grid:

    sideB(1) = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/2,2/), &
        coordDep1=(/1/), coordDep2=(/2/), &
        name='destination Grid on side B', rc=localrc)

    do i = 1, 1
        call ESMF_GridAddCoord(sideB(i), staggerloc=ESMF_STAGGERLOC_CENTER, &
            rc=localrc)

    enddo

    ! SideB grid
    centroidBX=(/0.75, 1.75/)
    centroidBY=(/0.75, 2.25/)
    call ESMF_GridGetCoord(sideB(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CENTER, coordDim=1, farrayPtr=coordX, &
                rc=localrc)

    coordX = centroidBX
    call ESMF_GridGetCoord(sideB(1), localDE=0, &
        staggerLoc=ESMF_STAGGERLOC_CENTER, coordDim=2, farrayPtr=coordY, &
                rc=localrc)

    coordY = centroidBY

Set up the mapping indices and weights from A side to the XGrid. For details of sequence indices, factorIndexList, and factorList, please see section 28.2.18 in the reference manual. Please refer to the figure above for interpretation of the sequence indices used here.

In order to compute the destination flux on sideB through the XGrid as an mediator, we need to set up the factorList (weights) and factorIndexList (indices) for sparse matrix multiplication in this formulation: dst_flux = W'*W*src_flux, where W' is the weight matrix from the XGrid to destination; and W is the weight matrix from source to the XGrid. The weight matrix is generated using destination area weighted algorithm. Please refer to figure 20 for details.

    ! Set up mapping from A1 -> X
    sparseMatA2X(1)%factorIndexList(1,1)=1    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,2)=2    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,3)=2    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,4)=3    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,5)=4    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,6)=4    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,7)=3    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,8)=4    ! src seq index (green)
    sparseMatA2X(1)%factorIndexList(1,9)=4    ! src seq index (green)

    sparseMatA2X(1)%factorIndexList(2,1)=1    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,2)=2    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,3)=3    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,4)=4    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,5)=5    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,6)=6    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,7)=7    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,8)=8    ! dst seq index (black)
    sparseMatA2X(1)%factorIndexList(2,9)=9    ! dst seq index (black)

    ! Set up mapping from A2 -> X
    sparseMatA2X(2)%factorIndexList(1,1)=1    ! src seq index (red)
    sparseMatA2X(2)%factorIndexList(1,2)=2    ! src seq index (red)
    sparseMatA2X(2)%factorIndexList(1,3)=2    ! src seq index (red)

    sparseMatA2X(2)%factorIndexList(2,1)=10   ! dst seq index (black)
    sparseMatA2X(2)%factorIndexList(2,2)=11   ! dst seq index (black)
    sparseMatA2X(2)%factorIndexList(2,3)=12   ! dst seq index (black)

Set up the mapping weights from side A to the XGrid:

    ! Note that the weights are dest area weighted, they are ratio
    ! of areas with destination area as the denominator.
    ! Set up mapping weights from A1 -> X
    sparseMatA2X(1)%factorList(:)=1.

    ! Set up mapping weights from A2 -> X
    sparseMatA2X(2)%factorList(:)=1.

Set up the mapping indices and weights from the XGrid to B side:

    ! Set up mapping from X -> B
    sparseMatX2B(1)%factorIndexList(1,1)=1    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,2)=2    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,3)=3    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,4)=4    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,5)=5    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,6)=6    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,7)=7    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,8)=8    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,9)=9    ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,10)=10  ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,11)=11  ! src seq index (black)
    sparseMatX2B(1)%factorIndexList(1,12)=12  ! src seq index (black)

    sparseMatX2B(1)%factorIndexList(2,1)=1    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,2)=1    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,3)=2    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,4)=1    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,5)=1    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,6)=2    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,7)=3    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,8)=3    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,9)=4    ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,10)=3   ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,11)=3   ! dst seq index (blue)
    sparseMatX2B(1)%factorIndexList(2,12)=4   ! dst seq index (blue)

    ! Set up mapping weights from X -> B
    sparseMatX2B(1)%factorList(1)=4./9.
    sparseMatX2B(1)%factorList(2)=2./9.
    sparseMatX2B(1)%factorList(3)=2./3.
    sparseMatX2B(1)%factorList(4)=2./9.
    sparseMatX2B(1)%factorList(5)=1./9.
    sparseMatX2B(1)%factorList(6)=1./3.
    sparseMatX2B(1)%factorList(7)=2./9.
    sparseMatX2B(1)%factorList(8)=1./9.
    sparseMatX2B(1)%factorList(9)=1./3.
    sparseMatX2B(1)%factorList(10)=4./9.
    sparseMatX2B(1)%factorList(11)=2./9.
    sparseMatX2B(1)%factorList(12)=2./3.

Optionally the area can be setup to compute surface area weighted flux integrals:

    ! Set up destination areas to adjust weighted flux
    xgrid_area(1) = 1.
    xgrid_area(2) = 0.5
    xgrid_area(3) = 0.5
    xgrid_area(4) = 0.5
    xgrid_area(5) = 0.25
    xgrid_area(6) = 0.25
    xgrid_area(7) = 0.5
    xgrid_area(8) = 0.25
    xgrid_area(9) = 0.25
    xgrid_area(10) = 1.
    xgrid_area(11) = 0.5
    xgrid_area(12) = 0.5

Create an XGrid based on the user supplied regridding parameters:

    xgrid = ESMF_XGridCreateFromSparseMat(sideAGrid=sideA, &
        sideBGrid=sideB, area=xgrid_area, &
        centroid=centroid, sparseMatA2X=sparseMatA2X, &
        sparseMatX2B=sparseMatX2B, rc=localrc)

Create an ESMF_Field on the XGrid:

    field = ESMF_FieldCreate(xgrid, typekind=ESMF_TYPEKIND_R8, &
                rc=localrc)

Query the Field for its Fortran data pointer and its exclusive bounds:

    call ESMF_FieldGet(field, farrayPtr=xfarrayPtr, &
        exclusiveLBound=xlb, exclusiveUBound=xub, rc=localrc)

Setup and initialize src and dst Fields on side A and side B Grids, source Fields have different source flux:

    do i = 1, 2
        srcField(i) = ESMF_FieldCreate(sideA(i), &
                typekind=ESMF_TYPEKIND_R8, rc=localrc)

        call ESMF_FieldGet(srcField(i), farrayPtr=farrayPtr, rc=localrc)

        farrayPtr = i
    enddo
    do i = 1, 1
        dstField(i) = ESMF_FieldCreate(sideB(i), &
                typekind=ESMF_TYPEKIND_R8, rc=localrc)

        call ESMF_FieldGet(dstField(i), farrayPtr=farrayPtr, rc=localrc)

        farrayPtr = 0.0
    enddo

The current implementation requires that Grids used to generate the XGrid must not match, i.e. they are different either topologically or geometrically or both. In this example, the first source Grid is topologically identical to the destination Grid but their geometric coordinates are different. This requirement will be relaxed in a future release.

First we compute the regrid routehandles, these routehandles can be used repeatedly afterwards. Then we initialize the values in the Fields. Finally we execute the Regrid.

    ! Compute regrid routehandles. The routehandles can be used
    ! repeatedly afterwards.
    ! From A -> X
    do i = 1, 2
        call ESMF_FieldRegridStore(xgrid, srcField(i), field, &
                routehandle=rh_src2xgrid(i), rc = localrc)

    enddo
    ! from X -> B
    do i = 1, 1
        call ESMF_FieldRegridStore(xgrid, field, dstField(i), &
                routehandle=rh_xgrid2dst(i), rc = localrc)

    enddo

    ! Initialize values in the source Fields on side A
    do i = 1, 2
        call ESMF_FieldGet(srcField(i), farrayPtr=farrayPtr, rc=localrc)

        farrayPtr = i
    enddo
    ! Initialize values in the destination Field on XGrid
    xfarrayPtr = 0.0
    ! Initialize values in the destination Field on Side B
    do i = 1, 1
        call ESMF_FieldGet(dstField(i), farrayPtr=farrayPtr, rc=localrc)

        farrayPtr = 0.0
    enddo

First we regrid from the Fields on side A to the Field on the XGrid:

    ! Execute regrid from A -> X
    do i = 1, 2
        call ESMF_FieldRegrid(srcField(i), field, &
            routehandle=rh_src2xgrid(i), &
            zeroregion=ESMF_REGION_SELECT, rc = localrc)

    enddo

Next we regrid from the Field on XGrid to the destination Field on side B:

    ! Execute the regrid store
    do i = 1, 1
        call ESMF_FieldRegrid(field, dstField(i), &
            routehandle=rh_xgrid2dst(i), rc = localrc)

    enddo

In the above example, we first set up all the required parameters to create an XGrid from user supplied input. Then we create Fields on the XGrid and the Grids on either side. Finally we use the ESMF_FieldRegrid() interface to perform a flux exchange from the source side to the destination side.


34.3.4 Query the XGrid for its internal information

One can query the XGrid for its internal information:

    call ESMF_XGridGet(xgrid, &
        sideAGridCount=ngridA, &    ! number of Grids on side A
        sideBGridCount=ngridB, &    ! number of Grids on side B
        sideAGrid=l_sideA, &    ! list of Grids on side A
        sideBGrid=l_sideB, &    ! list of Grids on side B
        area=l_area, &      ! list of area of XGrid
        centroid=l_centroid, &  ! list of centroid of XGrid
        distgridA=l_sideAdg, &  ! list of Distgrids on side A
        distgridM = distgrid, & ! balanced distgrid
        sparseMatA2X=l_sparseMatA2X, & !sparse matrix matmul parameters A to X
        sparseMatX2B=l_sparseMatX2B, & !sparse matrix matmul parameters X to B
        rc=localrc)

    call ESMF_XGridGet(xgrid, localDe=0, &
        elementCount=eleCount, &    ! elementCount on the localDE
        exclusiveCount=ec, &        ! exclusive count
        exclusiveLBound=elb, &      ! exclusive lower bound
        exclusiveUBound=eub, &      ! exclusive upper bound
        rc=localrc)

    call ESMF_XGridGet(xgrid, &
        xgridSide=ESMF_XGRIDSIDE_A, & ! side of the XGrid to query
        gridIndex=1, &              ! index of the distgrid
        distgrid=distgrid, &        ! the distgrid returned
        rc=localrc)


34.3.5 Destroying the XGrid and other resources

Clean up the resources by destroying the XGrid and other objects:

    ! After the regridding is successful.
    ! Clean up all the allocated resources:
    call ESMF_FieldDestroy(field, rc=localrc)

    call ESMF_XGridDestroy(xgrid, rc=localrc)

    do i = 1, 2
        call ESMF_FieldDestroy(srcField(i), rc = localrc)

        call ESMF_GridDestroy(sideA(i), rc = localrc)

    enddo

    do i = 1, 1
        call ESMF_FieldDestroy(dstField(i), rc = localrc)

        call ESMF_GridDestroy(sideB(i), rc = localrc)

    enddo

    deallocate(sparseMatA2X(1)%factorIndexList, sparseMatA2X(1)%factorList)
    deallocate(sparseMatA2X(2)%factorIndexList, sparseMatA2X(2)%factorList)
    deallocate(sparseMatX2B(1)%factorIndexList, sparseMatX2B(1)%factorList)

34.4 Restrictions and Future Work

34.4.1 Restrictions and Future Work

  1. CAUTION: Any Grid or Mesh pair picked from the A side and B side of the XGrid cannot point to the same Grid or Mesh in memory on a local PET. This prevents Regrid from selecting the right source and destination grid automatically to calculate the regridding routehandle. It's okay for the Grid and Mesh to have identical topological and geographical properties as long as they are stored in different memory.

34.5 Design and Implementation Notes

  1. The XGrid class is implemented in Fortran, and as such is defined inside the framework by a XGrid derived type and a set of subprograms (functions and subroutines) which operate on that derived type. The XGrid class contains information needed to create Grid, Field, and communication routehandle.

  2. XGrid follows the framework-wide convention of the unison creation and operation rule: All PETs which are part of the currently executing VM must create the same XGrids at the same point in their execution. In addition to the unison rule, XGrid creation also performs inter-PET communication within the current executing VM.

34.6 Class API

34.6.1 ESMF_XGridAssignment(=) - XGrid assignment


INTERFACE:

     interface assignment(=)
     xgrid1 = xgrid2
ARGUMENTS:
     type(ESMF_XGrid) :: xgrid1
     type(ESMF_XGrid) :: xgrid2
DESCRIPTION:

Assign xgrid1 as an alias to the same ESMF XGrid object in memory as xgrid2. If xgrid2 is invalid, then xgrid1 will be equally invalid after the assignment.

The arguments are:

xgrid1
The ESMF_XGrid object on the left hand side of the assignment.
xgrid2
The ESMF_XGrid object on the right hand side of the assignment.

34.6.2 ESMF_XGridOperator(==) - XGrid equality operator


INTERFACE:

     interface operator(==)
     if (xgrid1 == xgrid2) then ... endif
               OR
     result = (xgrid1 == xgrid2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_XGrid), intent(in) :: xgrid1
     type(ESMF_XGrid), intent(in) :: xgrid2
DESCRIPTION:

Test whether xgrid1 and xgrid2 are valid aliases to the same ESMF XGrid object in memory. For a more general comparison of two ESMF XGrids, going beyond the simple alias test, the ESMF_XGridMatch() function (not yet implemented) must be used.

The arguments are:

xgrid1
The ESMF_XGrid object on the left hand side of the equality operation.
xgrid2
The ESMF_XGrid object on the right hand side of the equality operation.

34.6.3 ESMF_XGridOperator(/=) - XGrid not equal operator


INTERFACE:

     interface operator(/=)
     if (xgrid1 /= xgrid2) then ... endif
               OR
     result = (xgrid1 /= xgrid2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_XGrid), intent(in) :: xgrid1
     type(ESMF_XGrid), intent(in) :: xgrid2
DESCRIPTION:

Test whether xgrid1 and xgrid2 are not valid aliases to the same ESMF XGrid object in memory. For a more general comparison of two ESMF XGrids, going beyond the simple alias test, the ESMF_XGridMatch() function (not yet implemented) must be used.

The arguments are:

xgrid1
The ESMF_XGrid object on the left hand side of the non-equality operation.
xgrid2
The ESMF_XGrid object on the right hand side of the non-equality operation.

34.6.4 ESMF_XGridCreate - Create an XGrid from lists of Grids and Meshes


INTERFACE:

 
 function ESMF_XGridCreate(&
     sideAGrid,              sideAMesh, &
     sideBGrid,              sideBMesh, &
     sideAGridPriority,      sideAMeshPriority, &
     sideBGridPriority,      sideBMeshPriority, &
     sideAMaskValues,        sideBMaskValues, &
     storeOverlay, &
     name, rc)
RETURN VALUE:
   type(ESMF_XGrid)                           :: ESMF_XGridCreate
ARGUMENTS:
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
   type(ESMF_Grid),      intent(in), optional :: sideAGrid(:)
   type(ESMF_Mesh),      intent(in), optional :: sideAMesh(:)
   type(ESMF_Grid),      intent(in), optional :: sideBGrid(:)
   type(ESMF_Mesh),      intent(in), optional :: sideBMesh(:)
   integer,              intent(in), optional :: sideAGridPriority(:)
   integer,              intent(in), optional :: sideAMeshPriority(:)
   integer,              intent(in), optional :: sideBGridPriority(:)
   integer,              intent(in), optional :: sideBMeshPriority(:)
   integer(ESMF_KIND_I4),intent(in), optional :: sideAMaskValues(:)
   integer(ESMF_KIND_I4),intent(in), optional :: sideBMaskValues(:)
   logical,              intent(in), optional :: storeOverlay
   character(len=*),     intent(in), optional :: name
   integer,              intent(out),optional :: rc
DESCRIPTION:

Create an XGrid from user supplied input: the list of Grids or Meshes on side A and side B, and other optional arguments. A user can supply both Grids and Meshes on one side to create the XGrid. By default, the Grids have a higher priority over Meshes but the order of priority can be adjusted by the optional GridPriority and MeshPriority arguments. The priority order of Grids and Meshes can also be interleaved by rearranging the optional GridPriority and MeshPriority arguments accordingly.

Sparse matrix multiplication coefficients are internally computed and uniquely determined by the Grids or Meshes provided in sideA and sideB. User can supply a single ESMF_Grid or an array of ESMF_Grid on either side of the ESMF_XGrid. For an array of ESMF_Grid or ESMF_Mesh in sideA or sideB, a merging process concatenates all the ESMF_Grids and ESMF_Meshes into a super mesh represented by ESMF_Mesh. The super mesh is then used to compute the XGrid. Grid or Mesh objects in sideA and sideB arguments must have coordinates defined for the corners of a Grid or Mesh cell. XGrid creation can be potentially memory expensive given the size of the input Grid and Mesh objects. By default, the super mesh is not stored to reduce memory usage. Once communication routehandles are computed using ESMF_FieldRegridStore() method through XGrid, all memory can be released by destroying the XGrid.

If sideA and sideB have a single Grid or Mesh object, it's erroneous if the two Grids or Meshes are spatially disjoint. It is also erroneous to specify a Grid or Mesh object in sideA or sideB that is spatially disjoint from the ESMF_XGrid.

This call is collective across the current VM. For more details please refer to the description 34.1 of the XGrid class. For an example and associated documentation using this method see section 34.3.1

The arguments are:

[sideAGrid]
Parametric 2D Grids on side A, for example, these Grids can be either Cartesian 2D or Spherical.
[sideAMesh]
Parametric 2D Meshes on side A, for example, these Meshes can be either Cartesian 2D or Spherical.
[sideBGrid]
Parametric 2D Grids on side B, for example, these Grids can be either Cartesian 2D or Spherical.
[sideBMesh]
Parametric 2D Meshes on side B, for example, these Meshes can be either Cartesian 2D or Spherical.
[sideAGridPriority]
Priority array of Grids on sideA during overlay generation. The sideAGridPriority array should be the same size as the sideAGrid array. The values in the array should range from 1 to size(sideAGrid)+size(sideAMesh). A Grid whose corresponding value in this array is lower than another side A Grid or Mesh, will take precedence over that Grid or Mesh during side A merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideAMeshPriority]
Priority array of Meshes on sideA during overlay generation. The sideAMeshPriority array should be the same size as the sideAMesh array. The values in the array should range from 1 to size(sideAGrid)+size(sideAMesh). A Mesh whose corresponding value in this array is lower than another side A Grid or Mesh, will take precedence over that Grid or Mesh during side A merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideBGridPriority]
Priority array of Grids on sideB during overlay generation. The sideBGridPriority array should be the same size as the sideBGrid array. The values in the array should range from 1 to size(sideBGrid)+size(sideBMesh). A Grid whose corresponding value in this array is lower than another side B Grid or Mesh, will take precedence over that Grid or Mesh during side B merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideBMeshPriority]
Priority array of Meshes on sideB during overlay generation. The sideBMeshPriority array should be the same size as the sideBMesh array. The values in the array should range from 1 to size(sideBGrid)+size(sideBMesh). A Mesh whose corresponding value in this array is lower than another side B Grid or Mesh, will take precedence over that Grid or Mesh during side B merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideAMaskValues]
Mask information can be set in the Grid (see 31.3.17) or Mesh (see 33.3.11) upon which the Field is built. The sideAMaskValues argument specifies the values in that mask information which indicate a point should be masked out. In other words, a location is masked if and only if the value for that location in the mask information matches one of the values listed in sideAMaskValues. If sideAMaskValues is not specified, no masking on side A will occur.
[sideBMaskValues]
Mask information can be set in the Grid (see 31.3.17) or Mesh (see 33.3.11) upon which the Field is built. The sideBMaskValues argument specifies the values in that mask information which indicate a point should be masked out. In other words, a location is masked if and only if the value for that location in the mask information matches one of the values listed in sideBMaskValues. If sideBMaskValues is not specified, no masking on side B will occur.
[storeOverlay]
Setting the storeOverlay optional argument to .false. (default) allows a user to bypass storage of the ESMF_Mesh used to represent the XGrid. Only a ESMF_DistGrid is stored to allow Field to be built on the XGrid. If the temporary mesh object is of interest, storeOverlay can be set to .true. so a user can retrieve it for future use.
[name]
name of the xgrid object.
[rc]
Return code; equals ESMF_SUCCESS only if the ESMF_XGrid is created.

34.6.5 ESMF_XGridCreateFromSparseMat an XGrid from raw input parameters


INTERFACE:

 
 function ESMF_XGridCreateFromSparseMat(&
     sideAGrid,              sideAMesh, &
     sideBGrid,              sideBMesh, &
     sideAGridPriority,      sideAMeshPriority, &
     sideBGridPriority,      sideBMeshPriority, &
     sparseMatA2X, sparseMatX2A, sparseMatB2X, sparseMatX2B, &
     area, centroid, &
     name, &
     rc)
RETURN VALUE:
     type(ESMF_XGrid) :: ESMF_XGridCreateFromSparseMat
ARGUMENTS:
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
 type(ESMF_Grid),      intent(in), optional :: sideAGrid(:)
 type(ESMF_Mesh),      intent(in), optional :: sideAMesh(:)
 type(ESMF_Grid),      intent(in), optional :: sideBGrid(:)
 type(ESMF_Mesh),      intent(in), optional :: sideBMesh(:)
 integer,              intent(in), optional :: sideAGridPriority(:)
 integer,              intent(in), optional :: sideAMeshPriority(:)
 integer,              intent(in), optional :: sideBGridPriority(:)
 integer,              intent(in), optional :: sideBMeshPriority(:)
 type(ESMF_XGridSpec), intent(in), optional :: sparseMatA2X(:)
 type(ESMF_XGridSpec), intent(in), optional :: sparseMatX2A(:)
 type(ESMF_XGridSpec), intent(in), optional :: sparseMatB2X(:)
 type(ESMF_XGridSpec), intent(in), optional :: sparseMatX2B(:)
 real(ESMF_KIND_R8),   intent(in), optional :: area(:)
 real(ESMF_KIND_R8),   intent(in), optional :: centroid(:,:)
 character (len=*),    intent(in), optional :: name
 integer,              intent(out),optional :: rc
DESCRIPTION:

Create an XGrid directly from user supplied sparse matrix parameters. User is responsible to supply all information necessary for communication calculation. For an example and associated documentation using this method see section 34.3.3

The arguments are:

[sideAGrid]
Parametric 2D Grids on side A, for example, these Grids can be either Cartesian 2D or Spherical.
[sideAMesh]
Parametric 2D Meshes on side A, for example, these Meshes can be either Cartesian 2D or Spherical.
[sideBGrid]
Parametric 2D Grids on side B, for example, these Grids can be either Cartesian 2D or Spherical.
[sideBMesh]
Parametric 2D Meshes on side B, for example, these Meshes can be either Cartesian 2D or Spherical.
[sideAGridPriority]
Priority array of Grids on sideA during overlay generation. The sideAGridPriority array should be the same size as the sideAGrid array. The values in the array should range from 1 to size(sideAGrid)+size(sideAMesh). A Grid whose corresponding value in this array is lower than another side A Grid or Mesh, will take precedence over that Grid or Mesh during side A merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideAMeshPriority]
Priority array of Meshes on sideA during overlay generation. The sideAMeshPriority array should be the same size as the sideAMesh array. The values in the array should range from 1 to size(sideAGrid)+size(sideAMesh). A Mesh whose corresponding value in this array is lower than another side A Grid or Mesh, will take precedence over that Grid or Mesh during side A merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideBGridPriority]
Priority array of Grids on sideB during overlay generation. The sideBGridPriority array should be the same size as the sideBGrid array. The values in the array should range from 1 to size(sideBGrid)+size(sideBMesh). A Grid whose corresponding value in this array is lower than another side B Grid or Mesh, will take precedence over that Grid or Mesh during side B merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sideBMeshPriority]
Priority array of Meshes on sideB during overlay generation. The sideBMeshPriority array should be the same size as the sideBMesh array. The values in the array should range from 1 to size(sideBGrid)+size(sideBMesh). A Mesh whose corresponding value in this array is lower than another side B Grid or Mesh, will take precedence over that Grid or Mesh during side B merging. In other words, if both have parts in the same region, then the object with the lower value will win, and the other Grid or Mesh part will be clipped away.
[sparseMatA2X]
indexlist from a Grid index space on side A to xgrid index space; indexFactorlist from a Grid index space on side A to xgrid index space.
[sparseMatX2A]
indexlist from xgrid index space to a Grid index space on side A; indexFactorlist from xgrid index space to a Grid index space on side A.
[sparseMatB2X]
indexlist from a Grid index space on side B to xgrid index space; indexFactorlist from a Grid index space on side B to xgrid index space.
[sparseMatX2B]
indexlist from xgrid index space to a Grid index space on side B; indexFactorlist from xgrid index space to a Grid index space on side B.
[area]
area of the xgrid cells.
[centroid]
coordinates at the area weighted center of the xgrid cells.
[name]
name of the xgrid object.
[rc]
Return code; equals ESMF_SUCCESS only if the ESMF_XGrid is created.

34.6.6 ESMF_XGridIsCreated - Check whether a XGrid object has been created


INTERFACE:

   function ESMF_XGridIsCreated(xgrid, rc)
RETURN VALUE:
     logical :: ESMF_XGridIsCreated
ARGUMENTS:
     type(ESMF_XGrid), intent(in)            :: xgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,             intent(out), optional :: rc
DESCRIPTION:

Return .true. if the xgrid has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

xgrid
ESMF_XGrid queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

34.6.7 ESMF_XGridDestroy - Release resources associated with an XGrid


INTERFACE:

 
   subroutine ESMF_XGridDestroy(xgrid, noGarbage, rc)
ARGUMENTS:
     type(ESMF_XGrid), intent(inout)          :: xgrid       
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,          intent(in),   optional :: noGarbage
     integer,          intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Destroys an ESMF_XGrid, releasing the resources associated with the object.

The arguments are:

xgrid
ESMF_XGrid object.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

34.6.8 ESMF_XGridGet - Get object-wide information from an XGrid


INTERFACE:

   ! Private name; call using ESMF_XGridGet()
 
 subroutine ESMF_XGridGetDefault(xgrid, &
     sideAGridCount, sideBGridCount, sideAMeshCount, sideBMeshCount, &
     coordSys, &
     dimCount, elementCount, &
     sideAGrid, sideBGrid, sideAMesh, sideBMesh, &
     mesh, &
     area, centroid, &
     distgridA, distgridB, distgridM, &
     sparseMatA2X, sparseMatX2A, sparseMatB2X, sparseMatX2B, &
     name, &
     rc)
ARGUMENTS:
 type(ESMF_XGrid),     intent(in)            :: xgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
 integer,              intent(out), optional :: sideAGridCount, sideBGridCount
 integer,              intent(out), optional :: sideAMeshCount, sideBMeshCount
 type(ESMF_CoordSys_Flag), intent(out), optional :: coordSys
 integer,              intent(out), optional :: dimCount
 integer,              intent(out), optional :: elementCount
 type(ESMF_Grid),      intent(out), optional :: sideAGrid(:), sideBGrid(:)
 type(ESMF_Mesh),      intent(out), optional :: sideAMesh(:), sideBMesh(:)
 type(ESMF_Mesh),      intent(out), optional :: mesh
 real(ESMF_KIND_R8),   intent(out), optional :: area(:)
 real(ESMF_KIND_R8),   intent(out), optional :: centroid(:,:)
 type(ESMF_DistGrid),  intent(out), optional :: distgridA(:)
 type(ESMF_DistGrid),  intent(out), optional :: distgridB(:)
 type(ESMF_DistGrid),  intent(out), optional :: distgridM
 type(ESMF_XGridSpec), intent(out), optional :: sparseMatA2X(:)
 type(ESMF_XGridSpec), intent(out), optional :: sparseMatX2A(:)
 type(ESMF_XGridSpec), intent(out), optional :: sparseMatB2X(:)
 type(ESMF_XGridSpec), intent(out), optional :: sparseMatX2B(:)
 character (len=*),    intent(out), optional :: name
 integer,              intent(out), optional :: rc
DESCRIPTION:

Get information about XGrid

The arguments are:

xgrid
The ESMF_XGrid object used to retrieve information from.
[sideAGridCount]
Total Number of Grids on the A side.
[sideBGridCount]
Total Number of Grids on the B side.
[sideAMeshCount]
Total Number of Meshes on the A side.
[sideBMeshCount]
Total Number of Meshes on the B side.
[coordSys]
The coordinate system of the XGrid's coordinate data.
[dimCount]
Number of dimension of the xgrid.
[elementCount]
Number of elements in exclusive region of the xgrid on this PET.
[sideAGrid]
List of 2D Grids on side A. Must enter with shape(sideAGrid)=(/sideAGridCount/).
[sideBGrid]
List of 2D Grids on side B. Must enter with shape(sideBGrid)=(/sideBGridCount/).
[sideAMesh]
List of 2D Meshes on side A. Must enter with shape(sideAMesh)=(/sideAMeshCount/).
[sideBMesh]
List of 2D Meshes on side B. Must enter with shape(sideBMesh)=(/sideBMeshCount/).
[mesh]
Super mesh stored in XGrid when storeOverlay is set true during XGrid creation.
[area]
Area of the xgrid cells on this PET. Must enter with shape(area)=(/elementCount/).
[centroid]
Coordinates at the area weighted center of the xgrid cells on this PET. Must enter with shape(centroid)=(/elementCount, dimCount/).
[distgridA]
List of distgrids whose sequence index list is an overlap between a Grid on sideA and the xgrid object. Must enter with shape(distgridA)=(/sideAGridCount+sideAMeshCount/).
[distgridB]
List of distgrids whose sequence index list is an overlap between a Grid on sideB and the xgrid object. Must enter with shape(distgridB)=(/sideBGridCount+sideBMeshCount/).
[distgridM]
The distgrid whose sequence index list fully describes the xgrid object.
[sparseMatA2X]
Indexlist from a Grid index space on side A to xgrid index space; indexFactorlist from a Grid index space on side A to xgrid index space. Must enter with shape(sparsematA2X)=(/sideAGridCount+sideAMeshCount/).
[sparseMatX2A]
Indexlist from xgrid index space to a Grid index space on side A; indexFactorlist from xgrid index space to a Grid index space on side A. Must enter with shape(sparsematX2A)=(/sideAGridCount+sideAMeshCount/).
[sparseMatB2X]
Indexlist from a Grid index space on side B to xgrid index space; indexFactorlist from a Grid index space on side B to xgrid index space. Must enter with shape(sparsematB2X)=(/sideBGridCount+sideBMeshCount/).
[sparseMatX2B]
Indexlist from xgrid index space to a Grid index space on side B; indexFactorlist from xgrid index space to a Grid index space on side B. Must enter with shape(sparsematX2B)=(/sideBGridCount+sideBMeshCount/).
[name]
Name of the xgrid object.
[rc]
Return code; equals ESMF_SUCCESS only if the ESMF_XGrid is created.

35 Geom Class

35.1 Description

The ESMF Geom class is used as a container for other ESMF geometry objects (e.g. an ESMF Grid). This allows a generic representation of a geometry to be passed around (e.g. through a coupled system) without it's specific type being known. Some operations are supported on a Geom object and more will be added over time as needed. However, if an unsupported operation is needed, then the specific geometry object can always be pulled out and operated on that way.

In addition to the geometry object, a Geom can also contain information describing a location on a geometry. For example, in the case of a Grid, a geometry object will also contain a stagger location. Having this location information allows the creation of Fields and other capabilities to be performed in the most generic way on a Geom object. For geometries where it is appropriate, the user can optionally specify this location information during the creation of a Geom object. However, if no location is specified, then default values for this information are provided which match those which would be used when creating a Field with the specific geometry (e.g. stagger location ESMF_STAGGERLOC_CENTER for a Grid).

35.2 Class API: Geom Methods

35.2.1 ESMF_GeomCreate - Create a Geom from a Grid


INTERFACE:

   ! Private name; call using ESMF_GeomCreate()
       function ESMF_GeomCreateGrid(grid,staggerloc, rc)
RETURN VALUE:
       type(ESMF_Geom) :: ESMF_GeomCreateGrid
ARGUMENTS:
        type(ESMF_Grid),       intent(in)             :: grid
        type(ESMF_StaggerLoc), intent(in), optional   :: staggerloc
        integer,               intent(out), optional  :: rc
DESCRIPTION:

Create an ESMF_Geom object from an ESMF_Grid object.

The arguments are:

grid
ESMF_Grid object from which to create the Geom.
[staggerloc]
Stagger location of data in grid cells. For valid predefined values see section 31.2.6. If not specified, defaults to ESMF_STAGGERLOC_CENTER.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

35.2.2 ESMF_GeomCreate - Create a Geom from a Mesh


INTERFACE:

   ! Private name; call using ESMF_GeomCreate()
       function ESMF_GeomCreateMesh(mesh, meshLoc, rc)
RETURN VALUE:
       type(ESMF_Geom) :: ESMF_GeomCreateMesh
ARGUMENTS:
        type(ESMF_Mesh),       intent(in)              :: mesh
        type(ESMF_MeshLoc),    intent(in),  optional   :: meshLoc
        integer,               intent(out), optional   :: rc
DESCRIPTION:

Create an ESMF_Geom object from an ESMF_Mesh object.

The arguments are:

mesh
ESMF_Mesh object from which to create the Geom.
[meshloc]
The part of the Mesh on which to build the Field. For valid predefined values see Section 54.41. If not set, defaults to ESMF_MESHLOC_NODE.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

35.2.3 ESMF_GeomCreate - Create a Geom from a LocStream


INTERFACE:

   ! Private name; call using ESMF_GeomCreate()
       function ESMF_GeomCreateLocStream(locstream, rc)
RETURN VALUE:
       type(ESMF_Geom) :: ESMF_GeomCreateLocStream
ARGUMENTS:
        type(ESMF_LocStream),  intent(in)              :: locstream
        integer,               intent(out),  optional  :: rc
DESCRIPTION:

Create an ESMF_Geom object from an ESMF_LocStream object.

The arguments are:

locstream
ESMF_LocStream object from which to create the Geom.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

35.2.4 ESMF_GeomCreate - Create a Geom from an XGrid


INTERFACE:

   ! Private name; call using ESMF_GeomCreate()
       function ESMF_GeomCreateXGrid(xgrid, xgridside, gridIndex, rc)
RETURN VALUE:
       type(ESMF_Geom) :: ESMF_GeomCreateXGrid
ARGUMENTS:
        type(ESMF_XGrid),     intent(in)                :: xgrid
        type(ESMF_XGridSide_Flag), intent(in), optional :: xgridSide
        integer,              intent(in), optional      :: gridIndex
        integer,              intent(out),optional      :: rc
DESCRIPTION:

Create an ESMF_Geom object from an ESMF_XGrid object.

The arguments are:

xgrid
ESMF_XGrid object from which to create the Geom.
[xgridSide]
Which side of the XGrid to create the Field on (either ESMF_XGRIDSIDE_A, ESMF_XGRIDSIDE_B, or ESMF_XGRIDSIDE_BALANCED). If not specified, then defaults to ESMF_XGRIDSIDE_BALANCED.
[gridindex]
If xgridSide is ESMF_XGRIDSIDE_A or ESMF_XGRIDSIDE_B then this index tells which Grid or Mesh on that side is being referred to. If not provided, defaults to 1.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

35.2.5 ESMF_GeomDestroy - Release resources associated with a Geom


INTERFACE:

       subroutine ESMF_GeomDestroy(geom, rc)
ARGUMENTS:
       type(ESMF_Geom)            :: geom
       integer, intent(out), optional :: rc
DESCRIPTION:

Destroys an ESMF_Geom object. This call does not destroy wrapped Grid, LocStream, or other objects.

The arguments are:

geom
ESMF_Geom to be destroyed.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

35.2.6 ESMF_GeomGet - Get information about a Geom


INTERFACE:

       subroutine ESMF_GeomGet(geom, &
           dimCount, rank, localDECount, distgrid, &
           distgridToGridMap, indexFlag, geomtype, &
           grid, staggerloc, mesh, meshloc, locstream, &
           xgrid, xgridside, gridIndex,rc)
ARGUMENTS:
       type(ESMF_Geom),   intent(in)            :: geom
       integer,               intent(out), optional :: dimCount
       integer,               intent(out), optional :: rank
       integer,               intent(out), optional :: localDECount
       type(ESMF_DistGrid),   intent(out), optional :: distgrid
       integer,               intent(out), optional :: distgridToGridMap(:)
       type(ESMF_Index_Flag),  intent(out), optional :: indexflag
       type(ESMF_GeomType_Flag),   intent(out), optional :: geomtype
       type(ESMF_Grid),       intent(out), optional :: grid
       type(ESMF_StaggerLoc), intent(out), optional :: staggerloc
       type(ESMF_Mesh),       intent(out), optional :: mesh
       type(ESMF_MeshLoc),    intent(out), optional :: meshloc
       type(ESMF_LocStream),  intent(out), optional :: locstream
       type(ESMF_XGrid),      intent(out), optional :: xgrid
       type(ESMF_XGridSide_Flag),  intent(out), optional :: xgridside
       integer,               intent(out), optional :: gridIndex
       integer,               intent(out), optional :: rc
DESCRIPTION:

Get various types of information about a Geom.

The arguments are:

geom
Geom to get the information from.
[dimCount]
The full number of dimensions of the Distgrid object underneath the Geom object.
[rank]
The count of the memory dimensions in this Geom object. Typically it's the same as dimCount. However, in some cases (e.g. arbitrarily distributed grids) it can be different.
[localDECount]
The number of DEs in this Geom object on this PET.
[distgrid]
The structure describing the distribution of the Geom object.
[distgridToGridMap]
List that has as many elements as the distgrid dimCount. This array describes mapping between the Geom object's dimensions and its Distgrid.
[indexflag]
Flag that indicates how the DE-local indices are to be defined.
[geomtype]
The type of geometry on which the Field is built. See section 54.23 for the range of values.
[grid]
If the Geom object holds a Grid, then this will pass out that Grid object.
[staggerloc]
If the Geom object holds a Grid, then this will pass out the staggerloc.
[mesh]
If the Geom object holds a Mesh, then this will pass out that Mesh object.
[meshloc]
If the Geom object holds a Mesh, then this will pass out the meshloc.
[locstream]
If the Geom object holds a LocStream, then this will pass out that LocStream object.
[xgrid]
If the Geom object holds an XGrid, then this will pass out that XGrid object.
[xgridSide]
If the Geom object holds an XGrid, then this will pass out the XGrid side.
[gridIndex]
If the Geom object holds an XGrid, then this will pass out the gridIndex.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36 DistGrid Class

36.1 Description

The ESMF DistGrid class sits on top of the DELayout class and holds domain information in index space. A DistGrid object captures the index space topology and describes its decomposition in terms of DEs. Combined with DELayout and VM the DistGrid defines the data distribution of a domain decomposition across the computational resources of an ESMF Component.

The global domain is defined as the union of logically rectangular (LR) sub-domains or tiles. The DistGrid create methods allow the specification of such a multi-tile global domain and its decomposition into exclusive, DE-local LR regions according to various degrees of user specified constraints. Complex index space topologies can be constructed by specifying connection relationships between tiles during creation.

The DistGrid class holds domain information for all DEs. Each DE is associated with a local LR region. No overlap of the regions is allowed. The DistGrid offers query methods that allow DE-local topology information to be extracted, e.g. for the construction of halos by higher classes.

A DistGrid object only contains decomposable dimensions. The minimum rank for a DistGrid object is 1. A maximum rank does not exist for DistGrid objects, however, ranks greater than 7 may lead to difficulties with respect to the Fortran API of higher classes based on DistGrid. The rank of a DELayout object contained within a DistGrid object must be equal to the DistGrid rank. Higher class objects that use the DistGrid, such as an Array object, may be of different rank than the associated DistGrid object. The higher class object will hold the mapping information between its dimensions and the DistGrid dimensions.

36.2 Constants


36.2.1 ESMF_DISTGRIDMATCH

DESCRIPTION:
Indicates the level to which two DistGrid variables match.

The type of this flag is:

type(ESMF_DistGridMatch_Flag)

The valid values are:

ESMF_DISTGRIDMATCH_INVALID:
Indicates a non-valid matching level. One or both DistGrid objects are invalid.
ESMF_DISTGRIDMATCH_NONE:
The lowest valid level of DistGrid matching. This indicates that the DistGrid objects don't match at any of the higher levels.
ESMF_DISTGRIDMATCH_INDEXSPACE:
The index space covered by the two DistGrid objects is identical. However, differences between the two objects prevents a higher matching level.
ESMF_DISTGRIDMATCH_TOPOLOGY:
The topology (i.e. index space and connections) defined by the two DistGrid objects is identical. However, differences between the two objects prevents a higher matching level.
ESMF_DISTGRIDMATCH_DECOMP:
The index space decomposition defined by the two DistGrid objects is identical. However, differences between the two objects prevents a higher matching level.
ESMF_DISTGRIDMATCH_EXACT:
The two DistGrid objects match in all aspects, including sequence indices. The only aspect that may differ between the two objects is their name.
ESMF_DISTGRIDMATCH_ALIAS:
Both DistGrid variables are aliases to the exact same DistGrid object in memory.

36.3 Use and Examples

The following examples demonstrate how to create, use and destroy DistGrid objects. In order to produce complete and valid DistGrid objects all of the ESMF_DistGridCreate() calls require to be called in unison i.e. on all PETs of a component with a complete set of valid arguments.

36.3.1 Single tile DistGrid with regular decomposition

The minimum information required to create an ESMF_DistGrid object for a single tile with default decomposition are the min and max of the tile in index space. The following call creates a DistGrid for a 1D index space tile with elements from 1 through 1000.

  distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/1000/), rc=rc)

A default DELayout with 1 DE per PET will be created during the ESMF_DistGridCreate() call. The 1000 elements of the specified 1D tile are then block decomposed into the available DEs, and distributed across the PETs (same number as DEs by default). Assuming execution on 4 PETs, the (min) $\sim$ (max) indices of the DE-local blocks will be:

     DE 0 - (1) ~ (250)
     DE 1 - (251) ~ (500)
     DE 2 - (501) ~ (750)
     DE 3 - (751) ~ (1000)

DistGrids with rank > 1 can also be created with default decompositions, specifying only the min and max indices of the tile. The following creates a 2D DistGrid for a 5x5 tile with default decomposition.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), rc=rc)

The default decomposition for a DistGrid of rank $N$ will be $(nDEs \times 1
\times ... \times 1) $, where $nDEs$ is the number of DEs in the DELayout and there are $N-1$ factors of $1$. For the 2D example above this means a $4 \times 1$ regular decomposition if executed on 4 PETs and will result in the following DE-local LR regions:

     DE 0 - (1,1) ~ (2,5)
     DE 1 - (3,1) ~ (3,5)
     DE 2 - (4,1) ~ (4,5)
     DE 3 - (5,1) ~ (5,5)

In many cases the default decomposition will not suffice for higher rank DistGrids (rank > 1). For this reason a decomposition descriptor regDecomp argument is available during ESMF_DistGridCreate(). The following call creates a DistGrid on the same 2D tile as before, but now with a user specified regular decomposition of $2 \times 3 = 6 $ DEs.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), rc=rc)

The default DE labeling sequence follows column major order for the regDecomp argument:

     -----------> 2nd dimension
     |  0  2  4
     |  1  3  5
     v
    1st dimension

By default grid points along all dimensions are homogeneously divided between the DEs. The maximum element count difference between DEs along any dimension is 1. The (min) $\sim$ (max) indices of the DE-local blocks of the above example are as follows:

     DE 0 - (1,1) ~ (3,2)
     DE 1 - (4,1) ~ (5,2)
     DE 2 - (1,3) ~ (3,4)
     DE 3 - (4,3) ~ (5,4)
     DE 4 - (1,5) ~ (3,5)
     DE 5 - (4,5) ~ (5,5)

The specifics of the tile decomposition into DE-local LR domains can be modified by the optional decompflag argument. The following line shows how this argument is used to keep ESMF's default decomposition in the first dimension but move extra grid points of the second dimension to the last DEs in that direction. Extra elements occur if the number of DEs for a certain dimension does not evenly divide its extent. In this example there are 2 extra grid points for the second dimension because its extent is 5 but there are 3 DEs along this index space axis.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), decompflag=(/ESMF_DECOMP_BALANCED, &
    ESMF_DECOMP_RESTLAST/), rc=rc)

Now DE 4 and DE 5 will hold the extra elements along the 2nd dimension.

     DE 0 - (1,1) ~ (3,1)
     DE 1 - (4,1) ~ (5,1)
     DE 2 - (1,2) ~ (3,2)
     DE 3 - (4,2) ~ (5,2)
     DE 4 - (1,3) ~ (3,5)
     DE 5 - (4,3) ~ (5,5)

An alternative way of indicating the DE-local LR regions is to list the index space coordinate as given by the associated DistGrid tile for each dimension. For this 2D example there are two lists (dim 1) / (dim 2) for each DE:

     DE 0 - (1,2,3) / (1)
     DE 1 - (4,5)   / (1)
     DE 2 - (1,2,3) / (2)
     DE 3 - (4,5)   / (2)
     DE 4 - (1,2,3) / (3,4,5)
     DE 5 - (4,5)   / (3,4,5)

Information about DE-local LR regions in the latter format can be obtained from the DistGrid object by use of ESMF_DistGridGet() methods:

  allocate(dimExtent(2, 0:5)) ! (dimCount, deCount)
  call ESMF_DistGridGet(distgrid, delayout=delayout, &
    indexCountPDe=dimExtent, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  call ESMF_DELayoutGet(delayout, localDeCount=localDeCount, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  allocate(localDeToDeMap(0:localDeCount-1))
  call ESMF_DELayoutGet(delayout, localDeToDeMap=localDeToDeMap, rc=rc)
  if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
  do localDe=0, localDeCount-1
    de = localDeToDeMap(localDe)
    do dim=1, 2
      allocate(localIndexList(dimExtent(dim, de))) ! allocate list 
                                                   ! to hold indices
      call ESMF_DistGridGet(distgrid, localDe=localDe, dim=dim, &
        indexList=localIndexList, rc=rc)
      if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
      print *, "local DE ", localDe," - DE ",de, &
        " localIndexList along dim=", dim," :: ", localIndexList
      deallocate(localIndexList)
    enddo
  enddo
  deallocate(localDeToDeMap)
  deallocate(dimExtent)

The advantage of the localIndexList format over the minIndex/maxIndex format is that it can be used directly for DE-local to tile index dereferencing. Furthermore the localIndexList allows to express very general decompositions such as the cyclic decompositions in the first dimension generated by the following call:

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), &
    decompflag=(/ESMF_DECOMP_CYCLIC,ESMF_DECOMP_RESTLAST/), rc=rc)

with decomposition:

     DE 0 - (1,3,5) / (1)
     DE 1 - (2,4)   / (1)
     DE 2 - (1,3,5) / (2)
     DE 3 - (2,4)   / (2)
     DE 4 - (1,3,5) / (3,4,5)
     DE 5 - (2,4)   / (3,4,5)

Finally, a DistGrid object is destroyed by calling

  call ESMF_DistGridDestroy(distgrid, rc=rc)

36.3.2 DistGrid and DELayout

The examples of this section use the 2D DistGrid of the previous section to show the interplay between DistGrid and DELayout. By default, i.e. without specifying the delayout argument, a DELayout will be created during DistGrid creation that provides as many DEs as the DistGrid object requires. The implicit call to ESMF_DELayoutCreate() is issued with a fixed number of DEs and default settings in all other aspects. The resulting DE to PET mapping depends on the number of PETs of the current VM context. Assuming 6 PETs in the VM

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), rc=rc)

will result in the following domain decomposition in terms of DEs

     0  2  4
     1  3  5
and their layout or distribution over the available PETs:
     DE 0  -> PET 0
     DE 1  -> PET 1
     DE 2  -> PET 2
     DE 3  -> PET 3
     DE 4  -> PET 4
     DE 5  -> PET 5

Running the same example on a 4 PET VM will not change the domain decomposition into 6 DEs as specified by

     0  2  4
     1  3  5
but the layout across PETs will now contain multiple DE-to-PET mapping with default cyclic distribution:
     DE 0  -> PET 0
     DE 1  -> PET 1
     DE 2  -> PET 2
     DE 3  -> PET 3
     DE 4  -> PET 0
     DE 5  -> PET 1

Sometimes it may be desirable for performance tuning to construct a DELayout with specific characteristics. For instance, if the 6 PETs of the above example are running on 3 nodes of a dual-SMP node cluster and there is a higher communication load along the first dimension of the model than along the second dimension it would be sensible to place DEs according to this knowledge.

The following example first creates a DELayout with 6 DEs where groups of 2 DEs are to be in fast connection. This DELayout is then used to create a DistGrid.

  delayout = ESMF_DELayoutCreate(deCount=6, deGrouping=(/(i/2,i=0,5)/), rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), delayout=delayout, rc=rc)

This will ensure a distribution of DEs across the cluster resource in the following way:

     0   2   4
     1   3   5
    SMP SMP SMP

The interplay between DistGrid and DELayout may at first seem complicated. The simple but important rule to understand is that DistGrid describes a domain decomposition and each domain is labeled with a DE number. The DELayout describes how these DEs are laid out over the compute resources of the VM, i.e. PETs. The DEs are purely logical elements of decomposition and may be relabeled to fit the algorithm or legacy code better. The following example demonstrates this by describing the exact same distribution of the domain data across the fictitious cluster of SMP-nodes with a different choice of DE labeling:

  delayout = ESMF_DELayoutCreate(deCount=6, deGrouping=(/(mod(i,3),i=0,5)/), &
    rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), deLabelList=(/0,3,1,4,2,5/), delayout=delayout, rc=rc)

Here the deLabelList argument changes the default DE label sequence from column major to row major. The DELayout compensates for this change in DE labeling by changing the deGrouping argument to map the first dimension to SMP nodes as before. The decomposition and layout now looks as follows:

     0   1   2
     3   4   5
    SMP SMP SMP

Finally, in order to achieve a completely user-defined distribution of the domain data across the PETs of the VM a DELayout may be created from a petMap before using it in the creation of a DistGrid. If for instance the desired distribution of a 2 x 3 decomposition puts the DEs of the first row onto 3 separate PETs (PET 0, 1, 2) and groups the DEs of the second row onto PET 3 a petMap must first be setup that takes the DE labeling of the DistGrid into account.The following lines of code result in the desired distribution using column major DE labeling by first create a DELayout and then using it in the DistGrid creation.

  delayout = ESMF_DELayoutCreate(petMap=(/0,3,1,3,2,3/), rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    regDecomp=(/2,3/), delayout=delayout, rc=rc)

This decomposes the global domain into

     0   2   4
     1   3   5
and associates the DEs to the following PETs:
     DE 0  -> PET 0
     DE 1  -> PET 3
     DE 2  -> PET 1
     DE 3  -> PET 3
     DE 4  -> PET 2
     DE 5  -> PET 3

36.3.3 Single tile DistGrid with decomposition by DE blocks

In the previous examples the DistGrid objects were created with regular decompositions. In some cases a regular decomposition may not be the most natural choice to decompose and distribute the index space. The DE block version of ESMF_DistGridCreate() offers more control over the precise decomposition. The following example shows how the deBlockList argument is used to determine exactly what index space block ends up on each DE.

A single 5x5 tile is decomposed into 6 DEs. To this end a list is constructed that holds the min and max indices of all six DE blocks. The DE blocks must be constructed to cover the index space without overlapping each other. It is okay to leave holes in the index space, i.e. the DE blocks do not completely cover the index space tile.

  allocate(deBlockList(2, 2, 6))  ! (dimCount, 2, deCount)
  deBlockList(:,1,1) = (/1,1/)  ! minIndex  1st deBlock
  deBlockList(:,2,1) = (/3,2/)  ! maxIndex  1st deBlock
  deBlockList(:,1,2) = (/4,1/)  ! minIndex  2nd deBlock
  deBlockList(:,2,2) = (/5,2/)  ! maxIndex  2nd deBlock
  deBlockList(:,1,3) = (/1,3/)
  deBlockList(:,2,3) = (/2,4/)
  deBlockList(:,1,4) = (/3,3/)
  deBlockList(:,2,4) = (/5,4/)
  deBlockList(:,1,5) = (/1,5/)
  deBlockList(:,2,5) = (/3,5/)
  deBlockList(:,1,6) = (/4,5/)  ! minIndex  6th deBlock
  deBlockList(:,2,6) = (/5,5/)  ! maxInbex  6th deBlock

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
    deBlockList=deBlockList, rc=rc)

36.3.4 2D multi-tile DistGrid with regular decomposition

Creating a DistGrid from a list of LR tiles is a straightforward extension of the single tile case. The first four arguments of ESMF_DistGridCreate() are promoted to rank 2 where the second dimension is the tile index.

The following 2D multi-tile domain consisting of 3 LR tiles will be used in the examples of this section:

  
     ----------------------------------------> 2nd dim
     |
     |                   (1,11)-----(1,20)
     |                   |               | 
     |                   |               | 
     |                   |               | 
     |                   |               | 
     |                   |               | 
     |                   (10,11)---(10,20)
     |  (11,1)----(11,10)(11,11)---(11,20)
     |  |               ||               |
     |  |               ||               |
     |  |               ||               |
     |  |               ||               |
     |  |               ||               |
     |  (20,1)----(20,10)(20,11)---(20,20)
     |
     |
     v
    1st dim

The first step in creating a multi-tile global domain is to construct the minIndex and maxIndex arrays.

  allocate(minIndexPTile(2,3))    ! (dimCount, tileCount)
  allocate(maxIndexPTile(2,3))    ! (dimCount, tileCount)
  minIndexPTile(:,1) = (/11,1/)
  maxIndexPTile(:,1) = (/20,10/)
  minIndexPTile(:,2) = (/11,11/)
  maxIndexPTile(:,2) = (/20,20/)
  minIndexPTile(:,3) = (/1,11/)
  maxIndexPTile(:,3) = (/10,20/)

Next the regular decomposition for each tile is set up in the regDecomp array. In this example each tile is associated with a single DE.

  allocate(regDecompPTile(2,3))    ! (dimCount, tileCount)
  regDecompPTile(:,1) = (/1,1/)    ! one DE
  regDecompPTile(:,2) = (/1,1/)    ! one DE
  regDecompPTile(:,3) = (/1,1/)    ! one DE

Finally the DistGrid can be created by calling

  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, regDecompPTile=regDecompPTile, rc=rc)

The default DE labeling sequence is identical to the tile labeling sequence and follows the sequence in which the tiles are defined during the create call. However, DE labels start at 0 whereas tile labels start at 1. In this case the DE labels look as:

           2
       0   1

Each tile can be decomposed differently into DEs. The default DE labeling follows the column major order for each tile. This is demonstrated in the following case where the multi-tile global domain is decomposed into 9 DEs,

  regDecompPTile(:,1) = (/2,2/)    ! 4 DEs
  regDecompPTile(:,2) = (/1,3/)    ! 3 DEs
  regDecompPTile(:,3) = (/2,1/)    ! 2 DEs
  
  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, regDecompPTile=regDecompPTile, rc=rc)

resulting in the following decomposition:

             +-------+
             |   7   |
             |       |
             |   8   |
     +-------+-------+
     | 0   2 |       |
     |       | 4 5 6 |
     | 1   3 |       |
     +-------+-------+

     DE 0 - (11,1)  ~ (15,5)
     DE 1 - (16,1)  ~ (20,5)
     DE 2 - (11,6)  ~ (15,10)
     DE 3 - (16,6)  ~ (20,10)
     DE 4 - (11,11) ~ (20,14)
     DE 5 - (11,15) ~ (20,17)
     DE 6 - (11,18) ~ (20,20)
     DE 7 - (1,11)  ~ (5,20)
     DE 8 - (6,11)  ~ (10,20)

The decompflag and deLabelList arguments can be used much like in the single LR domain case to overwrite the default grid decomposition (per tile) and to change the overall DE labeling sequence, respectively.


36.3.5 Arbitrary DistGrids with user-supplied sequence indices

The third, and most flexible way of creating an ESMF DistGrid object is by specifying the arbitrary sequence indices of all the index space elements associated with a particular DE. The concept of sequence index comes into the DistGrid class through the support it implements for the communication methods of higher classes: Arrays and Fields. This support is based by associating a unique sequence index with each DistGrid index tuple. The sequence index can be used to address every Array or Field element. By default, the DistGrid does not actually generate and store the sequence index of each element. Instead a default sequence through the elements is implemented in the DistGrid code. This default sequence is used internally when needed.

The DistGrid class provides two ESMF_DistGridCreate() calls that allow the user to specify arbitrary sequence indices, overriding the use of the default sequence index scheme. The user sequence indices are passed to the DistGrid in form of 1d Fortran arrays, one array on each PET. The local size of this array on each PET determines the number of DistGrid elements on the PET. The supplied sequence indices must be unique across all PETs.

  allocate(arbSeqIndexList(10))   ! each PET will have 10 elements
  
  do i=1, 10
    arbSeqIndexList(i) = (i-1)*petCount + localPet ! initialize unique 
                                                   ! seq. indices
  enddo

A default DELayout will be created automatically during ESMF_DistGridCreate(), associating 1 DE per PET.

  distgrid = ESMF_DistGridCreate(arbSeqIndexList=arbSeqIndexList, rc=rc)

The user provided sequence index array can be deallocated once it has been used.

  deallocate(arbSeqIndexList)

The distgrid object can be used just like any other DistGrid object. The "arbitrary" nature of distgrid will only become visible during Array or Field communication methods, where source and destination objects map elements according to the sequence indices provided by the associated DistGrid objects.

  call ESMF_DistGridDestroy(distgrid, rc=rc)

The second ESMF_DistGridCreate() call, that accepts the arbSeqIndexList argument, allows the user to specify additional, regular DistGrid dimensions. These additional DistGrid dimensions are not decomposed across DEs, but instead are simply "added" or "multiplied" to the 1D arbitrary dimension.

The same arbSeqIndexList array as before is used to define the user supplied sequence indices.

  allocate(arbSeqIndexList(10))   ! each PET will have 10 elements
  
  do i=1, 10
    arbSeqIndexList(i) = (i-1)*petCount + localPet  ! initialize unique 
                                                    ! seq. indices
  enddo

The additional DistGrid dimensions are specified in the usual manner using minIndex and maxIndex arguments. The dimCount of the resulting DistGrid is the size of the minIndex and maxIndex arguments plus 1 for the arbitrary dimension. The arbDim argument is used to indicate which or the resulting DistGrid dimensions is associated with the arbitrary sequence indices provided by the user.

  distgrid = ESMF_DistGridCreate(arbSeqIndexList=arbSeqIndexList, &
    arbDim=1, minIndexPTile=(/1,1/), maxIndexPTile=(/5,7/), rc=rc)

  deallocate(arbSeqIndexList)

  call ESMF_DistGridDestroy(distgrid, rc=rc)

36.3.6 DistGrid Connections - Definition

By default all of the edges of the index space tiles making up a DistGrid are open. There is no sense of connectedness between the tiles. This situation is shown for a simple 2 tile DistGrid.

  allocate(minIndexPTile(2,2))    ! (dimCount, tileCount)
  allocate(maxIndexPTile(2,2))    ! (dimCount, tileCount)
  minIndexPTile(:,1) = (/1,1/)
  maxIndexPTile(:,1) = (/10,10/)
  minIndexPTile(:,2) = (/11,1/)
  maxIndexPTile(:,2) = (/20,10/)
  
  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, rc=rc)

Figure 21: Two 10x10 index space tiles next to each other without connections. Both tiles operate in the same global index space chosen by ESMF_INDEX_GLOBAL when creating the DistGrid object. The index tuples held by the DistGrid are represented by the vertices of the shown grid structure. The index tuple (11,3), which is referenced in the text, is marked by a black circle.
\includegraphics{dgconnect_2tiles_not_connected.eps}

Connections between index space tiles are specified during DistGrid creation through the connectionList argument. This argument takes a list of elements of type(ESMF_DistGridConnection). Each element refers to one specific connection between any two tiles.

Each connection is defined by 4 parameters:

The underlying principle of the DistGrid connections is that all supported connections can be written as a forward transformation of the form

\begin{displaymath}
\vec a \rightarrow \vec b = \hat R \vec a + \vec P.
\end{displaymath} (4)

This transform takes the index space tuple $\vec a$ of a point in the reference frame of tile "A" and expresses it as tuple $\vec b$ in terms of the index space defined by tile "B". Here $\hat R$ is a general rotation operator, and $\vec P$ is a translation vector in index space. $\hat R$ and $\vec P$ correspond to the orientationVector and positionVector, respectively.

As an example consider the index space point marked by the black circle in figure 21. In the reference frame of tile 1 the point has an index tuple of (11,3). Because of the global index space (ESMF_INDEX_GLOBAL), the point has the same index tuple of (11,3) in the reference frame of tile 2. Therefore, the connection that connects the right edge of tile 1 with the left edge of tile 2 has $\hat R ={1\!\!1}$ (default orientation) and $\vec P = (0,0)$. Therefore the connection can be set by the following code. The resulting situation is shown in figure 22.

  allocate(connectionList(1))
  call ESMF_DistGridConnectionSet(connection=connectionList(1), &
    tileIndexA=1, tileIndexB=2, positionVector=(/0,0/), rc=rc)

  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, connectionList=connectionList, &
    rc=rc)  ! defaults to ESMF_INDEX_GLOBAL

The same topology can be defined for ESMF_INDEX_DELOCAL indexing. However, the positionVector must be adjusted for the fact that now the same point in index space has different index tuples depending on what tile's reference frame is used.

With local indexing both tiles start at (1,1) and end at (10,10).

  allocate(minIndexPTile(2,2))    ! (dimCount, tileCount)
  allocate(maxIndexPTile(2,2))    ! (dimCount, tileCount)
  minIndexPTile(:,1) = (/1,1/)
  maxIndexPTile(:,1) = (/10,10/)
  minIndexPTile(:,2) = (/1,1/)
  maxIndexPTile(:,2) = (/10,10/)

To see the impact that the index scheme has on the positionVector, again consider the same highlighted index space point. The index tuple for this point is still (11,3) in the reference frame of tile 1 (tile "A" of the connection). However, in the reference frame of of tile 2 (tile "B" of the connection)) it has changed to (1,3) due to local indexing. Therefore, using form (4), we find that the position vector must be $\vec P = \vec b - \vec a = (1,3) - (11,3) = (-10,0)$.

  allocate(connectionList(1))
  call ESMF_DistGridConnectionSet(connection=connectionList(1), &
    tileIndexA=1, tileIndexB=2, positionVector=(/-10,0/), rc=rc)

  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, connectionList=connectionList, &
    indexflag=ESMF_INDEX_DELOCAL, rc=rc)

Figure 22: Two 10x10 index space tiles next to each other with a single connection between the right edge of tile 1 and the left edge of tile 2. The index tuple (11,3), which is referenced in the text, is marked by a black circle.
\includegraphics{dgconnect_2tiles_connected.eps}

Further note that every forward transformation has an associated inverse, or backward transformation from tile "B" into the reference frame of tile "A". Inverting the forward transform yields the backward transform as

\begin{displaymath}
\vec b \rightarrow \vec a = \hat R^{-1} \vec b - \hat R^{-1} \vec P.
\end{displaymath} (5)

The DistGrid implicitly considers the corresponding backward connection for every forward connection that is specified explicitly. In other words, DistGrid connections are bidirectional.

Before going into the details of how the orientationVector and positionVector arguments correspond to $\hat R$ and $\vec P$ for more complex cases, it is useful to explore what class of connections are covered by the above introduced form (4) of $\vec a \rightarrow \vec b$.

First consider the case where tile "A" is rotated by $\hat R$ relative to tile "B" around a general pivot point $\vec p$ given in terms of the index space of tile "A":


$\displaystyle \vec a \rightarrow \vec b$ $\textstyle =$ $\displaystyle \hat R (\vec a - \vec p) + \vec p$  
  $\textstyle =$ $\displaystyle \hat R \vec a + ({1\!\!1} - \hat R) \vec p$ (6)

With substitution

\begin{displaymath}
\vec P = ({1\!\!1} - \hat R) \vec p
\end{displaymath} (7)

form (4) is recovered.

Next consider transform (6) followed by a translation $\vec t$ of tile "B" relative to tile "A":


\begin{displaymath}
\vec a \rightarrow \vec b =
\hat R \vec a + ({1\!\!1} - \hat R) \vec p + \vec t.
\end{displaymath} (8)

Again form (4) is recovered with the appropriate subsitution:

\begin{displaymath}
\vec P = ({1\!\!1} - \hat R) \vec p + \vec t.
\end{displaymath} (9)

Equation (9) is the general definition of the positionVector argument for DistGrid connections. It allows two tiles to be connected according to the relationship expressed by (8). Note that this formualation of tile connections is more general than connecting an edge of a tile to the edge of another tile. Instead a DistGrid connection is specified as a general relationship between the two index spaces, accounting for possible rotation and translation. This formuation supports situations where some elements of the connected tiles overlap with each other in index space. The ESMF DistGrid class leverages this feature when representing topologies that lead to redundancies of elements. Examples for this are the bipolar cut line in a tripole grid, or the edges of a cubed sphere.

By definition, DistGrid connections associate an index tuple of one tile with exactly one index tuple expressed in the reference frame of another tile. This restricts the supported rotations $\hat R$ to multiples of $90^{\circ}$. Also allowing invesion of index space dimensions leads to 8 unique operations in two dimension shown in table 3.


Table 3: The 8 unique rotational operations in 2 dimensional index space. The associated orientationVector argument for each operation is also shown.
   $\hat R$ orientationVector  
 $0^{\circ}$ $\left( \begin{array}{rr}
1 & 0 \\
0 & 1 \end{array} \right)$ $\left( \begin{array}{r}
1 \\
2 \end{array} \right)$  
 $90^{\circ}$ $\left( \begin{array}{rr}
0 & -1 \\
1 & 0 \end{array} \right)$ $\left( \begin{array}{r}
-2 \\
1 \end{array} \right)$  
 $180^{\circ}$ $\left( \begin{array}{rr}
-1 & 0 \\
0 & -1 \end{array} \right)$ $\left( \begin{array}{r}
-1 \\
-2 \end{array} \right)$  
 $270^{\circ}$ $\left( \begin{array}{rr}
0 & 1 \\
-1 & 0 \end{array} \right)$ $\left( \begin{array}{r}
2 \\
-1 \end{array} \right)$  
 $0^{\circ}$ + inversion dim 1 $\left( \begin{array}{rr}
-1 & 0 \\
0 & 1 \end{array} \right)$ $\left( \begin{array}{r}
-1 \\
2 \end{array} \right)$  
 $0^{\circ}$ + inversion dim 2 $\left( \begin{array}{rr}
1 & 0 \\
0 & -1 \end{array} \right)$ $\left( \begin{array}{r}
1 \\
-2 \end{array} \right)$  
 $90^{\circ}$ + inversion dim 1 $\left( \begin{array}{rr}
0 & 1 \\
1 & 0 \end{array} \right)$ $\left( \begin{array}{r}
2 \\
1 \end{array} \right)$  
 $90^{\circ}$ + inversion dim 2 $\left( \begin{array}{rr}
0 & -1 \\
-1 & 0 \end{array} \right)$ $\left( \begin{array}{r}
-2 \\
-1 \end{array} \right)$  

The orientationVector is simply a more compact format holding the same information provided by the 8 rotational matrices. The first (or top) element of the orientation vector indicates which dimension of the tile "A" index tuple is used for the first dimension of the tile "B" tuple. The second (or bottom) element of the orientation vector indicates which dimension of the tile "A" index tuple is used for the second dimenson of the tile "B" tuple. If an orientation vector entry is negative, the sign of the associated tuple element is inverted when going from tile "A" to tile "B" reference frame. Table 3 provides the corresponding orientationVector argument for each of the 8 2D rotational operations.

36.3.7 DistGrid Connections - Single tile periodic and pole connections

The concept of DistGrid connections is not limited to cases with multiple tiles. Even a single tile DistGrid can have connections. In this instance tileA and tileB simply reference the same tile. A very common case is that of a single tile with periodic boundary conditions.

First consider a single tile DistGrid without connections.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/50,20/), rc=rc)

In order to better visualize the topology, the first index space dimension is associated with the longitude ( $0^{\circ}..360^{\circ}$), and the second dimension with latitude ( $-80^{\circ}..+80^{\circ}$) of the unit sphere (using an ESMF_Grid object) as shown in figure 23.

Figure 23: A single 50x20 index space tile without connections. For better visualization the index space points are plotted on the unit circle. The gap between the right and left edge of the tile is visible. Further the top and the bottom edges of the tile are visibly without connection.
\includegraphics{dgconnect_1tile_not_connected.eps}

A single DistGrid connection is needed to connect the right edge of the index space tile with its left edge. Connecting a tile with itself in such manner leads to a periodic topology.

First the connectionList needs to be allocated for a single connection. Then the connection is defined with both tileIndexA and tileIndexB set to 1, referring to the first, and only tile in this case.

  allocate(connectionList(1))
  call ESMF_DistGridConnectionSet(connection=connectionList(1), &
    tileIndexA=1, tileIndexB=1, positionVector=(/-50,0/), rc=rc)

The positionVector is determined by transformation (4), the fact that there is no rotation involved, and that stepping over the right edge needs to connect back to the left edge. Therefore $\vec P = \vec b - \vec a = (1,j) - (51,j) =
(-50,0)$. Here $j$ stands for an arbitrary value along the second index space dimension.

Creating a DistGrid on the same index space tile, but with this connection, results in a periodic boundary condition along the first dimension. This is shown in figure 24.

  distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/50,20/), &
    connectionList=connectionList, rc=rc)

Figure 24: A single 50x20 index space tile with periodic connection along the first dimension.
\includegraphics{dgconnect_1tile_periodic1_connected.eps}

In general it is more useful to express the position vector of a connection in terms of the tile minIndex and maxIndex components. For this we define the same index space tile in a set of variables.

  allocate(minIndex(2))    ! (dimCount)
  allocate(maxIndex(2))    ! (dimCount)
  minIndex(:) = (/1,1/)
  maxIndex(:) = (/50,20/)

Now we can code any connection on this tile in terms of minIndex and maxIndex. For purpose of demonstration we define periodic boundary conditions along both index space dimensions. The resulting torus topology is depicted in figure 25.

  allocate(connectionList(2))
  call ESMF_DistGridConnectionSet(connection=connectionList(1), & ! 1st connection
    tileIndexA=1, tileIndexB=1, &   ! periodic along i
    positionVector=(/ -(maxIndex(1)-minIndex(1)+1) , 0/), &  
    rc=rc)

  call ESMF_DistGridConnectionSet(connection=connectionList(2), & ! 2nd connection
    tileIndexA=1, tileIndexB=1, &   ! periodic along j
    positionVector=(/ 0 , -(maxIndex(2)-minIndex(2)+1) /), &  
    rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=minIndex, maxIndex=maxIndex, &
    connectionList=connectionList, rc=rc)

Figure 25: A single 50x20 index space tile with periodic connections along both directions. The topology is that of a torus, however, because of the chosen spherical coordinates the connection through the middle has the shape of a cylinder.
\includegraphics{dgconnect_1tile_periodic2_connected.eps}

While the topology shown in figure 25 is that of a torus, the coordinates chosen are actually those of a sphere. Next we replace the periodic connection along $j$ (i.e. the second index space dimension) with a more fitting pole connection at the top of the sphere (i.e. at $j_{max}$).

For the orientation vector associated with a regular pole connection at $j_{max}$ we first look at how the two index space directions are affected. Looking at a point with $i$ along the first dimension, and a second point $i+1$ that is just to the right of the first point, we see that as the pole is being crossed, the second point maps just right of the first point. Therefore, the orientation of the first index space dimension is unaffected by the pole connection. However, for the second dimension we find that increasing $j$ on one side corresponds to a dereasing $j$ across the pole. We thus have found the general fact that orientationVector=(1,-2) for a pole connection across the $j$ direction.

In order to find the position vector of the polar connection we consider starting at a general point ($i$,$j_{max}$) at the top edge of the tile. Crossing the pole this takes us to a point that is again right on the top edge with $j=j_{max}$, and is $180^\circ$ rotated along the first dimension. This means $i=mod(i+i_{size}/2, i_{size})$, with $i_{size}=i_{max}
-i_{min}+1$. In practice the modulo operation is automatically taken care of by the periodic connection along $i$. We can therefore write:


\begin{displaymath}
\vec a = \left( \begin{array}{l}
i \\
j_{max}+1 \end{arr...
...in{array}{l}
i + i_{size}/2\\
j_{max} \end{array} \right).
\end{displaymath} (10)

Using this observation, together with table 3 to translate the polar orientationVector into a standard rotation operation $\hat R$, we get the position vector from equation (4):


$\displaystyle \vec P$ $\textstyle =$ $\displaystyle \vec b - \hat R \vec a$  
  $\textstyle =$ $\displaystyle \left( \begin{array}{l}
i + i_{size}/2\\
j_{max} \end{array} \ri...
...\end{array} \right)
\left( \begin{array}{l}
i \\
j_{max}+1 \end{array} \right)$  
  $\textstyle =$ $\displaystyle \left( \begin{array}{l}
i_{size}/2\\
2j_{max} +1 \end{array} \right).$ (11)

  allocate(connectionList(2))
  call ESMF_DistGridConnectionSet(connection=connectionList(1), & ! 1st connection
    tileIndexA=1, tileIndexB=1, &   ! periodic along i
    positionVector=(/-(maxIndex(1)-minIndex(1)+1),0/), & 
    rc=rc)

  call ESMF_DistGridConnectionSet(connection=connectionList(2), & ! 2nd connection
    tileIndexA=1, tileIndexB=1, &   ! pole at j_max
    orientationVector=(/1,-2/), &
    positionVector=(/ (maxIndex(1)-minIndex(1)+1)/2 , 2*maxIndex(2)+1 /), & 
    rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=minIndex, maxIndex=maxIndex, &
    connectionList=connectionList, rc=rc)

The pole connection at $j_{max}$ can clearly be seen in figure 26. Note that the chosen perspective hides the fact that the lower edge of the index space tile remains open. In other words there is still a hole at the bottom of the sphere that cannot be seen. Only three of the four sides have been connected so far: The first connection connects the left and the right tile edges. The second connection connects the top edge to itself to form the pole. A third connection would be needed, e.g. to form a pole at the bottom edge much like the top edge. This would then complete a perfectly spherical topology with a single tile.

Figure 26: A single 50x20 index space tile with periodic connection along $i$, and pole at $j_{max}$. The hole at $j_{min}$ is hidden from sight.
\includegraphics{dgconnect_1tile_peripole_connected.eps}

The final single tile topology discussed in this section is that of a tripole. A tripolar sphere has the typical spherical periodic boundary condition along one direction (e.g. connecting the left and the right tile edge), and a regular monopole at one of the other edges of the tile. However, instead of defining a second monopole at the opposite edge, a bipole connection is chosen.

Topologically a bipole connection can be thought of folding the respective edge at the middle point back onto itself. Assuming the bipole at the top edge, i.e. at $j_{max}$, we get mappings across the bipole of $(i_{min}, j_{max}+1) \rightarrow (i_{max}, j_{max})$, $(i_{min}+1, j_{max}+1) \rightarrow (i_{max}-1, j_{max})$, and so forth. This means that compared to the regular pole connection, the bipolar orientation vector reverses the $i$ direction in addition to the $j$ direction: orientationVector=(-1,-2).

Using the bipolar mapping just mentioned for a point at $i_{min}$, together with table 3 to translate the polar orientationVector into a standard rotation operation $\hat R$, we can solve for the position vector according to equation (4):


$\displaystyle \vec P$ $\textstyle =$ $\displaystyle \vec b - \hat R \vec a$  
  $\textstyle =$ $\displaystyle \left( \begin{array}{l}
i_{max}\\
j_{max} \end{array} \right)
- ...
...rray} \right)
\left( \begin{array}{l}
i_{min} \\
j_{max}+1 \end{array} \right)$  
  $\textstyle =$ $\displaystyle \left( \begin{array}{l}
i_{max}+i_{min}\\
2j_{max} +1 \end{array} \right).$ (12)

Figure 27 visualizes the bipolar topology at the top edge of the tile. Note, however, that the coordinates are perfectly spherical. Consequently there is no "drawing shut" of the cut line as would be expected for a true bipolar geometry. Still, the two poles are becoming visible at the two opposing ends of the top circle, where the distance between the connection lines is starting to go to zero.

Figure 27: A single 50x20 index space tile with periodic connection along $i$, and bi-pole at $j_{max}$. The regular pole connection at $j_{min}$ is hidden from sight.
\includegraphics{dgconnect_1tile_peribipole_connected.eps}

  allocate(connectionList(3))
  call ESMF_DistGridConnectionSet(connection=connectionList(1), & ! 1st connection
    tileIndexA=1, tileIndexB=1, &   ! periodic along i
    positionVector=(/-(maxIndex(1)-minIndex(1)+1),0/), & 
    rc=rc)

  call ESMF_DistGridConnectionSet(connection=connectionList(2), & ! 2nd connection
    tileIndexA=1, tileIndexB=1, &   ! pole at j_min
    orientationVector=(/1,-2/), &
    positionVector=(/ (maxIndex(1)-minIndex(1)+1)/2 , 2*minIndex(2)+1 /), & 
    rc=rc)

  call ESMF_DistGridConnectionSet(connection=connectionList(3), & ! 3rd connection
    tileIndexA=1, tileIndexB=1, &   ! bi-pole at j_max
    orientationVector=(/-1,-2/), &
    positionVector=(/ maxIndex(1)+minIndex(1) , 2*maxIndex(2)+1 /), & 
    rc=rc)

  distgrid = ESMF_DistGridCreate(minIndex=minIndex, maxIndex=maxIndex, &
    connectionList=connectionList, rc=rc)

36.3.8 DistGrid Connections - Multi tile connections

Starting point of the multi-tile connection examples will be the six tile case shown in figure 28. All six tiles are identical squares of size 10x10.

Figure 28: Six 10x10 square index space tiles without connections. The tile number is indicated by color as indicated by the legend.
\includegraphics{dgconnect_cusph_not_connected.eps}

One geometrical interpretation of the six tiles shown is that of an unfolded cube. In fact, the way that the tiles are arranged in the 2D plane does suggest the cubic interpretation. In order to turn the six tiles into a cubic topology, each tile must be connected to its neighbors on all four sides. In total there will be 12 connections that need to be made.

Choosing global indexing, the depicted six tile case can be created in the following way:

  allocate(minIndexPTile(2,6))    ! (dimCount, tileCount)
  allocate(maxIndexPTile(2,6))    ! (dimCount, tileCount)
  size = 10                       ! number of index space points along tile sides
  !- tile 1
  tile=1
  minIndexPTile(1,tile)=1
  minIndexPTile(2,tile)=1
  maxIndexPTile(1,tile)=minIndexPTile(1,tile)+size-1
  maxIndexPTile(2,tile)=minIndexPTile(2,tile)+size-1
  !- tile 2
  tile=2
  minIndexPTile(1,tile)=maxIndexPTile(1,tile-1)+1
  minIndexPTile(2,tile)=minIndexPTile(2,tile-1)
  maxIndexPTile(1,tile)=minIndexPTile(1,tile)+size-1
  maxIndexPTile(2,tile)=minIndexPTile(2,tile)+size-1
  !- tile 3
  tile=3
  minIndexPTile(1,tile)=minIndexPTile(1,tile-1)
  minIndexPTile(2,tile)=maxIndexPTile(2,tile-1)+1
  maxIndexPTile(1,tile)=minIndexPTile(1,tile)+size-1
  maxIndexPTile(2,tile)=minIndexPTile(2,tile)+size-1
  !- tile 4
  tile=4
  minIndexPTile(1,tile)=maxIndexPTile(1,tile-1)+1
  minIndexPTile(2,tile)=minIndexPTile(2,tile-1)
  maxIndexPTile(1,tile)=minIndexPTile(1,tile)+size-1
  maxIndexPTile(2,tile)=minIndexPTile(2,tile)+size-1
  !- tile 5
  tile=5
  minIndexPTile(1,tile)=minIndexPTile(1,tile-1)
  minIndexPTile(2,tile)=maxIndexPTile(2,tile-1)+1
  maxIndexPTile(1,tile)=minIndexPTile(1,tile)+size-1
  maxIndexPTile(2,tile)=minIndexPTile(2,tile)+size-1
  !- tile 6
  tile=6
  minIndexPTile(1,tile)=maxIndexPTile(1,tile-1)+1
  minIndexPTile(2,tile)=minIndexPTile(2,tile-1)
  maxIndexPTile(1,tile)=minIndexPTile(1,tile)+size-1
  maxIndexPTile(2,tile)=minIndexPTile(2,tile)+size-1
  
  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, rc=rc)

The five connections between tiles 1&2, 2&3, 3&4, 4&5, 5&6 are trivial. There are no rotations, which means that the orientationVector argument can be ommitted in these connections. Further, because of the global index space, there are no translations either, which means that positionVector=(0,0) for these five connections. The resulting topology is shown in figure 29.

  allocate(connectionList(5))
  !- connection 1
  conn=1
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=1, tileIndexB=2, positionVector=(/0, 0/), rc=rc)
  !- connection 2
  conn=2
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=2, tileIndexB=3, positionVector=(/0, 0/), rc=rc)
  !- connection 3
  conn=3
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=3, tileIndexB=4, positionVector=(/0, 0/), rc=rc)
  !- connection 4
  conn=4
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=4, tileIndexB=5, positionVector=(/0, 0/), rc=rc)
  !- connection 5
  conn=5
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=5, tileIndexB=6, positionVector=(/0, 0/), rc=rc)

  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, connectionList=connectionList, rc=rc)

Figure 29: The six tiles of an unfolded cube with five connections defined.
\includegraphics{dgconnect_cusph_5connected.eps}

The sixth connection that does not involve a rotation is that between tile 1&6. While there is no rotation involved, it does include a translation because the bottom edge of tile 1 must reach all the way to the top edge of tile 6. This involves a translation along both the $i$ and the $j$ dimension.

Using the same procedure introduced in the previous section, we chose an arbitrary index space point close to the connection and write it in terms of both tiles that we want to connect. E.g. the first point of the top edge of tile 6 is

( minIndexPTile(1,6) , maxIndexPTile(2,6) )

in terms of tile 6. However, in terms of tile 1, going through the connection, it is

( minIndexPTile(1,1) , minIndexPTile(2,1)-1 ).

According to the general transformation relationship (4) the position vector $\vec P$ for the forward transform tile 1 $\rightarrow$ tile 6 is then given as the difference between these two representations. Figure 30 visualizes the situation.

  !- connection 6
  conn=6
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=1, tileIndexB=6, &
    positionVector=(/minIndexPTile(1,6)-minIndexPTile(1,1),     &
                     maxIndexPTile(2,6)-minIndexPTile(2,1)+1/), &
    rc=rc)
  
  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, connectionList=connectionList, rc=rc)

Figure 30: The six tiles of an unfolded cube with all six connections that do not involve any rotation of tiles.
\includegraphics{dgconnect_cusph_6connected.eps}

The six remaining connections all involve rotations. The procedure for finding the correct orientationVector and positionVector arguments still remains the same: First determine the direction of the connection to be formulated. This is important because for the forward connection the rotation applies to tile "A". Once the correct rotation operation $\hat R$ is pinned down, an arbitrary point close to the connection is chosen. This point can either be on tile "A" or "B". It is written then written in terms of tile "A" index space $\vec a$, and in terms of tile "B" index space $\vec b$. Obviously one of those formulations (either $\vec a$ or $\vec b$) will take advantage of the connection, i.e. it will actually step outside the reference tile in order to reach the chosen point. Finally the position vector $\vec P$ of the connection is determined by expression (4) as the difference:


\begin{displaymath}
\vec P = \vec b - \hat R \vec a.
\end{displaymath} (13)

Following the above outlined procedure for connection tile 1 $\rightarrow$ tile 3, we find first that tile 1 needs to be rotated clockwise by $90^\circ$. This rotation lines up the top edge of tile 1 with the left edge of tile 3. A clockwise rotation of $90^\circ$ corresponds to a counterclockwise rotation by $270^\circ$ given in table 3. We therefore know that orientationVector=(2,-1) for this connection, and the associated operation is $\hat R=\left(\begin{array}{rr}
0 & 1 \\
-1 & 0 \end{array} \right)$.

Next we chose the first point on the top edge of tile 1 as a reference point. In terms of tile 1 this point has coordinates

$\vec a$ = ( minIndexPTile(1,1) , maxIndexPTile(2,1) ).

The same point in terms of tile 3 (going through the connection) has coordinates

$\vec b$ = ( minIndexPTile(1,3)-1 , maxIndexPTile(2,3) ).

Using equation (13) we find the position vector and can write down the connection:

  allocate(connectionList(2))
  !- connection 1
  conn=1
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=1, tileIndexB=3, &
    orientationVector=(/2,-1/), & ! 270 degree rotation of tile A
    positionVector=(/minIndexPTile(1,3)-1-maxIndexPTile(2,1),   &
                     maxIndexPTile(2,3)+minIndexPTile(1,1)/), &
    rc=rc)

For greater clarity figure 31 only shows two connections. Besides the connection just defined between tile 1 and 3, the other connection shown is between tile 4 and 6. Defining the connection as forward going from tile 4 to tile 6 means that tile 4 needs to be rotated in such a way that its right edge meets up with the bottom edge of tile 6. This requires a counterclockwise rotation of tile 4 by $90^\circ$. From table 3 we then get orientationVector=(-2,1), and $\hat R=\left(\begin{array}{rr}
0 & -1 \\
1 & 0 \end{array} \right)$.

Choosing the left most point on the bottom edge of tile 6 as the reference point, we find the coordinates in terms of tile 4 (through the connection)

$\vec a$ = ( maxIndexPTile(1,4)+1 , maxIndexPTile(2,4) ),

and in terms of tile 6

$\vec b$ = ( minIndexPTile(1,6) , minIndexPTile(2,6) ).

Again using equation (13) we find the position vector and can implement the second connection:

  !- connection 2
  conn=2
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=4, tileIndexB=6, &
    orientationVector=(/-2,1/), & ! 90 degree rotation of tile A
    positionVector=(/minIndexPTile(1,6)+maxIndexPTile(2,4),   &
                     minIndexPTile(2,6)-maxIndexPTile(1,4)-1/), &
    rc=rc)

  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, connectionList=connectionList, rc=rc)

Figure 31: The six tiles of an unfolded cube with two connections that involve rotation of tiles.
\includegraphics{dgconnect_cusph_2rotconnected.eps}

The remaining four connections with rotations can be determined following the exact same recipe. The following code finally defines all 12 connections needed to connect the six index space tiles into a cubic topology.

  allocate(connectionList(12))

  !- connection 1: tile 1 -> tile 2
  conn=1
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=1, tileIndexB=2, positionVector=(/0, 0/), rc=rc)

  !- connection 2: tile 2 -> tile 3
  conn=2
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=2, tileIndexB=3, positionVector=(/0, 0/), rc=rc)

  !- connection 3: tile 3 -> tile 4
  conn=3
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=3, tileIndexB=4, positionVector=(/0, 0/), rc=rc)

  !- connection 4: tile 4 -> tile 5
  conn=4
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=4, tileIndexB=5, positionVector=(/0, 0/), rc=rc)

  !- connection 5: tile 5 -> tile 6
  conn=5
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=5, tileIndexB=6, positionVector=(/0, 0/), rc=rc)

  !- connection 6: tile 1 -> tile 6
  conn=6
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=1, tileIndexB=6, &
    positionVector=(/minIndexPTile(1,6)-minIndexPTile(1,1),     &
                     maxIndexPTile(2,6)-minIndexPTile(2,1)+1/), &
    rc=rc)

  !- connection 7: tile 1 -> tile 3
  conn=7
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=1, tileIndexB=3, &
    orientationVector=(/2,-1/), & ! 270 degree rotation of tile A
    positionVector=(/minIndexPTile(1,3)-1-maxIndexPTile(2,1), &
                     maxIndexPTile(2,3)+minIndexPTile(1,1)/), &
    rc=rc)

  !- connection 8: tile 3 -> tile 5
  conn=8
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=3, tileIndexB=5, &
    orientationVector=(/2,-1/), & ! 270 degree rotation of tile A
    positionVector=(/minIndexPTile(1,5)-1-maxIndexPTile(2,3), &
                     maxIndexPTile(2,5)+minIndexPTile(1,3)/), &
    rc=rc)

  !- connection 9: tile 5 -> tile 1
  conn=9
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=5, tileIndexB=1, &
    orientationVector=(/2,-1/), & ! 270 degree rotation of tile A
    positionVector=(/minIndexPTile(1,1)-1-maxIndexPTile(2,5), &
                     maxIndexPTile(2,1)+minIndexPTile(1,5)/), &
    rc=rc)

  !- connection 10: tile 2 -> tile 4
  conn=10
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=2, tileIndexB=4, &
    orientationVector=(/-2,1/), & ! 90 degree rotation of tile A
    positionVector=(/minIndexPTile(1,4)+maxIndexPTile(2,2),     &
                     minIndexPTile(2,4)-maxIndexPTile(1,2)-1/), &
    rc=rc)

  !- connection 11: tile 4 -> tile 6
  conn=11
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=4, tileIndexB=6, &
    orientationVector=(/-2,1/), & ! 90 degree rotation of tile A
    positionVector=(/minIndexPTile(1,6)+maxIndexPTile(2,4),     &
                     minIndexPTile(2,6)-maxIndexPTile(1,4)-1/), &
    rc=rc)

  !- connection 12: tile 6 -> tile 2
  conn=12
  call ESMF_DistGridConnectionSet(connection=connectionList(conn), &
    tileIndexA=6, tileIndexB=2, &
    orientationVector=(/-2,1/), & ! 90 degree rotation of tile A
    positionVector=(/minIndexPTile(1,2)+maxIndexPTile(2,6),     &
                     minIndexPTile(2,2)-maxIndexPTile(1,6)-1/), &
    rc=rc)
  
  ! - create the DistGrid with 6 tiles and 12 connections
  distgrid = ESMF_DistGridCreate(minIndexPTile=minIndexPTile, &
    maxIndexPTile=maxIndexPTile, connectionList=connectionList, rc=rc)

For better visualization the resulting cubic topology is plotted in 3D. Each index space point is associated with a longitude and latitude value of the unit sphere. Combined with the cubic topology formed by the six index space tiles, this results in a cubed sphere representation shown in figure 32.

Figure 32: Six index space tiles with all 12 connections to form a cubic topology. The coordinates at every index space point are chosen to form a spherical geometry, resulting in a cubed sphere.
\includegraphics{dgconnect_cusph_12connected.eps}

36.4 Restrictions and Future Work

36.5 Design and Implementation Notes

This section will be updated as the implementation of the DistGrid class nears completion.

36.6 Class API

36.6.1 ESMF_DistGridAssignment(=) - DistGrid assignment


INTERFACE:

     interface assignment(=)
     distgrid1 = distgrid2
ARGUMENTS:
     type(ESMF_DistGrid) :: distgrid1
     type(ESMF_DistGrid) :: distgrid2
STATUS:

DESCRIPTION:

Assign distgrid1 as an alias to the same ESMF DistGrid object in memory as distgrid2. If distgrid2 is invalid, then distgrid1 will be equally invalid after the assignment.

The arguments are:

distgrid1
The ESMF_DistGrid object on the left hand side of the assignment.
distgrid2
The ESMF_DistGrid object on the right hand side of the assignment.

36.6.2 ESMF_DistGridOperator(==) - DistGrid equality operator


INTERFACE:

   interface operator(==)
     if (distgrid1 == distgrid2) then ... endif
               OR
     result = (distgrid1 == distgrid2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_DistGrid), intent(in) :: distgrid1
     type(ESMF_DistGrid), intent(in) :: distgrid2
STATUS:

DESCRIPTION:

Test whether distgrid1 and distgrid2 are valid aliases to the same ESMF DistGrid object in memory. For a more general comparison of two ESMF DistGrids, going beyond the simple alias test, the ESMF_DistGridMatch() function (not yet fully implemented) must be used.

The arguments are:

distgrid1
The ESMF_DistGrid object on the left hand side of the equality operation.
distgrid2
The ESMF_DistGrid object on the right hand side of the equality operation.

36.6.3 ESMF_DistGridOperator(/=) - DistGrid not equal operator


INTERFACE:

   interface operator(/=)
     if (distgrid1 /= distgrid2) then ... endif
               OR
     result = (distgrid1 /= distgrid2)
RETURN VALUE:
     logical :: result
ARGUMENTS:
     type(ESMF_DistGrid), intent(in) :: distgrid1
     type(ESMF_DistGrid), intent(in) :: distgrid2
STATUS:

DESCRIPTION:

Test whether distgrid1 and distgrid2 are not valid aliases to the same ESMF DistGrid object in memory. For a more general comparison of two ESMF DistGrids, going beyond the simple alias test, the ESMF_DistGridMatch() function (not yet fully implemented) must be used.

The arguments are:

distgrid1
The ESMF_DistGrid object on the left hand side of the non-equality operation.
distgrid2
The ESMF_DistGrid object on the right hand side of the non-equality operation.

36.6.4 ESMF_DistGridCreate - Create DistGrid object from DistGrid


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDG(distgrid, &
     firstExtra, lastExtra, indexflag, connectionList, balanceflag, &
     delayout, vm, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDG
ARGUMENTS:
     type(ESMF_DistGrid),           intent(in)            :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, target,               intent(in),  optional :: firstExtra(:)
     integer, target,               intent(in),  optional :: lastExtra(:)
     type(ESMF_Index_Flag),         intent(in),  optional :: indexflag
     type(ESMF_DistGridConnection), intent(in),  optional :: connectionList(:)
     logical,                       intent(in),  optional :: balanceflag
     type(ESMF_DELayout),           intent(in),  optional :: delayout
     type(ESMF_VM),                 intent(in),  optional :: vm
     integer,                       intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create a new DistGrid from an existing DistGrid, keeping the decomposition unchanged, unless balanceflag=.true. (see below). The firstExtraPTile and lastExtraPTile arguments allow extra elements to be added at the first/last edge DE in each dimension. The method also allows the indexflag to be set. Further, if the connectionList argument provided in it will be used to set connections in the newly created DistGrid, otherwise the connections of the incoming DistGrid will be used.

The balanceflag argument allows a change in the decomposition, and thus of the number of DEs. An attempt is made to decompose the index space into as many DEs as there are PETs in the VM for which the DistGrid is created. See the argument description for details.

Setting the balanceflag argument to .true. is currently incompatible with providing any of the other optional arguments.

The arguments are:

distgrid
Incoming DistGrid object.
[firstExtra]
Extra elements added to the first DE along each dimension. This increases the size of the index space compared to that of the incoming distgrid. The decomposition of the enlarged index space is constructed to align with the original index space provided by distgrid. The default is a zero vector.
[lastExtra]
Extra elements added to the last DE along each dimension. This increases the size of the index space compared to that of the incoming distgrid. The decomposition of the enlarged index space is constructed to align with the original index space provided by distgrid. The default is a zero vector.
[indexflag]
If present, override the indexflag setting of the incoming distgrid. See section 54.29 for a complete list of options. By default use the indexflag setting of distgrid.
[connectionList]
If present, override the connections of the incoming distgrid. See section 36.7.2 for the associated Set() method. By default use the connections defined in distgrid.
[balanceflag]
If set to .true, rebalance the incoming distgrid decompositon to exactly one DE per PET. The DEs along each dimension are chosen to provide the most balanced decomposition across all dimensions. The default is .false., leaving the decomposition, and therefore the number of DEs unchanged. For this case, PETs might end up with no DE, one DE, or multiple DEs.
[delayout]
If present, override the DELayout of the incoming distgrid. By default use the DELayout defined in distgrid.
[vm]
If present, the DistGrid object and the DELayout object are created on the specified ESMF_VM object. The default is to use the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.5 ESMF_DistGridCreate - Create DistGrid object from DistGrid (multi-tile version)


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDGT(distgrid, firstExtraPTile, &
     lastExtraPTile, indexflag, connectionList, balanceflag, &
     delayout, vm, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDGT
ARGUMENTS:
     type(ESMF_DistGrid),           intent(in)            :: distgrid
     integer, target,               intent(in)            :: firstExtraPTile(:,:)
     integer, target,               intent(in)            :: lastExtraPTile(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_Index_Flag),         intent(in),  optional :: indexflag
     type(ESMF_DistGridConnection), intent(in),  optional :: connectionList(:)
     logical,                       intent(in),  optional :: balanceflag
     type(ESMF_DELayout),           intent(in),  optional :: delayout
     type(ESMF_VM),                 intent(in),  optional :: vm
     integer,                       intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create a new DistGrid from an existing DistGrid, keeping the decomposition unchanged, unless balanceflag=.true. (see below). The firstExtraPTile and lastExtraPTile arguments allow extra elements to be added at the first/last edge DE in each dimension. The method also allows the indexflag to be set. Further, if the connectionList argument provided in it will be used to set connections in the newly created DistGrid, otherwise the connections of the incoming DistGrid will be used.

The balanceflag argument allows a change in the decomposition, and thus of the number of DEs. An attempt is made to decompose the index space into as many DEs as there are PETs in the VM for which the DistGrid is created. See the argument description for details.

Setting the balanceflag argument to .true. is currently incompatible with providing any of the other optional arguments.

The arguments are:

distgrid
Incoming DistGrid object.
firstExtraPTile
Extra elements added to the first DE along each dimension for each tile. This increases the size of the index space compared to that of the incoming distgrid. The decomposition of the enlarged index space is constructed to align with the original index space provided by distgrid. The default is a zero vector.
lastExtraPTile
Extra elements added to the last DE along each dimension for each tile. This increases the size of the index space compared to that of the incoming distgrid. The decomposition of the enlarged index space is constructed to align with the original index space provided by distgrid. The default is a zero vector.
[indexflag]
If present, override the indexflag setting of the incoming distgrid. See section 54.29 for a complete list of options. By default use the indexflag setting of distgrid.
[connectionList]
If present, override the connections of the incoming distgrid. See section 36.7.2 for the associated Set() method. By default use the connections defined in distgrid.
[balanceflag]
If set to .true, rebalance the incoming distgrid decompositon. An attempt is made to come out at exactly one DE per PET. However, if there are more tiles than PETs, the lower PETs will hold more than one DE. For cases where there are more PETs than tiles, each tile is decomposed into DEs as to provide the most balanced decomposition across all dimensions. The default is .false., leaving the decomposition, and therefore the number of DEs unchanged. For this case, PETs might end up with no DE, one DE, or multiple DEs.
[delayout]
If present, override the DELayout of the incoming distgrid. By default use the DELayout defined in distgrid.
[vm]
If present, the DistGrid object and the DELayout object are created on the specified ESMF_VM object. The default is to use the VM of the current context.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.6 ESMF_DistGridCreate - Create DistGrid object with regular decomposition


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateRD(minIndex, maxIndex, regDecomp, &
     decompflag, regDecompFirstExtra, regDecompLastExtra, deLabelList, &
     indexflag, connectionList, delayout, vm, indexTK, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateRD
ARGUMENTS:
     integer,                        intent(in)            :: minIndex(:)
     integer,                        intent(in)            :: maxIndex(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                target, intent(in),  optional :: regDecomp(:)
     type(ESMF_Decomp_Flag), target, intent(in),  optional :: decompflag(:)
     integer,                target, intent(in),  optional :: regDecompFirstExtra(:)
     integer,                target, intent(in),  optional :: regDecompLastExtra(:)
     integer,                target, intent(in),  optional :: deLabelList(:)
     type(ESMF_Index_Flag),          intent(in),  optional :: indexflag
     type(ESMF_DistGridConnection),  intent(in),  optional :: connectionList(:)
     type(ESMF_DELayout),            intent(in),  optional :: delayout
     type(ESMF_VM),                  intent(in),  optional :: vm
     type(ESMF_TypeKind_Flag),       intent(in),  optional :: indexTK
     integer,                        intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_DistGrid from a single logically rectangular tile. The tile has a regular decomposition, where the tile is decomposed into a fixed number of DEs along each dimension. A regular decomposition of a single tile is expressed by a single regDecomp list of DE counts in each dimension.

The arguments are:

minIndex
Index space tuple of the lower corner of the single tile.
maxIndex
Index space tuple of the upper corner of the single tile.
[regDecomp]
List of DE counts for each dimension. The total deCount is determined as the product of regDecomp elements. By default regDecomp = (/deCount,1,...,1/), where deCount is the number of DEs in the delayout. If the default delayout is used, the deCount is equal to petCount. This leads to a simple 1 DE per PET distribution, where the decompsition is only along the first dimension.
[decompflag]
List of decomposition flags indicating how each dimension of the tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions. See section 54.14 for a list of valid decomposition options.
[regDecompFirstExtra]
Specify how many extra elements on the first DEs along each dimension to consider when applying the regular decomposition algorithm. This does not add extra elements to the index space defined by minIndex and maxIndex. Instead regDecompFirstExtra is used to correctly interpret the specified index space: The regDecomp is first applied to the index space without the extra elements. The extra elements are then added back in to arrive at the final decomposition. This is useful when aligning the decomposition of index spaces that only differ in extra elements along the edges, e.g. when dealing with different stagger locations. The default is a zero vector, assuming no extra elements.
[regDecompLastExtra]
Specify how many extra elements on the last DEs along each dimension to consider when applying the regular decomposition algorithm. This does not add extra elements to the index space defined by minIndex and maxIndex. Instead regDecompLastExtra is used to correctly interpret the specified index space: The regDecomp is first applied to the index space without the extra elements. The extra elements are then added back in to arrive at the final decomposition. This is useful when aligning the decomposition of index spaces that only differ in extra elements along the edges, e.g. when dealing with different stagger locations. The default is a zero vector, assuming no extra elements.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the column major order of the regDecomp argument.
[indexflag]
Indicates whether the indices provided by the minIndex and maxIndex arguments are forming a global index space or not. This does not affect the indices held by the DistGrid object, which are always identical to what was specified by minIndex and maxIndex, regardless of the indexflag setting. However, it does affect whether an ESMF_Array object created on the DistGrid can choose global indexing or not. The default is ESMF_INDEX_DELOCAL. See section 54.29 for a complete list of options.
[connectionList]
List of ESMF_DistGridConnection objects, defining connections between DistGrid tiles in index space. See section 36.7.2 for the associated Set() method.
[delayout]
ESMF_DELayout object to be used. If a DELayout object is specified its deCount must match the number indicated by regDecomp. By default a new DELayout object will be created with the correct number of DEs.
[vm]
If present, the DistGrid object (and the DELayout object if not provided) are created on the specified ESMF_VM object. The default is to use the VM of the current context.
[indexTK]
Typekind used for global sequence indexing. See section 54.61 for a list of typekind options. Only integer types are supported. The default is to have ESMF automatically choose between ESMF_TYPEKIND_I4 and ESMF_TYPEKIND_I8, depending on whether the global number of elements held by the DistGrid is below or above the 32-bit limit, respectively. Because of the use of signed integers for sequence indices, element counts of $> 2^{31}-1 = 2,147,483,647$ will switch to 64-bit indexing.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.7 ESMF_DistGridCreate - Create DistGrid object with regular decomposition (multi-tile version)


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateRDT(minIndexPTile, maxIndexPTile, &
     regDecompPTile, decompflagPTile, regDecompFirstExtraPTile,&
     regDecompLastExtraPTile, deLabelList, indexflag, connectionList, &
     delayout, vm, indexTK, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateRDT
ARGUMENTS:
     integer,                        intent(in)            :: minIndexPTile(:,:)
     integer,                        intent(in)            :: maxIndexPTile(:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                        intent(in),  optional :: regDecompPTile(:,:)
     type(ESMF_Decomp_Flag), target, intent(in),  optional :: decompflagPTile(:,:)
     integer,                target, intent(in),  optional :: regDecompFirstExtraPTile(:,:)
     integer,                target, intent(in),  optional :: regDecompLastExtraPTile(:,:)
     integer,                        intent(in),  optional :: deLabelList(:)
     type(ESMF_Index_Flag),          intent(in),  optional :: indexflag
     type(ESMF_DistGridConnection),  intent(in),  optional :: connectionList(:)
     type(ESMF_DELayout),            intent(in),  optional :: delayout
     type(ESMF_VM),                  intent(in),  optional :: vm
     type(ESMF_TypeKind_Flag),       intent(in),  optional :: indexTK
     integer,                        intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_DistGrid from multiple logically rectangular tiles. Each tile has a regular decomposition, where the tile is decomposed into a fixed number of DEs along each dimension. A regular decomposition of a multi-tile DistGrid is expressed by a list of DE count vectors, one vector for each tile. If a DELayout is specified, it must contain at least as many DEs as there are tiles.

The arguments are:

minIndexPTile
The first index provides the index space tuple of the lower corner of a tile. The second index indicates the tile number.
maxIndexPTile
The first index provides the index space tuple of the upper corner of a tile. The second index indicates the tile number.
[regDecompPTile]
List of DE counts for each dimension. The second index steps through the tiles. The total deCount is determined as ths sum over the products of regDecomp elements for each tile. By default each tile is decomposed only along the first dimension. The default number of DEs per tile is at least 1, but may be greater for the leading tiles if the deCount is greater than the tileCount. If no DELayout is specified, the deCount is by default set equal to the number of PETs (petCount), or the number of tiles (tileCount), which ever is greater. This means that as long as petCount > tileCount, the resulting default distribution will be 1 DE per PET. Notice that some tiles may be decomposed into more DEs than other tiles.
[decompflagPTile]
List of decomposition flags indicating how each dimension of each tile is to be divided between the DEs. The default setting is ESMF_DECOMP_BALANCED in all dimensions for all tiles. See section 54.14 for a list of valid decomposition flag options. The second index indicates the tile number.
[regDecompFirstExtraPTile]
Specify how many extra elements on the first DEs along each dimension to consider when applying the regular decomposition algorithm. This does not add extra elements to the index space defined by minIndex and maxIndex. Instead regDecompFirstExtraPTile is used to correctly interpret the specified index space: The regDecomp is first applied to the index space without the extra elements. The extra elements are then added back in to arrive at the final decomposition. This is useful when aligning the decomposition of index spaces that only differ in extra elements along the edges, e.g. when dealing with different stagger locations. The default is a zero vector, assuming no extra elements.
[regDecompLastExtraPTile]
Specify how many extra elements on the last DEs along each dimension to consider when applying the regular decomposition algorithm. This does not add extra elements to the index space defined by minIndex and maxIndex. Instead regDecompLastExtraPTile is used to correctly interpret the specified index space: The regDecomp is first applied to the index space without the extra elements. The extra elements are then added back in to arrive at the final decomposition. This is useful when aligning the decomposition of index spaces that only differ in extra elements along the edges, e.g. when dealing with different stagger locations. The default is a zero vector, assuming no extra elements.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the column major order of the regDecompPTile elements in the sequence as they appear following the tile index.
[indexflag]
Indicates whether the indices provided by the minIndexPTile and maxIndexPTile arguments are forming a global index space or not. This does not affect the indices held by the DistGrid object, which are always identical to what was specified by minIndexPTile and maxIndexPTile, regardless of the indexflag setting. However, it does affect whether an ESMF_Array object created on the DistGrid can choose global indexing or not. The default is ESMF_INDEX_DELOCAL. See section 54.29 for a complete list of options.
[connectionList]
List of ESMF_DistGridConnection objects, defining connections between DistGrid tiles in index space. See section 36.7.2 for the associated Set() method.
[delayout]
Optional ESMF_DELayout object to be used. By default a new DELayout object will be created with as many DEs as there are PETs, or tiles, which ever is greater. If a DELayout object is specified, the number of DEs must match regDecompPTile, if present. In the case that regDecompPTile was not specified, the deCount must be at least that of the default DELayout. The regDecompPTile will be constructed accordingly.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[indexTK]
Typekind used for global sequence indexing. See section 54.61 for a list of typekind options. Only integer types are supported. The default is to have ESMF automatically choose between ESMF_TYPEKIND_I4 and ESMF_TYPEKIND_I8, depending on whether the global number of elements held by the DistGrid is below or above the 32-bit limit, respectively. Because of the use of signed integers for sequence indices, element counts of $> 2^{31}-1 = 2,147,483,647$ will switch to 64-bit indexing.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.8 ESMF_DistGridCreate - Create DistGrid object with DE blocks


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDB(minIndex, maxIndex, deBlockList, &
     deLabelList, indexflag, connectionList, delayout, vm, &
     indexTK, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDB
ARGUMENTS:
     integer,                       intent(in)            :: minIndex(:)
     integer,                       intent(in)            :: maxIndex(:)
     integer,                       intent(in)            :: deBlockList(:,:,:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                       intent(in),  optional :: deLabelList(:)
     type(ESMF_Index_Flag),         intent(in),  optional :: indexflag
     type(ESMF_DistGridConnection), intent(in),  optional :: connectionList(:)
     type(ESMF_DELayout),           intent(in),  optional :: delayout
     type(ESMF_VM),                 intent(in),  optional :: vm
     type(ESMF_TypeKind_Flag),      intent(in),  optional :: indexTK
     integer,                       intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_DistGrid from a single logically rectangular tile with decomposition specified by deBlockList.

The arguments are:

minIndex
Index space tuple of the lower corner of the single tile.
maxIndex
Index space tuple of the upper corner of the single tile.
deBlockList
List of DE-local blocks. The third index of deBlockList steps through the deBlock elements (i.e. deCount), which are defined by the first two indices. The first index must be of size dimCount and the second index must be of size 2. Each element of deBlockList defined by the first two indices hold the following information.
                     +---------------------------------------> 2nd index
                     |    1               2           
                     | 1  minIndex(1)    maxIndex(1)
                     | 2  minIndex(2)    maxIndex(2)
                     | .  minIndex(.)    maxIndex(.)
                     | .
                     v
                    1st index
It is required that there be no overlap between the DE blocks.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the order of DEs in the deBlockList argument.
[indexflag]
Indicates whether the indices provided by the minIndex and maxIndex arguments are forming a global index space or not. This does not affect the indices held by the DistGrid object, which are always identical to what was specified by minIndex and maxIndex, regardless of the indexflag setting. However, it does affect whether an ESMF_Array object created on the DistGrid can choose global indexing or not. The default is ESMF_INDEX_DELOCAL. See section 54.29 for a complete list of options.
[connectionList]
List of ESMF_DistGridConnection objects, defining connections between DistGrid tiles in index space. See section 36.7.2 for the associated Set() method.
[delayout]
Optional ESMF_DELayout object to be used. By default a new DELayout object will be created with the correct number of DEs. If a DELayout object is specified its number of DEs must match the number indicated by regDecomp.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[indexTK]
Typekind used for global sequence indexing. See section 54.61 for a list of typekind options. Only integer types are supported. The default is to have ESMF automatically choose between ESMF_TYPEKIND_I4 and ESMF_TYPEKIND_I8, depending on whether the global number of elements held by the DistGrid is below or above the 32-bit limit, respectively. Because of the use of signed integers for sequence indices, element counts of $> 2^{31}-1 = 2,147,483,647$ will switch to 64-bit indexing.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.9 ESMF_DistGridCreate - Create DistGrid object with DE blocks (multi-tile version)


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDBT(minIndexPTile, maxIndexPTile, deBlockList, &
     deToTileMap, deLabelList, indexflag, connectionList, &
     delayout, vm, indexTK, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDBT
ARGUMENTS:
     integer,                       intent(in)            :: minIndexPTile(:,:)
     integer,                       intent(in)            :: maxIndexPTile(:,:)
     integer,                       intent(in)            :: deBlockList(:,:,:)
     integer,                       intent(in)            :: deToTileMap(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                       intent(in),  optional :: deLabelList(:)
     type(ESMF_Index_Flag),         intent(in),  optional :: indexflag
     type(ESMF_DistGridConnection), intent(in),  optional :: connectionList(:)
     type(ESMF_DELayout),           intent(in),  optional :: delayout
     type(ESMF_VM),                 intent(in),  optional :: vm
     type(ESMF_TypeKind_Flag),      intent(in),  optional :: indexTK
     integer,                       intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_DistGrid on multiple logically rectangular tiles with decomposition specified by deBlockList.

The arguments are:

minIndexPTile
The first index provides the index space tuple of the lower corner of a tile. The second index indicates the tile number.
maxIndexPTile
The first index provides the index space tuple of the upper corner of a tile. The second index indicates the tile number.
deBlockList
List of DE-local blocks. The third index of deBlockList steps through the deBlock elements (i.e. deCount), which are defined by the first two indices. The first index must be of size dimCount and the second index must be of size 2. Each element of deBlockList defined by the first two indices hold the following information.
                     +---------------------------------------> 2nd index
                     |    1               2           
                     | 1  minIndex(1)    maxIndex(1)
                     | 2  minIndex(2)    maxIndex(2)
                     | .  minIndex(.)    maxIndex(.)
                     | .
                     v
                    1st index
It is required that there be no overlap between the DE blocks.
deToTileMap
List assigning each DE to a specific tile. The size of deToTileMap must be equal to deCount. The order of DEs is the same as in deBlockList.
[deLabelList]
List assigning DE labels to the default sequence of DEs. The default sequence is given by the order of DEs in the deBlockList argument.
[indexflag]
Indicates whether the indices provided by the minIndexPTile and maxIndexPTile arguments are forming a global index space or not. This does not affect the indices held by the DistGrid object, which are always identical to what was specified by minIndexPTile and maxIndexPTile, regardless of the indexflag setting. However, it does affect whether an ESMF_Array object created on the DistGrid can choose global indexing or not. The default is ESMF_INDEX_DELOCAL. See section 54.29 for a complete list of options.
[connectionList]
List of ESMF_DistGridConnection objects, defining connections between DistGrid tiles in index space. See section 36.7.2 for the associated Set() method.
[delayout]
Optional ESMF_DELayout object to be used. By default a new DELayout object will be created with the correct number of DEs. If a DELayout object is specified its number of DEs must match the number indicated by regDecomp.
[vm]
Optional ESMF_VM object of the current context. Providing the VM of the current context will lower the method's overhead.
[indexTK]
Typekind used for global sequence indexing. See section 54.61 for a list of typekind options. Only integer types are supported. The default is to have ESMF automatically choose between ESMF_TYPEKIND_I4 and ESMF_TYPEKIND_I8, depending on whether the global number of elements held by the DistGrid is below or above the 32-bit limit, respectively. Because of the use of signed integers for sequence indices, element counts of $> 2^{31}-1 = 2,147,483,647$ will switch to 64-bit indexing.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.10 ESMF_DistGridCreate - Create 1D DistGrid object from user's arbitrary sequence index list 1 DE per PET


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDBAI1D1DE(arbSeqIndexList, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDBAI1D1DE
ARGUMENTS:
     integer, intent(in)            :: arbSeqIndexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_DistGrid of dimCount 1 from a PET-local list of sequence indices. The PET-local size of the arbSeqIndexList argument determines the number of local elements in the created DistGrid. The sequence indices must be unique across all PETs. A default DELayout with 1 DE per PET across all PETs of the current VM is automatically created.

This is a collective method across the current VM.

The arguments are:

arbSeqIndexList
List of arbitrary sequence indices that reside on the local PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.11 ESMF_DistGridCreate - Create 1D DistGrid object from user's arbitrary 64-bit sequence index list 1 DE per PET


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDBAI1D1DEI8(arbSeqIndexList, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDBAI1D1DEI8
ARGUMENTS:
     integer(ESMF_KIND_I8),  intent(in)            :: arbSeqIndexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
DESCRIPTION:

Create an ESMF_DistGrid of dimCount 1 from a PET-local list of sequence indices. The PET-local size of the arbSeqIndexList argument determines the number of local elements in the created DistGrid. The sequence indices must be unique across all PETs. A default DELayout with 1 DE per PET across all PETs of the current VM is automatically created.

This is a collective method across the current VM.

The arguments are:

arbSeqIndexList
List of arbitrary sequence indices that reside on the local PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.12 ESMF_DistGridCreate - Create 1D DistGrid object from user's arbitrary sequence index list multiple DE/PET


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDBAI1D(arbSeqIndexList, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDBAI1D
ARGUMENTS:
     type(ESMF_PtrInt1D), intent(in) :: arbSeqIndexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional  :: rc
DESCRIPTION:

Create an ESMF_DistGrid of dimCount 1 from a PET-local list of sequence index lists. The PET-local size of the arbSeqIndexList argument determines the number of local DEs in the created DistGrid. Each of the local DEs is associated with as many index space elements as there are arbitrary sequence indices in the associated list. The sequence indices must be unique across all DEs. A default DELayout with the correct number of DEs per PET is automatically created.

This is a collective method across the current VM.

The arguments are:

arbSeqIndexList
List of arbitrary sequence index lists that reside on the local PET.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.13 ESMF_DistGridCreate - Create (1+n)D DistGrid object from user's arbitrary sequence index list and minIndexPTile/maxIndexPTile


INTERFACE:

   ! Private name; call using ESMF_DistGridCreate()
   function ESMF_DistGridCreateDBAI(arbSeqIndexList, arbDim, &
     minIndexPTile, maxIndexPTile, rc)
RETURN VALUE:
     type(ESMF_DistGrid) :: ESMF_DistGridCreateDBAI
ARGUMENTS:
     integer, intent(in)            :: arbSeqIndexList(:)
     integer, intent(in)            :: arbDim
     integer, intent(in)            :: minIndexPTile(:)
     integer, intent(in)            :: maxIndexPTile(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer, intent(out), optional :: rc
STATUS:

DESCRIPTION:

Create an ESMF_DistGrid of dimCount $1+n$, where $n=$ size(minIndexPTile) = size(maxIndexPTile).

The resulting DistGrid will have a 1D distribution determined by the PET-local arbSeqIndexList. The PET-local size of the arbSeqIndexList argument determines the number of local elements along the arbitrarily distributed dimension in the created DistGrid. The sequence indices must be unique across all PETs. The associated, automatically created DELayout will have 1 DE per PET across all PETs of the current VM.

In addition to the arbitrarily distributed dimension, regular DistGrid dimensions can be specified in minIndexPTile and maxIndexPTile. The $n$ dimensional subspace spanned by the regular dimensions is "multiplied" with the arbitrary dimension on each DE, to form a $1+n$ dimensional total index space described by the DistGrid object. The arbDim argument allows to specify which dimension in the resulting DistGrid corresponds to the arbitrarily distributed one.

This is a collective method across the current VM.

The arguments are:

arbSeqIndexList
List of arbitrary sequence indices that reside on the local PET.
arbDim
Dimension of the arbitrary distribution.
minIndexPTile
Index space tuple of the lower corner of the tile. The arbitrary dimension is not included in this tile
maxIndexPTile
Index space tuple of the upper corner of the tile. The arbitrary dimension is not included in this tile
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.14 ESMF_DistGridDestroy - Release resources associated with a DistGrid


INTERFACE:

   subroutine ESMF_DistGridDestroy(distgrid, noGarbage, rc)
ARGUMENTS:
     type(ESMF_DistGrid), intent(inout)          :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,             intent(in),   optional :: noGarbage
     integer,             intent(out),  optional :: rc
STATUS:

DESCRIPTION:

Destroys an ESMF_DistGrid, releasing the resources associated with the object.

By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.

The arguments are:

distgrid
ESMF_DistGrid object to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.15 ESMF_DistGridGet - Get object-wide DistGrid information


INTERFACE:

   ! Private name; call using ESMF_DistGridGet()
   subroutine ESMF_DistGridGetDefault(distgrid, delayout, &
     dimCount, tileCount, deCount, localDeCount, minIndexPTile, maxIndexPTile, &
     elementCountPTile, elementCountPTileI8, minIndexPDe, maxIndexPDe, &
     elementCountPDe, elementCountPDeI8, localDeToDeMap, deToTileMap, &
     indexCountPDe, collocation, regDecompFlag, indexTK, indexflag, &
     connectionCount, connectionList, rc)
ARGUMENTS:
     type(ESMF_DistGrid),      intent(in)            :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     type(ESMF_DELayout),      intent(out), optional :: delayout
     integer,                  intent(out), optional :: dimCount
     integer,                  intent(out), optional :: tileCount
     integer,                  intent(out), optional :: deCount
     integer,                  intent(out), optional :: localDeCount
     integer,          target, intent(out), optional :: minIndexPTile(:,:)
     integer,          target, intent(out), optional :: maxIndexPTile(:,:)
     integer,          target, intent(out), optional :: elementCountPTile(:)
 integer(ESMF_KIND_I8),target, intent(out), optional :: elementCountPTileI8(:)
     integer,          target, intent(out), optional :: minIndexPDe(:,:)
     integer,          target, intent(out), optional :: maxIndexPDe(:,:)
     integer,          target, intent(out), optional :: elementCountPDe(:)
 integer(ESMF_KIND_I8),target, intent(out), optional :: elementCountPDeI8(:)
     integer,          target, intent(out), optional :: localDeToDeMap(:)
     integer,          target, intent(out), optional :: deToTileMap(:)
     integer,          target, intent(out), optional :: indexCountPDe(:,:)
     integer,          target, intent(out), optional :: collocation(:)
     logical,                  intent(out), optional :: regDecompFlag
     type(ESMF_TypeKind_Flag), intent(out), optional :: indexTK
     type(ESMF_Index_Flag),    intent(out), optional :: indexflag
     integer,                  intent(out), optional :: connectionCount
     type(ESMF_DistGridConnection), &
                       target, intent(out), optional :: connectionList(:)
     integer,                  intent(out), optional :: rc
STATUS:

DESCRIPTION:

Access internal DistGrid information.

The arguments are:

distgrid
Queried ESMF_DistGrid object.
[delayout]
ESMF_DELayout object associated with distgrid.
[dimCount]
Number of dimensions (rank) of distgrid.
[tileCount]
Number of tiles in distgrid.
[deCount]
Number of DEs in the DELayout in distgrid.
[localDeCount]
Number of local DEs in the DELayout in distgrid on this PET.
[minIndexPTile]
Lower index space corner per tile. Must enter allocated with shape(minIndexPTile) == (/dimCount, tileCount/).
[maxIndexPTile]
Upper index space corner per tile. Must enter allocated with shape(maxIndexPTile) == (/dimCount, tileCount/).
[elementCountPTile]
Number of elements in the exclusive region per tile. Must enter allocated with shape(elementCountPTile) == (/tileCount/). An error will be returned if any of the counts goes above the 32-bit limit.
[elementCountPTileI8]
Same as elementCountPTile, but of 64-bit integer kind.
[minIndexPDe]
Lower index space corner per DE. Must enter allocated with shape(minIndexPDe) == (/dimCount, deCount/).
[maxIndexPDe]
Upper index space corner per DE. Must enter allocated with shape(maxIndexPDe) == (/dimCount, deCount/).
[elementCountPDe]
Number of elements in the exclusive region per DE. Must enter allocated with shape(elementCountPDe) == (/deCount/). An error will be returned if any of the counts goes above the 32-bit limit.
[elementCountPDeI8]
Same as elementCountPDe, but of 64-bit integer kind.
[localDeToDeMap]
Global DE index for each local DE. Must enter allocated with shape(localDeToDeMap) == (/localDeCount/). It is recommended to use a lower bound of 0 for localDeToDeMap, in order to support direct indexing into this map with a zero-based localDe variable.
[deToTileMap]
Map each DE uniquely to a tile. Must enter allocated with shape(deToTileMap) == (/deCount/). It is recommended to use a lower bound of 0 for deToTileMap, in order to support direct indexing into this map with a zero-based de variable.
[indexCountPDe]
Number of indices for each dimension per DE. Must enter allocated with shape(indexCountPDe) == (/dimCount, deCount/).
[collocation]
Collocation identifier for each dimension. Must enter allocated with shape(collocation) == (/dimCount/).
[regDecompFlag]
Decomposition scheme. A return value of .true. indicates a regular decomposition, i.e. the decomposition is based on a logically rectangular scheme with specific number of DEs along each dimension. A return value of .false. indicates that the decomposition was not generated from a regular decomposition description, e.g. a deBlockList was used instead.
[indexTK]
Typekind used by the global sequence indexing. See section 54.61 for a list of typekind options. Only the integer types are supported for sequence indices.
[indexflag]
Return the indexing option used by the distgrid object. See section 54.29 for a complete list of options.
[connectionCount]
Number of explicitly defined connections in distgrid.
[connectionList]
List of explicitly defined connections in distgrid. Must enter allocated with shape(connectionList) == (/connectionCount/).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.16 ESMF_DistGridGet - Get DE-local DistGrid information


INTERFACE:

   ! Private name; call using ESMF_DistGridGet()
   subroutine ESMF_DistGridGetPLocalDe(distgrid, localDe, &
     de, tile, collocation, arbSeqIndexFlag, seqIndexList, seqIndexListI8, &
     elementCount, elementCountI8, rc)
ARGUMENTS:
     type(ESMF_DistGrid),      intent(in)            :: distgrid
     integer,                  intent(in)            :: localDe
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                  intent(out), optional :: de
     integer,                  intent(out), optional :: tile
     integer,                  intent(in),  optional :: collocation
     logical,                  intent(out), optional :: arbSeqIndexFlag
     integer,          target, intent(out), optional :: seqIndexList(:)
 integer(ESMF_KIND_I8),target, intent(out), optional :: seqIndexListI8(:)
     integer,                  intent(out), optional :: elementCount
     integer,                  intent(out), optional :: elementCountI8
     integer,                  intent(out), optional :: rc
STATUS:

DESCRIPTION:

Access internal DistGrid information.

The arguments are:

distgrid
Queried ESMF_DistGrid object.
localDe
Local DE for which information is requested. [0,..,localDeCount-1]
[de]
The global DE associated with the localDe. DE indexing starts at 0.
[tile]
The tile on which the localDe is located. Tile indexing starts at 1.
[collocation]
Collocation for which information is requested. Default to first collocation in collocation list.
[arbSeqIndexFlag]
A returned value of .true. indicates that the collocation is associated with arbitrary sequence indices. For .false., canoncial sequence indices are used.
[seqIndexList]
The sequence indices associated with the localDe. This argument must enter allocated with a size equal to elementCountPDe(localDeToDeMap(localDe)). An error will be returned if any of the sequence indices are above the 32-bit limit.
[seqIndexListI8]
Same as seqIndexList, but of 64-bit integer kind.
[elementCount]
Number of elements in the localDe, i.e. identical to elementCountPDe(localDeToDeMap(localDe)). An error will be returned if the count is above the 32-bit limit.
[elementCountI8]
Same as elementCount, but of 64-bit integer kind.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.17 ESMF_DistGridGet - Get DE-local DistGrid information for a specific dimension


INTERFACE:

   ! Private name; call using ESMF_DistGridGet()
   subroutine ESMF_DistGridGetPLocalDePDim(distgrid, localDe, dim, &
            indexList, rc)
ARGUMENTS:
     type(ESMF_DistGrid),    intent(in)            :: distgrid
     integer,                intent(in)            :: localDe
     integer,                intent(in)            :: dim
     integer,        target, intent(out)           :: indexList(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
STATUS:

DESCRIPTION:

Access internal DistGrid information.

The arguments are:

distgrid
Queried ESMF_DistGrid object.
localDe
Local DE for which information is requested. [0,..,localDeCount-1]
dim
Dimension for which information is requested. [1,..,dimCount]
indexList
Upon return this holds the list of DistGrid tile-local indices for localDe along dimension dim. The supplied variable must be at least of size indexCountPDe(dim, localDeToDeMap(localDe)).
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.18 ESMF_DistGridIsCreated - Check whether a DistGrid object has been created


INTERFACE:

   function ESMF_DistGridIsCreated(distgrid, rc)
RETURN VALUE:
     logical :: ESMF_DistGridIsCreated
ARGUMENTS:
     type(ESMF_DistGrid), intent(in)            :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,             intent(out), optional :: rc
DESCRIPTION:

Return .true. if the distgrid has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

distgrid
ESMF_DistGrid queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.19 ESMF_DistGridMatch - Check if two DistGrid objects match


INTERFACE:

   function ESMF_DistGridMatch(distgrid1, distgrid2, rc)
RETURN VALUE:
     type(ESMF_DistGridMatch_Flag) :: ESMF_DistGridMatch
ARGUMENTS:
     type(ESMF_DistGrid),  intent(in)            :: distgrid1
     type(ESMF_DistGrid),  intent(in)            :: distgrid2
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(out), optional :: rc
DESCRIPTION:

Determine to which level distgrid1 and distgrid2 match.

Returns a range of values of type ESMF_DistGridMatch_Flag, indicating how closely the DistGrids match. For a description of the possible return values, see 36.2.1. Note that this call only performs PET local matching. Different return values may be returned on different PETs for the same DistGrid pair.

The arguments are:

distgrid1
ESMF_DistGrid object.
distgrid2
ESMF_DistGrid object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.20 ESMF_DistGridPrint - Print DistGrid information


INTERFACE:

   subroutine ESMF_DistGridPrint(distgrid, rc)
ARGUMENTS:
     type(ESMF_DistGrid),  intent(in)            :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(out), optional :: rc
STATUS:

DESCRIPTION:

Prints internal information about the specified ESMF_DistGrid object to stdout.

The arguments are:

distgrid
Specified ESMF_DistGrid object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.21 ESMF_DistGridSet - Set arbitrary sequence for a specific local DE


INTERFACE:

   ! Private name; call using ESMF_DistGridSet()
   subroutine ESMF_DistGridSetPLocalDe(distgrid, localDe, collocation, &
     seqIndexList, seqIndexListI8, rc)
ARGUMENTS:
     type(ESMF_DistGrid),      intent(inout)         :: distgrid
     integer,                  intent(in)            :: localDe
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                  intent(in),  optional :: collocation
     integer,          target, intent(in),  optional :: seqIndexList(:)
 integer(ESMF_KIND_I8),target, intent(in),  optional :: seqIndexListI8(:)
     integer,                  intent(out), optional :: rc
DESCRIPTION:

Set DistGrid information for a specific local DE.

The arguments are:

distgrid
Queried ESMF_DistGrid object.
localDe
Local DE for which information is set. [0,..,localDeCount-1]
[collocation]
Collocation for which information is set. Default to first collocation in collocation list.
[seqIndexList]
Sequence indices for the elements on localDe. The seqIndexList must provide exactly elementCountPDe(localDeToDeMap(localDe) many entries. When this argument is present, the indexTK of distgrid will be set to ESMF_TYPEKIND_I4. This argument is mutually exclusive with seqIndexListI8. Only one or the other must be provided. An error will be returned otherwise.
[seqIndexListI8]
Same as seqIndexList, but of 64-bit integer kind. When this argument is present, the indexTK of distgrid will be set to ESMF_TYPEKIND_I8. This argument is mutually exclusive with seqIndexList. Only one or the other must be provided. An error will be returned otherwise.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.6.22 ESMF_DistGridValidate - Validate DistGrid internals


INTERFACE:

   subroutine ESMF_DistGridValidate(distgrid, rc)
ARGUMENTS:
     type(ESMF_DistGrid),  intent(in)            :: distgrid
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,              intent(out), optional :: rc
STATUS:

DESCRIPTION:

Validates that the distgrid is internally consistent. The method returns an error code if problems are found.

The arguments are:

distgrid
Specified ESMF_DistGrid object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.


36.7 Class API: DistGridConnection Methods

36.7.1 ESMF_DistGridConnectionGet - Get DistGridConnection


INTERFACE:

   subroutine ESMF_DistGridConnectionGet(connection, &
     tileIndexA, tileIndexB, dimCount, positionVector, orientationVector, rc)
ARGUMENTS:
     type(ESMF_DistGridConnection), intent(in)            :: connection
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                       intent(out), optional :: tileIndexA
     integer,                       intent(out), optional :: tileIndexB
     integer,                       intent(out), optional :: dimCount
     integer,                       intent(out), optional :: positionVector(:)
     integer,                       intent(out), optional :: orientationVector(:)
     integer,                       intent(out), optional :: rc
DESCRIPTION:

Get connection parameters from an ESMF_DistGridConnection object. This interface provides access to all variables required to create a new connection using the ESMF_DistGridConnectionSet() method.

The arguments are:

connection
DistGridConnection object.
[tileIndexA]
Index of one of the two connected tiles.
[tileIndexB]
Index of the other connected tile.
[dimCount]
Number of dimensions of positionVector.
[positionVector]
Position of tile B's minIndex with respect to tile A's minIndex. This array's size should be at least equal to dimCount.
[orientationVector]
Lists which dimension of tile A is associated to which dimension of tile B. Negative index values may be used to indicate a reversal in index orientation. Should be at least of size dimCount.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

36.7.2 ESMF_DistGridConnectionSet - Set DistGridConnection


INTERFACE:

   subroutine ESMF_DistGridConnectionSet(connection, tileIndexA, tileIndexB, &
     positionVector, orientationVector, rc)
ARGUMENTS:
     type(ESMF_DistGridConnection),intent(out)         :: connection
     integer,                     intent(in)           :: tileIndexA
     integer,                     intent(in)           :: tileIndexB
     integer,                     intent(in)           :: positionVector(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                     intent(in), optional :: orientationVector(:)
     integer,                     intent(out), optional:: rc
STATUS:

DESCRIPTION:

Set an ESMF_DistGridConnection object to represent a connection according to the provided index space information.

The arguments are:

connection
DistGridConnection object.
tileIndexA
Index of one of the two tiles that are to be connected.
tileIndexB
Index of one of the two tiles that are to be connected.
positionVector
Position of tile B's minIndex with respect to tile A's minIndex.
[orientationVector]
Associates each dimension of tile A with a dimension in tile B's index space. Negative index values may be used to indicate a reversal in index orientation. It is erroneous to associate multiple dimensions of tile A with the same index in tile B. By default orientationVector = (/1,2,3,.../), i.e. same orientation as tile A.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.


36.8 Class API: DistGridRegDecomp Methods

36.8.1 ESMF_DistGridRegDecompSetCubic - Construct a DistGrid regDecomp


INTERFACE:

   subroutine ESMF_DistGridRegDecompSetCubic(regDecomp, deCount, rc)
ARGUMENTS:
     integer,        target, intent(out)           :: regDecomp(:)
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),  optional :: deCount
     integer,                intent(out), optional :: rc
DESCRIPTION:

Construct a regular decomposition argument for DistGrid that is most cubic in dimCount dimensions, and multiplies out to deCount DEs. The factorization is stable monotonic decreasing, ensuring that earlier entries in regDecomp are larger or equal to the later entires.

The arguments are:

regDecomp
The regular decomposition description being constructed.
[deCount]
The number of DEs. Defaults to petCount.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37 RouteHandle Class

37.1 Description

The ESMF RouteHandle class provides a unified interface for all route-based communication methods across the Field, FieldBundle, Array, and ArrayBundle classes. All route-based communication methods implement a pre-computation step, returning a RouteHandle, an execution step, and a release step. Typically the pre-computation, or Store() step will be a lot more expensive (both in memory and time) than the execution step. The idea is that once precomputed, a RouteHandle will be executed many times over during a model run, making the execution time a very performance critical piece of code. In ESMF, Regridding, Redisting, and Haloing are implemented as route-based communication methods. The following sections discuss the RouteHandle concepts that apply uniformly to all route-based communication methods, across all of the above mentioned classes.

37.2 Use and Examples

The user interacts with the RouteHandle class through the route-based communication methods of Field, FieldBundle, Array, and ArrayBundle. The usage of these methods are described in detail under their respective class documentation section. The following examples focus on the RouteHandle aspects common across classes and methods.


37.2.1 Bit-for-bit reproducibility

Bit-for-bit (bfb) reproducibility is at the core of the regression testing schemes of many scientific model codes. The bfb requirement makes it possible to easily compare the numerical results of simulation runs using standard binary diff tools.

While bfb reproducibility is desirable (and often required) for regression testing, it does limit the available performance optimization opportunities. Especially in highly parallelized code, best performance is often achieved by allowing operations to occur in a flexible order. Under some conditions, however, a change in the order of numerical operations leads to small numerical differences in the results, breaking bfb reproducibility.

ESMF provides the following three levels of bfb reproducibility support, with the associated performance optimization implications:

The following discussion uses very simple numerical examples to demonstrate how the order of terms in a sum can lead to results that are not bit-for-bit identical. The examples use single precision, ESMF_KIND_R4 numbers, but the concepts apply the same to double precision, ESMF_KIND_R8; only that the decimals, for which bfb differences in the sums occur, are different ones.

With sumA, sumB, sumC, sumD, and sumE all of type real(ESMF_KIND_R4), one finds the following bfb differences:

  sumA = (0.5 + 0.1) + 0.1        ! results in 0.700000048
  sumB = 0.5 + (0.1 + 0.1)        ! results in 0.699999988
  
  sumC = 0.5 +  0.2 + 0.1  + 0.1  ! results in 0.900000036
  sumD = 0.5 + (0.2 + 0.1) + 0.1  ! results in 0.900000036
  sumE = 0.5 + (0.2 + 0.1 + 0.1)  ! results in 0.899999976

These differences result from the fact that many decimals (even very simple ones like 0.1 or 0.2) lead to periodic binary floating point numbers. Periodic floating point numbers must be truncated when represented by a finite number of bits, leading to small rounding errors. Further truncation occurs when the radix point of two numbers must be aligned during floating point arithmetic, resulting in bit shifts for one of the numbers. The resulting truncation error depends on the precise numbers that need alignment. As a result, executing the "same" sum in a different order can lead to different truncation steps and consequently in results that are not bit-for-bit identical.

In order to help users with the implementation of their bfb requirement, ESMF provides different levels of control over the term order in sparse matrix multiplications, while at the same time offering performance optimization options. In all there are three arguments that will be introduced in the following paragraphs: srcTermProcessing, termorderflag, and pipelineDepth.

For the purpose of demonstration, a one-dimensional, arbitrarily distributed source Array is constructed. There are three Array elements on each of the four PETs. Their local storage indices, sequence indices, and data values are as follows:

  
           +-----+-------+----------------+------------+
           | PET | index | sequence index | data value |
           +-----+-------+----------------+------------+
           |  0  |   1   |          1     |     0.5    |
           |  0  |   2   |          6     |     0.1    |
           |  0  |   3   |          9     |     0.1    |
           +-----+-------+----------------+------------+
           |  1  |   1   |          4     |     0.5    |
           |  1  |   2   |          3     |     0.1    |
           |  1  |   3   |         10     |     0.1    |
           +-----+-------+----------------+------------+
           |  2  |   1   |         11     |     0.5    |
           |  2  |   2   |          7     |     0.1    |
           |  2  |   3   |          5     |     0.1    |
           +-----+-------+----------------+------------+
           |  3  |   1   |          8     |     0.1    |
           |  3  |   2   |          2     |     0.2    |
           |  3  |   3   |         12     |     0.1    |
           +-----+-------+----------------+------------+

The destination Array consists of only a single element, located on PET 0:

  
           +-----+-------+----------------+------------+
           | PET | index | sequence index | data value |
           +-----+-------+----------------+------------+
           |  0  |   1   |          1     |     n/a    |
           +-----+-------+----------------+------------+

As a first example consider the following sparse matrix with three entries:

    factorIndexList(1,1) = 1  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 6  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 9  ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

In ESMF, the order in which the sparse matrix entries are specified in factorIndexList and factorList, or on which PET they are provided, is completely irrelevant. The term order in the resulting sparse matrix sums is not affected by it.

There is one aspect of the sparse matrix format, however, that is relevant to the bfb considerations: When multiple entries for the same (src, dst) pair are present in a sparse matrix definition, the entries are summed into a single (src, dst) entry. Therefore, even if there are multiple sparse matrix entries for the same (src, dst) pair, there will only be a single term for it in the resulting expression.

Going back to the three term sparse matrix definition above, the canonical term order is defined by the source sequence indices in ascending order. With (src,dst) denoting the sparse matrix factors, and s(src) and d(dst) denoting source and destination Array elements, respectively, for src and dst sequence indices, the sum in canonical order is:

d(1) = (1,1)*s(1) + (6,1)*s(6) + (9,1)*s(9)

For simplicity, the factors in all of the examples are set to 1.0, allowing us to drop them in the expressions. This helps focus on the critical issue - term order:

d(1) = s(1) + s(6) + s(9)

There are two parameters that affect term order in the ESMF sparse matrix multiplication (SMM), and therefore must be considered in the context of bfb reproducibility. First there is the srcTermProcessing parameter which controls grouping of source terms located on the same PET. The value of the srcTermProcessing parameter indicates the maximum number of terms that may be grouped into partial sums on the source PET. Setting srcTermProcessing to 1 means that no partial sums are formed on the source side, however, the source terms are multiplied with their respective sparse matrix factor before being sent to the destination PET. Setting srcTermProcessing to 0 prevents these products from being carried out on the source side, and the source Array elements are sent unmodified. Depending on the distribution of the source Array, values greater than 1 for srcTermProcessing can lead to partial sums and thus may have impact on the bfb reproducibility of the SMM.

The second parameter that may have bfb effects comes into play at execution-time of a precomputed RouteHandle. It is accessible via the termorderflag argument; a typed flag with the following values:

ESMF_TERMORDER_SRCSEQ

First using srcTermProcessing=0 at store time and termorderflag=ESMF_TERMORDER_SRCSEQ at execution time, the canonical term order is expected:

d(1) = s(1) + s(6) + s(9) = 0.5 + 0.1 + 0.1 = sumA

  ! forced srcTermProcessing
  srcTermProcessing = 0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCSEQ, rc=rc)

  if (localPet == 0) then
    print *, "result SRCSEQ#1 = ", farrayPtr(1), " expect: ", sumA
    if (farrayPtr(1) /= sumA) &
      finalrc = ESMF_FAILURE
  endif

The order of source terms across PETs is expected to have no effect on the bfb reproducibility of the result for ESMF_TERMORDER_SRCSEQ. To test this, a sparse matrix is used where the source terms originate from different PETs.

    factorIndexList(1,1) = 4  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 5  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 12 ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

Again the srcTermProcessing argument is kept at 0, ensuring that none of the source terms are grouped into partial sums.

  ! forced srcTermProcessing
  srcTermProcessing = 0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCSEQ, rc=rc)

Under ESMF_TERMORDER_SRCSEQ it does not matter on which PET a source term is located, the order of source terms is strictly defined by the order of source sequence indices:

d(1) = s(4) + s(5) + s(12) = 0.5 + 0.1 + 0.1 = sumA

  if (localPet == 0) then
    print *, "result SRCSEQ#2 = ", farrayPtr(1), " expect: ", sumA
    if (farrayPtr(1) /= sumA) &
      finalrc = ESMF_FAILURE
  endif

The same sparse matrix leads to bfb differences in the result when executed with the ESMF_TERMORDER_SRCPET option. This is demonstrated further down in result SRCPET#4.

ESMF_TERMORDER_SRCPET

All source terms coming from the same PET

In the following examples the srcTermProcessing argument at store-time is first set to 0, forcing all of the source terms to be sent to the destination PET unmodified. We start by going back to the initial sparse matrix where all of the source terms are located on the same PET.

    factorIndexList(1,1) = 1  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 6  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 9  ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

  ! forced srcTermProcessing
  srcTermProcessing=0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

Then, at execution time, the ESMF_TERMORDER_SRCPET option is used.

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

Here all of the source elements originate from the same PET (PET 0). This fact, together with the ESMF_TERMORDER_SRCPET execution-time option, results in the following canonical term order:

d(1) = s(1) + s(6) + s(9) = 0.5 + 0.1 + 0.1 = sumA

This is exactly the same term order that was used above to produce the result stored in sumA.

  if (localPet == 0) then
    print *, "result SRCPET#1 = ", farrayPtr(1), " expect: ", sumA
    if (farrayPtr(1) /= sumA) &
      finalrc = ESMF_FAILURE
  endif

The sequence indices of the source terms are the only relevant aspect in determining the source term order. Consider, for example, the following sparse matrix, where again all source terms are located on the same PET (PET 2):

    factorIndexList(1,1) = 11 ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 5  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 7  ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

This time the source term order in memory is not the same as their sequence index order. Specifically, the sequence indices of the source terms, in the order they are stored in memory, is 11, 7, 5 (see the source Array diagram above for reference). Further, as mentioned already, the order of entries in the sparse matrix also have not bearing on the term order in the SMM sums. Then, for the ESMF_TERMORDER_SRCPET option, and because all source terms are located on the same PET, the resulting source term order is the canonical one determined by the source term sequence indices alone:

d(1) = s(5) + s(7) + s(11)

Filling in the source element data, we find

d(1) = 0.1 + 0.1 + 0.5,

which is expected to be bfb equivalent to the result stored in sumB from above.

  ! forced srcTermProcessing
  srcTermProcessing=0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#2 = ", farrayPtr(1), " expect: ", sumB
    if (farrayPtr(1) /= sumB) &
      finalrc = ESMF_FAILURE
  endif

Source terms coming from different PETs

When the source terms are distributed across multiple PETs, the ESMF_TERMORDER_SRCPET option first bundles the terms according to the PET on which they are stored. These source term "bundles" are then arranged in an order that depends on the source PET position relative to the destination PET: starting with the bundle for which the source PET is the same as the destination PET, the source term bundles are placed in descending order with respect to their source PET, modulo petCount. The terms within each source term bundle are further sorted in the canonical order according to their sequence index.

The following sparse matrix demonstrates the effect of the ESMF_TERMORDER_SRCPET option.

    factorIndexList(1,1) = 1  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 3  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 7  ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

Here the source terms are located on PETs 0, 1, and 2. Using a [] notion to indicate the source PET of each term, the term order under ESMF_TERMORDER_SRCPET is given by:

d(1) = s(1)[0] + s(7)[2] + s(3)[1] = 0.5 + 0.1 + 0.1

This is again the same order of terms that was used to produce the result stored in sumA above.

  ! forced srcTermProcessing
  srcTermProcessing=0
    
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#3 = ", farrayPtr(1), " expect: ", sumA
    if (farrayPtr(1) /= sumA) &
      finalrc = ESMF_FAILURE
  endif

In the above example, the fact that the terms were ordered by source PET first, did not lead to numerical bfb differences compared to the canonical source term order. However, this was purely coincidental in the way the numbers worked out for this example. The following case looks at a situation where the source PET order does lead to a result that shows bfb differences compared to the canonical term order.

    factorIndexList(1,1) = 4  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 5  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 12 ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

The canonical source term order of this SMM sum, determined by the source sequence indices alone, is:

d(1) = s(4) + s(5) + s(12) = 0.5 + 0.1 + 0.1,

which again would lead to a result that is bfb identical to sumA. However, this is not the term order resulting from the ESMF_TERMORDER_SRCPET option. The actual order for this option is:

d(1) = s(12)[3] + s(5)[2] + s(4)[1] = 0.1 + 0.1 + 0.5,

resulting in a sum that is bfb identical to sumB instead.

  ! forced srcTermProcessing
  srcTermProcessing=0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#4 = ", farrayPtr(1), " expect: ", sumB
    if (farrayPtr(1) /= sumB) &
      finalrc = ESMF_FAILURE
  endif

Grouping of source terms coming from the same PET

So far the srcTermProcessing argument was kept at 0, and therefore source term grouping had not to be considered. Source term grouping is only possible for terms that originate from the same PET. In preparation for a closer look at the bfb effects of source term grouping, consider a sparse matrix where two of the source terms are located on the same PET.

    factorIndexList(1,1) = 1  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 5  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 7  ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.

Here one of the source terms is located on PET 0 while the other two source terms are originating on PET 2. Keeping the srcTermProcessing argument at 0 first, the term order under ESMF_TERMORDER_SRCPET is given by:

d(1) = s(1)[0] + s(5)[2] + s(7)[2] = 0.5 + 0.1 + 0.1

And again the result is expected to be bfb identical to the number stored in sumA.

  ! forced srcTermProcessing
  srcTermProcessing=0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#5 = ", farrayPtr(1), " expect: ", sumA
    if (farrayPtr(1) /= sumA) &
      finalrc = ESMF_FAILURE
  endif

The same result is also expected with srcTermProcessing set to 1. A value of 1 indicates that the multiplication of the source term with its sparse matrix factor is carried out on the source side before being sent to the destination PET. The final sum is still carried out in the same order on the destination PET, essentially resulting in the exact same bfb identical sum as for srcTermProcessing set to 0.

  ! forced srcTermProcessing
  srcTermProcessing=1
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#6 = ", farrayPtr(1), " expect: ", sumA
    if (farrayPtr(1) /= sumA) &
      finalrc = ESMF_FAILURE
  endif

Increasing the srcTermProcessing argument to 2 (or higher) results in source term grouping of the terms (up to the number specified in srcTermProcessing) that are on the same source PET.

d(1) = s(1)[0] + ( s(5)[2] + s(7)[2] ) = 0.5 + (0.1 + 0.1)

This result is bfb identical to first adding 0.1 and 0.1 into a partial sum, and then adding this sum to 0.5. This is the exact grouping of terms that was used to obtain the result stored in sumB from above.

  ! forced srcTermProcessing
  srcTermProcessing=2
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#7 = ", farrayPtr(1), " expect: ", sumB
    if (farrayPtr(1) /= sumB) &
      finalrc = ESMF_FAILURE
  endif

In order to explore the effects of the srcTermProcessing argument further, more terms on the same source PET are needed in the SMM sum. The following sparse matrix has four entries, three of which originate from the same PET (PET 3).

    factorIndexList(1,1) = 1  ! src seq index
    factorIndexList(2,1) = 1  ! dst seq index
    factorList(1) = 1.
    factorIndexList(1,2) = 2  ! src seq index
    factorIndexList(2,2) = 1  ! dst seq index
    factorList(2) = 1.
    factorIndexList(1,3) = 8  ! src seq index
    factorIndexList(2,3) = 1  ! dst seq index
    factorList(3) = 1.
    factorIndexList(1,4) = 12 ! src seq index
    factorIndexList(2,4) = 1  ! dst seq index
    factorList(4) = 1.

Setting the srcTermProcessing argument back to 0 puts the terms in PET order, and canonical order for each PET bundle.

d(1) = s(1)[0] + s(2)[3] + s(8)[3] + s(12)[3] = 0.5 + 0.2 + 0.1 + 0.1

The bfb identical result for this sum was calculated and stored in variable sumC above.

  ! forced srcTermProcessing
  srcTermProcessing=0
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#8 = ", farrayPtr(1), " expect: ", sumC
    if (farrayPtr(1) /= sumC) &
      finalrc = ESMF_FAILURE
  endif

Setting the srcTermProcessing argument to a value of 2 results in the following source term grouping:

d(1) = s(1)[0] + ( s(2)[3] + s(8)[3] ) + s(12)[3] = 0.5 + ( 0.2 + 0.1 ) + 0.1,

where the (0.2 + 0.1) partial sum is carried out on source PET 3, and then sent to the destination PET (PET 0), together with the unmodified data from source element 8 (0.1). The final sum is performed on PET 0. The result is identical to the precomputed value stored in sumD. The numbers work out in a way where this result is bfb identical to the previous result, i.e. sumC. However, this bfb match is purely coincidental.

  ! forced srcTermProcessing
  srcTermProcessing=2
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#9 = ", farrayPtr(1), " expect: ", sumD
    if (farrayPtr(1) /= sumD) &
      finalrc = ESMF_FAILURE
  endif

Increasing the srcTermProcessing argument up to 3 results in a three term partial sum on PET 3:

d(1) = s(1)[0] + ( s(2)[3] + s(8)[3] + s(12)[3] ) = 0.5 + ( 0.2 + 0.1 + 0.1 ).

Again the final sum is performed on PET 0. The result is bfb identical to the number stored in sumE, which, for the chosen numbers, works out to have a bfb difference compared to sumC and sumD.

  ! forced srcTermProcessing
  srcTermProcessing=3
  
  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if (localPet == 0) then
    print *, "result SRCPET#10 = ", farrayPtr(1), " expect: ", sumE
    if (farrayPtr(1) /= sumE) &
      finalrc = ESMF_FAILURE
  endif

Reproducibility and Performance

The above examples show how bit-for-bit (bfb) reproducibility is a result of controlling the term order. ESMF offers several options to control the term order in the sparse matrix multiplication (SMM) implementation:

The term order in a SMM operation does not only affect the bfb reproducibility of the result, but also affects the SMM performance. The precise performance implications of a specific term order are complicated and strongly depend on the exact problem structure, as well as on the details of the compute hardware. ESMF implements an auto-tuning mechanism that can be used to conveniently determine a close to optimal set of SMM performance parameters.

There are two SMM performance parameters in ESMF that are encoded into a RouteHandle during store-time: srcTermProcessing and pipelineDepth. The first one affects the term order in the SMM sums and has been discussed in detail above. The second parameter, pipelineDepth, determines how many in- and out-bound messages may be outstanding on each PET. It has no effect on the term order and does not lead to bfb differences in the SMM results. However, in order to achieve good performance reproducibility, the user has the option to pass in a fixed value of the pipelineDepth argument when precomputing RouteHandles.

Store calls that take the srcTermProcessing and/or pipelineDepth argument specify them as optional with intent(inout). Omitting the argument when calling, or passing a variable that is set to a negative number, indicates that the respective parameter needs to be determined by the library. Further, if a variable with a negative value was passed in, then the variable is overwritten and replaced by the auto-tuned value on return. Through this mechanism a user can leverage the built-in auto-tuning feature of ESMF to obtain the best possible performance for a specific problem on a particular compute hardware, while still ensuring bfb and performance reproducibility between runs. The following example shows code that first checks if previously stored SMM performance parameters are available in a file on disk, and then either reads and uses them, or else uses auto-tuning to determine the parameters before writing them to file. For simplicity the same sparse matrix as in the previous example is used.

  ! precondition the arguments for auto-tuning and overwriting
  srcTermProcessing = -1  ! init negative value
  pipelineDepth     = -1  ! init negative value

  ! get a free Fortran i/o unit
  call ESMF_UtilIOUnitGet(unit=iounit, rc=rc)

  ! try to open the file that holds the SMM parameters
  open(unit=iounit, file="smmParameters.dat", status="old", action="read", &
    form="unformatted", iostat=iostat)
  
  if (iostat == 0) then
    ! the file was present -> read from it and close it again
    read(unit=iounit, iostat=iostat) srcTermProcessing, pipelineDepth, &
      sumCompare
    close(unit=iounit)
  endif
  
  if ((localPet == 0) .and. (iostat == 0)) then
    print *, "SMM parameters successfully read from file"
    print *, " srcTermProcessing=", srcTermProcessing, " pipelineDepth=", &
      pipelineDepth, " ==>> sumCompare=", sumCompare
  endif

  call ESMF_ArraySMMStore(srcArray, dstArray, &
    factorIndexList=factorIndexList, factorList=factorList, &
    routehandle=rh, srcTermProcessing=srcTermProcessing, &
    pipelineDepth=pipelineDepth, rc=rc)

  call ESMF_ArraySMM(srcArray, dstArray, routehandle=rh, &
    termorderflag=ESMF_TERMORDER_SRCPET, rc=rc)

  if ((localPet == 0) .and. (iostat /= 0)) then
    print *, "SMM parameters determined via auto-tuning -> dump to file"
    open(unit=iounit, file="smmParameters.dat", status="unknown", &
      action="write", form="unformatted")
    write(unit=iounit) srcTermProcessing, pipelineDepth, farrayPtr(1)
    close(unit=iounit)
  endif
  
  if (localPet == 0) then
    if (iostat /= 0) then
      ! cannot do bfb comparison of the result without reference
      print *, "result SRCPET#11 = ", farrayPtr(1)
    else
      ! do bfb comparison of the result against reference
      print *, "result SRCPET#11 = ", farrayPtr(1), " expect: ", sumCompare
      if (farrayPtr(1) /= sumCompare) then
        finalrc = ESMF_FAILURE
        write (msg, *) "Numerical difference detected: ", &
          farrayPtr(1)-sumCompare
        call ESMF_LogWrite(msg, ESMF_LOGMSG_INFO)
      endif
    endif
  endif

Running this example for the first time exercises the auto-tuning branch. The auto-tuned srcTermProcessing and pipelineDepth parameters are then used in the SMM execution, as well as written to file. The SMM result variable is also written to the same file for test purposes. Any subsequent execution of the same example branches into the code that reads the previously determined SMM execution parameters from file, re-using them during store-time. This ensures bfb reproducibility of the SMM result, which is tested in this example by comparing to the previously stored value.


37.2.2 Asynchronous RouteHandle communication with VMEpoch

The RouteHandle based communication calls of the Array and Field classes provide the routesyncflag argument. This argument allows the user to specify that the initiated call should not block. Additional calls are necessary to wait for a previously initiated communication call to finish. For a detailed discussion see section 28.2.20. Building on these primitives, asynchronous communications patterns can be implemented in user code. However, a more elegant option to achive asynchronous behavior between sending and receiving PETs of RouteHandle based communications is provided by the VMEpoch feature discussed here.

The VMEpoch is a low level message aggregation and buffering approach. The VM level details are discussed under the ESMF_VM section in 51.3.11.

There are several advantages of VMEpoch over the direct use of non-blocking RouteHandle based communication calls:

The ESMF_VMEpoch API consists of two interfaces: ESMF_VMEpochEnter() and ESMF_VMEpochExit(). Inside an epoch, communication calls are aggregated. Data transfers on the src side are not issued until the epoch is exited. On the dst side, a single data transfer is received from any of the sending PETs, and then divided over the individual receive calls.

In the following code example, the srcArray has DEs on PET 0 and 1, while dstArray has DEs on PET 2 and 3. Both Arrays are operating on the same global index space. A Redist() RouteHandle rh is created in the usual manner.

  call ESMF_ArrayRedistStore(srcArray, dstArray, routehandle=rh, rc=rc)

The precomputed rh can be used as usual. Here the use inside an active VMEpoch is demonstrated.

First enter the VMEpoch using ESMF_VMEpochEnter(), specifying the kind of epoch. Currently only a single VMEpoch kind is available: ESMF_VMEPOCH_BUFFER.

  call ESMF_VMEpochEnter(epoch=ESMF_VMEPOCH_BUFFER, rc=rc)

SRC side (PET 0 & 1): The sending PETs do not block.

DST side (PET 2 & 3): The receiving PETs do not block.

Notice that ESMF implements a throttle on the VMEpoch as to limit the number of queued message between PETs. This is necessary to protect the receiving side in the EAGER regime where MPI implementations send the data from the sending to the receiving side, assuming the receiving side will always be able to buffer. In cases where the sending side runs far ahead of the receiving side, this strategy can lead to increasing memory pressure on the receiving side, ultimatily resulting in out-of-memory conditions. The default throttle in VMEpoch is set to $10$ outstanding message between any two PETs. It can be adjusted by specifying the throttle argument when calling ESMF_VMEpochEnter().

Next the actual communication method, ESMF_ArrayRedist(), is called in the usual manner.

  call ESMF_ArrayRedist(srcArray, dstArray, routehandle=rh, rc=rc)

SRC side (PET 0 & 1): The sending PETs block until the locally needed send buffers are available, and all local data manipulations and data movements into the send buffers have completed.

Waiting for the send buffers comes into play when a VMEpoch is entered and exited multiple times. The same send buffer is reused each time for the same src-dst-PET pairs (and grown automatically if needed). Each send buffer becomes available once the MPI layer has indicated that the associated, previous MPI_Isend() has completed locally.

Once the send buffer for a specific PET pair is available, the local data movements defined by the rh and message aggregation must complete before returning. For Regrid() and SMM() operations the srcTermProcessing argument specified during Store() determines the amount of local data processing. Once returned, it is safe to modify the srcArray data on the local PET.

DST side (PET 2 & 3): The receiving PETs block on the aggregated data from the src side for which the local PET has a dependency defined via the rh. Once received, the data is processed locally, and moved into the final location under the dstArray. On return, it is safe to access the data in dstArray on the local PET.

Notice that any number of RouteHandle based communication calls can be made inside the same active VMEpoch. In fact, aggregating messages from multiple exchanges is the typical use case of the VMEpoch approach. Additional communication calls cam either involve different RouteHandles, or even the same rh for different src/dst Array pairs.

When using communication calls that allow the specification of the termorderflag, e.g. ESMF_FieldRegrid(), ESMF_ArraySMM(), etc. inside a VMEpoch, it is important to set it to either ESMF_TERMORDER_SRCPET or ESMF_TERMORDER_SRCSEQ. The default value of ESMF_TERMORDER_FREE is not compatible with VMEpoch.

Finally the active VMEpoch is exited by calling ESMF_VMEpochExit().

  call ESMF_VMEpochExit(rc=rc)

SRC side (PET 0 & 1): The sending PETs post their local MPI_Isend() calls. This is non-blocking.

DST side (PET 2 & 3): The receiving PETs do not block.

As part of the final clean-up the rh is being released as usual.

  call ESMF_ArrayRedistRelease(routehandle=rh, rc=rc)


37.2.3 Creating a RouteHandle from an existing RouteHandle - Transfer to a different set of PETs

Typically a RouteHandle object is created indirectly, i.e. without explicitly calling the ESMF_RouteHandleCreate() method. The RouteHandle object is a byproduct of calling communication Store() methods like ESMF_FieldRegridStore().

One exception to this rule is when creating a duplicate RouteHandle from an existing RouteHandle object. In this case the ESMF_RouteHandleCreate() method is used explicitly. While this method allows to create a duplicate RouteHandle on the exact same set of PETs as the original RouteHandle, the real purpose of duplication is the transfer of a precomputed RouteHandle to a different set of PETs. This is an efficient way to reduce the total time spent in Store() calls, for situations where the same communication pattern repeats for multiple components.

This example demonstrates the transfer of a RouteHandle from one set of PETs to another by first introducing three components. Component A is defined on the first half of available PETs.

  petCountA = petCount/2  ! component A gets half the PETs

  allocate(petListA(petCountA))
  do i=1, petCountA
    petListA(i) = i-1 ! PETs are base 0
  enddo
  
  compA = ESMF_GridCompCreate(petList=petListA, rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

The other two components, B1 and B2, split the remaining PETs evenly.

  petCountR = petCount - petCountA
  petCountB1 = petCountR / 2
  
  allocate(petListB1(petCountB1))
  do i=1, petCountB1
    petListB1(i) = petCountA + i-1 ! PETs are base 0
  enddo

  allocate(petListB2(petCountR-petCountB1))
  do i=1, petCountR-petCountB1
    petListB2(i) = petCountA + petCountB1 + i-1 ! PETs are base 0
  enddo

  compB1 = ESMF_GridCompCreate(petList=petListB1, rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

  compB2 = ESMF_GridCompCreate(petList=petListB2, rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

Skipping all of the standard superstructure code, assume that fieldA has been created by component A, has been reconciled across all PETs via a StateReconcile() call, and accessed via a StateGet(). The same is true for fieldB1 and fieldB2 from components B1 and B2, respectively.

Now the RouteHandle rh1 for a Redist operation is precomputed between fieldA and fieldB1.

  call ESMF_FieldRedistStore(srcField=fieldA, dstField=fieldB1, &
    routehandle=rh1, rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

The communication pattern stored in rh1 is between the PETs associated with component A and those associated with component B1. Now component B2 is simply a second instance of the same component code as B1, but on a different set of PETs. The ESMF_RouteHandleCreate() method can be used to transfer rh1 to the set of PETs that is consistent with fieldA to fieldB2 communication.

In order to transfer a RouteHandle to a different set of PETs, the originPetList and targetPetList must be constructed. The originPetList is the union of source and destination PETs (in that order) for which rh1 was explicitly computed via the Store() call:

  allocate(originPetList(size(petListA)+size(petListB1)))
  originPetList(1:size(petListA)) = petListA(:)
  originPetList(size(petListA)+1:) = petListB1(:)

The targetPetList is the union of source and destination PETs (in that order) for which the target RouteHandle (i.e. rh2) will be defined:

  allocate(targetPetList(size(petListA)+size(petListB2)))
  targetPetList(1:size(petListA)) = petListA(:)
  targetPetList(size(petListA)+1:) = petListB2(:)

Now the new RouteHandle rh2 can be created easily from the exising RouteHandle rh1, suppling the origin and target petLists.

  rh2 = ESMF_RouteHandleCreate(rh1, originPetList=originPetList, &
    targetPetList=targetPetList, rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

The new RouteHandle rh2 is completely independent of the original RouteHandle. In fact, it is perfectly fine to destroy (or release) rh1 while holding on to rh2.

  call ESMF_RouteHandleDestroy(rh1, noGarbage=.true., rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

Finally the rh2 object can be used to redistribute data from fieldA to fieldB2.

  call ESMF_FieldRedist(srcField=fieldA, dstField=fieldB2, &
    routehandle=rh2, rc=rc)
  if (ESMF_LogFoundError(rcToCheck=rc, msg=ESMF_LOGERR_PASSTHRU, &
    line=__LINE__, &
    file=__FILE__)) &
    call ESMF_Finalize(endflag=ESMF_END_ABORT)

The communication pattern held by rh2 is idential to what whould have been created by an explicit ESMF_FieldRedistStore() call. However, the ESMF_RouteHandleCreate() call used to create rh2 from rh1 is much faster than the full RedistStore() operation.


37.2.4 Write a RouteHandle to file and creating a RouteHandle from file

Communication Store() methods, like ESMF_FieldRegridStore(), are used to create RouteHandles. These methods can be expensive, both with respect to temporary memory requirements as well as the time they require to execute. Often the associated cost is acceptable because Store() calls are typically used during the initialization phase of the application. The cost of RouteHandle generation is therefore armorized over the entire run phase of the application, where the RouteHandle is applied over and over to transfer data according to the same communication pattern.

However, especially for short production runs, an expensive initialization time can become problematic. In such cases it is useful to write the RouteHandle to file. Subsequent application runs can then re-create the RouteHandle during initialization, simply from file at a fraction of the time of the original Store() call.

First a RouteHandle must be created using one of the ESMF Store() methods.

  call ESMF_FieldRedistStore(srcField=fieldA, dstField=fieldB, &
    routehandle=rh1, rc=rc)

Now the RouteHandle object rh1 can be written to file using the collective ESMF_RouteHandleWrite() method.

  call ESMF_RouteHandleWrite(rh1, fileName="testWrite.RH", rc=rc)

This creates a single binary file with name testWrite.RH. The information from across all PETs that define rh1 is contained in this file.

At this point, the original RouteHandle is no longer needed and can be destroyed.

  call ESMF_RouteHandleDestroy(rh1, noGarbage=.true., rc=rc)

The RouteHandle just deleted can easily be re-created using the ESMF_RouteHandleCreate() method that accepts the file name as an argument. This is a collective method that must be called on exactly the same number of PETs that was used for the original Store() and Write() calls that generated the file.

  rh2 = ESMF_RouteHandleCreate(fileName="testWrite.RH", rc=rc)

Finally the re-created RouteHandle, rh2, can be used to execute the communication pattern originally computed in rh1.

  call ESMF_FieldRedist(srcField=fieldA, dstField=fieldB, &
    routehandle=rh2, rc=rc)

Once done with rh2, the RouteHandle can be destroyed as usual.

  call ESMF_RouteHandleDestroy(rh2, noGarbage=.true., rc=rc)


37.2.5 Reusablity of RouteHandles and interleaved distributed and undistributed dimensions

A RouteHandle object is typically created during a communication Store() call, e.g. an ESMF_FieldRegridStore(). Other communication methods with Store() are Halo, Redist, and SMM. The primary input objects of a Store() call are either Fields, Arrays, FieldBundles, or ArrayBundles. There will be an object for the source side, and another object for the destination side. Both objects must be of the same type.

  srcField = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, rc=rc)

  dstField = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, rc=rc)

  call ESMF_FieldRegridStore(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, rc=rc)

The purpose of the explicit Store() call is to separate out the expensive part of creating the RouteHandle object for a specific communication patter, from the less expensive part of applying it. Applying the RouteHandle results in data movement between the source and destination objects. Once a RouteHandle is available, it is reusable. This means it can be applied over and over again to communicate data from the source to the destination object.

  do i=1, 10
    ! repeatedly applying the routehandle
    call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
      routehandle=routehandle, rc=rc)
  enddo

Reusability of a RouteHandle object extends beyond re-applying it to the same source/destination object pair that was used during Store(). The same RouteHandle can be applied to a different object pair, as long as these criterial are met:

The following examples will discuss in detail what this means in practice.

First consider the case where a second pair of source and destination Fields is created identical to the first set. The precomputed RouteHandle is immediatly reusable for this new Field pair to carry out the regrid operation.

  srcField2 = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, rc=rc)

  dstField2 = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, rc=rc)

  ! applying the same routehandle to a different pair of fields
  call ESMF_FieldRegrid(srcField=srcField2, dstField=dstField2, &
    routehandle=routehandle, rc=rc)

The same RouteHandle stays re-usable even for a Field pair where source and destination have one or more additional undistributed dimensions. Here a single undistributed dimension is added. By default all undistributed dimensions will be ordered after the distributed dimensions provided by the Grid object.

  srcField3 = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), &  ! undistributed dim last
    rc=rc)

  dstField3 = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), & ! undistributed dim last
    rc=rc)

  ! applying the same routehandle to a different pair of fields
  call ESMF_FieldRegrid(srcField=srcField3, dstField=dstField3, &
    routehandle=routehandle, rc=rc)

The undistributed dimension can also be moved into the first position, and the same RouteHandle can still be re-used. Specifying the order of dimensions in a Field is accomplished by providing the gridToFieldMap. Here the Grid dimensions are mapped to 2nd and 3rd Field dimensions, moving the undistributed dimension into the leading position.

  srcField4 = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), &
    gridToFieldMap=(/2,3/), rc=rc)  ! undistributed dim 1st

  dstField4 = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), &
    gridToFieldMap=(/2,3/), rc=rc)  ! undistributed dim 1st

  ! applying the same routehandle to a different pair of fields
  call ESMF_FieldRegrid(srcField=srcField4, dstField=dstField4, &
    routehandle=routehandle, rc=rc)

It is not necessary that the undistributed dimension is in the same position on the source and destination Field. The only criteria that needs to be satisfied is that both source and destination have the same number of undistributed elements. Here the RouteHandle is re-used for a Field pair where the destination Field interleaves the undistributed dimension between the two distributed dimensions. At the same time the source Field keeps the undistributed dimension in leading position.

  srcField5 = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), &
    gridToFieldMap=(/2,3/), rc=rc)  ! undistributed dim 1st

  dstField5 = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), &
    gridToFieldMap=(/1,3/), rc=rc)  ! undistributed dim 2nd

  ! applying the same routehandle to a different pair of fields
  call ESMF_FieldRegrid(srcField=srcField5, dstField=dstField5, &
    routehandle=routehandle, rc=rc)

In the following example the undistributed elements on the source side are spread across two undistributed dimensions. Of course the product of the two dimension sizes must equal the number of undistributed elements on the destination side, in order to fulfil the element count criteria. Here this number is 10. At two undistributed dimension on the source side are placed in first and fourth position using the gridToFieldMap. The same RouteHandle is applied to this Field pair, resulting in the desired regrid operation.

  srcField6 = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1,1/), ungriddedUBound=(/2,5/), &
    gridToFieldMap=(/2,3/), rc=rc)  ! undistributed dims 1st and 4th

  dstField6 = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, &
    ungriddedLBound=(/1/), ungriddedUBound=(/10/), &
    gridToFieldMap=(/1,3/), rc=rc)  ! undistributed dim 2nd

  ! applying the same routehandle to a different pair of fields
  call ESMF_FieldRegrid(srcField=srcField6, dstField=dstField6, &
    routehandle=routehandle, rc=rc)

While the RouteHandle was precomputed using a specific source/destination Field pair, we have seen how it can be re-used as long as the memory layout associated with the distributed (i.e. gridded) dimensions does not change. A natural extension of this feature is to allow the same RouteHandle to be re-used when source and destination are FieldBundles instead of Fields. The only requirement here is that both sides contain the same number of elements, and that each pair constructed from the source and destination side is compatible with the original pair used as shown in the examples above. Here this criteria is simply met by constructing the source and destination FieldBundles from the exact Fields used in the previous examples.

  srcFieldBundle = ESMF_FieldBundleCreate(fieldList=(/srcField, &
    srcField2, srcField3, srcField4, srcField5, srcField6/), rc=rc)

  dstFieldBundle = ESMF_FieldBundleCreate(fieldList=(/dstField, &
    dstField2, dstField3, dstField4, dstField5, dstField6/), rc=rc)

  ! applying the same routehandle to a pair of FieldBundles
  call ESMF_FieldBundleRegrid(srcFieldBundle, dstFieldBundle, &
    routehandle=routehandle, rc=rc)

On a fundamental level, RouteHandles are re-usable across objects that have the same memory layout for their distributed dimensions. Since ESMF Fields are built on top of ESMF Arrays, it is possible to re-use the same RouteHandle that was precomputed for a Field pair and apply it to a matching Array pair.

For this example, the easiest way to create Arrays with the same memory layout in the distributed dimensions is to query the source and destination Grid objects for their DistGrids. Then source and destination Arrays can be easily constructed.

  call ESMF_GridGet(srcGrid, distgrid=srcDistGrid, rc=rc)

  call ESMF_GridGet(dstGrid, distgrid=dstDistGrid, rc=rc)

  srcArray = ESMF_ArrayCreate(srcDistGrid, ESMF_TYPEKIND_R8, rc=rc)

  dstArray = ESMF_ArrayCreate(dstDistGrid, ESMF_TYPEKIND_R8, rc=rc)

  ! applying the same routehandle to an Array pair
  call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
    routehandle=routehandle, rc=rc)

Finally the resources associated with the RouteHandle object are released. The recommended way to do this is by calling into the Release() method associated with the Store() method used to create the RouteHandle.

  call ESMF_FieldRegridRelease(routehandle, rc=rc)


37.2.6 Dynamic Masking

When a RouteHandle object is created during an ESMF_FieldRegridStore() call, masking information can be provided by the user. This type of masking is said to be static, and is described in section 24.2.10. It is static, because the masks set the maximum limits of the regrid operation, which cannot be changed later. All subsequent executions of the same RouteHandle can only use elements - source or destination - that were not masked during the Store() call.

Once a RouteHandle object is available, whether it was created with or without static masking, the associated regrid operation can further be masking during RouteHandle execution . This is called dynamic masking, because it can dynamically change between subsequent RouteHandle executions. The RouteHandle itself remains unchange during this process. The dynamic masking information is processed on the fly as the RouteHandle is applied.

The following example demonstrates dynamic masking for a regrid operation between two Field objects. Although it is supported, here the regrid operation between srcField and dstField is computed without static masking.

Note that since the intention is to later use the generated RouteHandle for dynamic masking, it is important to provide the srcTermProcessing argument, which must be set equal to 0. Doing this ensures that all of the multiplying with interpolation weights, and summing of terms, is carried out on the destination side. This is critical for dynamic masking.

  srcTermProcessing=0

  call ESMF_FieldRegridStore(srcField=srcField, dstField=dstField, &
    srcTermProcessing=srcTermProcessing, routehandle=routehandle, rc=rc)

Now that routehandle is available, it can be used to execute the regrid operation over and over during the course of the simualtion run.

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, rc=rc)

Assume that during the course of the simulation the srcField becomes partially masked. This masking may be dynamically changing, as would be the case for the ice cover over the arctic ocean. Then the regrid operation represented by routehandle should dynamically adjust to only use unmasked source elements.

The dynamic masking behavior can be achieved in ESMF by setting srcField elements to a special value.

  call ESMF_FieldGet(srcField, farrayPtr=farrayPtr, rc=rc)

  ! setting an arbitrary local source element to special value 'srcMaskValue'
  farrayPtr(lbound(farrayPtr,1)+3,lbound(farrayPtr,2)+3) = srcMaskValue

Then set up an ESMF_DynamicMask object that holds information about the special mask value. The dynamic mask object further holds a pointer to the routine that will be called in order to handle dynamically masked elements.

  call ESMF_DynamicMaskSetR8R8R8(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValue, &
    dynamicMaskRoutine=simpleDynMaskProc, &
    rc=rc)

The names of the specific DynamicMaskSet methods all carry a typekind-triplet suffix. Here the suffix is R8R8R8. This indicates that the dynamicMaskRoutine argument provided is expected to deal with real(ESMF_KIND_R8) destination data (first R8 typekind), real(ESMF_KIND_R8) factors (second R8 typekind), and real(ESMF_KIND_R8) source data (third R8 typekind).

Now when the routehandle is executed, and the dynamicMask object is passed into the ESMF_FieldRegrid() call,

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, dynamicMask=dynamicMask, rc=rc)

ESMF will scan the srcField for elements that have data equal to that set by dynamicSrcMaskValue. If any are found, they are passed into the routine provided via the dynamicMaskRoutine argument.

The procedure passed through the dynamicMaskRoutine argument must satisfy exactly the following predefined interface:

    interface
      subroutine ESMF_DynamicMaskRoutineR8R8R8(dynMaskList, &
        dynamicSrcMaskValue, dynamicDstMaskValue, rc)
        use ESMF_UtilTypesMod
        implicit none
        type(ESMF_DynamicMaskElementR8R8R8), pointer        :: dynMaskList(:)
        real(ESMF_KIND_R8),            intent(in), optional :: dynamicSrcMaskValue
        real(ESMF_KIND_R8),            intent(in), optional :: dynamicDstMaskValue
        integer,                       intent(out)  :: rc
      end subroutine
    end interface

The first argument accepted according to this interface is an array of type ESMF_DynamicMaskElement. Each element of this array corresponds to a single element in the dstField that is affected by dynamic masking. For each such dstElement the complete interpolation stencile is provided by the ESMF_DynamicMaskElement derived type:

    type ESMF_DynamicMaskElementR8R8R8
      real(ESMF_KIND_R8), pointer       :: dstElement
      real(ESMF_KIND_R8), allocatable   :: factor(:)
      real(ESMF_KIND_R8), allocatable   :: srcElement(:)
    end type

Here the dstElement is a pointer to the actual element in the dstField. Thus, assigning dstElement to a value, immediately results in a value change of the element inside the dstField object. Further, the size of the factor(:) and srcElement(:) arrays is identical to each other and corresponds to the number of source elements in the interpolation stencile. Without dynamic masking, the dstElement would simply be calculated as the scalar product of factor(:) and srcElement(:).

By providing the dynamicMaskRoutine, the user has full control as to what exactly happens to destination elements that are affected by dynamic masking. For the current example, where some source elements may be marked by a special masking value, a simple scheme could be to only use non-masked source elements to calculate destination elements. The result then needs to be renormalized in order to account for the missing source elements. This could be implemented similar to the following subroutine:

    subroutine simpleDynMaskProc(dynamicMaskList, dynamicSrcMaskValue, &
      dynamicDstMaskValue, rc)
      type(ESMF_DynamicMaskElementR8R8R8), pointer        :: dynamicMaskList(:)
      real(ESMF_KIND_R8),            intent(in), optional :: dynamicSrcMaskValue
      real(ESMF_KIND_R8),            intent(in), optional :: dynamicDstMaskValue
      integer,                       intent(out)          :: rc
      integer :: i, j
      real(ESMF_KIND_R8)  :: renorm
      if (associated(dynamicMaskList)) then
        do i=1, size(dynamicMaskList)
          dynamicMaskList(i)%dstElement = 0.d0 ! set to zero
          renorm = 0.d0 ! reset
          do j=1, size(dynamicMaskList(i)%factor)
            if (.not. &
              match(dynamicSrcMaskValue,dynamicMaskList(i)%srcElement(j))) then
              dynamicMaskList(i)%dstElement = dynamicMaskList(i)%dstElement &
                + dynamicMaskList(i)%factor(j) &
                * dynamicMaskList(i)%srcElement(j)
              renorm = renorm + dynamicMaskList(i)%factor(j)
            endif
          enddo
          if (renorm > 0.d0) then
            dynamicMaskList(i)%dstElement = dynamicMaskList(i)%dstElement / renorm
          else if (present(dynamicSrcMaskValue)) then
            dynamicMaskList(i)%dstElement = dynamicSrcMaskValue
          else
            rc = ESMF_RC_ARG_BAD  ! error detected
            return
          endif
        enddo
      endif
      ! return successfully
      rc = ESMF_SUCCESS
    end subroutine

So far in the example only the srcField had been dynamically masked. However, elements in the dstField can be masked as well, following exactly the same manner.

First ensure that the dstField is in a well defined condition. This can be achived by reseting it, e.g. to zero, using the ESMF_FieldFill() method.

  call ESMF_FieldFill(dstField, dataFillScheme="const", const1=0.d0, rc=rc)

Now some of the destination elements are set to a defined masking value.

  call ESMF_FieldGet(dstField, farrayPtr=farrayPtr, rc=rc)

  ! setting an arbitrary local destination element to special value 'dstMaskValue'
  farrayPtr(lbound(farrayPtr,1)+1,lbound(farrayPtr,2)+1) = dstMaskValue

The dynamicMask is reset using the same DynamicMaskSet method as before, but in addition to the previous arguments, dynamicDstMaskValue is also specified.

  call ESMF_DynamicMaskSetR8R8R8(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValue, &
    dynamicDstMaskValue=dstMaskValue, &
    dynamicMaskRoutine=simpleDynMaskProc, &
    rc=rc)

Passing the reset dynamicMask object into ESMF_FieldRegrid() causes ESMF to not only look for source elements that match dynamicSrcMaskValue, but also destination elements that match dynamicDstMaskValue.

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, zeroregion=ESMF_REGION_EMPTY, &
    dynamicMask=dynamicMask, rc=rc)

Again an adequate procedure is supplied through dynamicMaskRoutine. For the current case, however, a suitable procedure would be inspecting the dstElement as well as all the dstElements provided via the dynMaskList argument.

Notice the zeroregion = ESMF_REGION_EMPTY specification in the ESMF_FieldRegrid() call. This setting ensures that values in the dstField remain unchanged until they are checked for dynamicDstMaskValue.

The DynamicMaskSet methods provide an argument of logical type, called handleAllElements. By default it is set to .false., which means that only elements affected by dynamic masking - as described above - are passed to the dynamicMaskRoutine. However, when handleAllElements is set to .true., all local elements on each PET are made available to the dynamicMaskRoutine. This allows the user supplied procedure to implement fully customized handling of the interpolation from source to destination, using the information supplied by ESMF.

To demonstrate this, a custom routine simpleHandleAllProc() is passed in as dynamicMaskRoutine, and handleAllElements is set to .true.. All other aspects of the user interface remain unchanged.

  call ESMF_DynamicMaskSetR8R8R8(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValue, &
    dynamicDstMaskValue=-2.d0, &
    dynamicMaskRoutine=simpleHandleAllProc, &
    handleAllElements=.true., &
    rc=rc)

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, zeroregion=ESMF_REGION_EMPTY, &
    dynamicMask=dynamicMask, rc=rc)

Dynamic masking is also available for source and destination fields that contain leading undistributed dimensions. When ESMF applies the regridding weights, it interprets the product space of leading undistributed dimensions of a Field or Array as the elements of a vector. In this approach the interpolation becomes a vector operation. When applying the concept of dynamic masking to such a vector operation, without making further assumptions, it must be assumed that different vector elements may be affected differently by the dynamic mask. ESMF therefore unrolls the vector dimension when constructing the information passed to the dynamicMaskRoutine. As a consequence of this, masking routines do not generally have to consider vectorization explicitly.

The concept is demonstrated by creating source and destination fields with one leading undistributed dimension.

  srcField = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R8, &
    gridToFieldMap=(/2,3/), ungriddedLBound=(/1/), ungriddedUBound=(/20/), &
    rc=rc)

  dstField = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R8, &
    gridToFieldMap=(/2,3/), ungriddedLBound=(/1/), ungriddedUBound=(/20/), &
    rc=rc)

A regrid operation is computed in the usual manner. In order to make the resulting RouteHandle object suitable for dynamic masking, computations are pushed completely onto the destination PETs, as in previous examples, by setting the srcTermProcessing argument to zero.

  srcTermProcessing=0

  call ESMF_FieldRegridStore(srcField=srcField, dstField=dstField, &
    srcTermProcessing=srcTermProcessing, routehandle=routehandle, rc=rc)

The same dynamicMaskRoutine as before can be used when setting up the ESMF_DynamicMask object. However, the source and destination Fields now contain 20 undistributed elements at each distributed location, and the dynamic mask routine will handle all elements that are affected by the dynamic mask conditions.

  call ESMF_DynamicMaskSetR8R8R8(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValue, &
    dynamicDstMaskValue=dstMaskValue, &
    dynamicMaskRoutine=simpleDynMaskProc, &
    rc=rc)

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, zeroregion=ESMF_REGION_EMPTY, &
    dynamicMask=dynamicMask, rc=rc)

Setting the handleAllElements to .true. will pass all elements to the dynamicMaskRoutine. There are 20 times as many elements on the source and destination side, and therefore the dynamic masking routine will handle exactly 20 times as many elements compared to the case without undistributed dimension.

  call ESMF_DynamicMaskSetR8R8R8(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValue, &
    dynamicDstMaskValue=-2.d0, &
    dynamicMaskRoutine=simpleHandleAllProc, &
    handleAllElements=.true., &
    rc=rc)

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, zeroregion=ESMF_REGION_EMPTY, &
    dynamicMask=dynamicMask, rc=rc)

For the case with handleAllElements=.true., where the entire vector of undistributed elements is passed to dynamicMaskRoutine at every distributed location, an alternative implementation option exists for the dynamic masking routine. In some cases this alternative may result in more efficient code because it allows to vectorize over the undistributed elements when summing up the interpolation terms. The alternative interface for dynamicMaskRoutine is:

    interface
      subroutine ESMF_DynamicMaskRoutineR8R8R8V(dynMaskList, &
        dynamicSrcMaskValue, dynamicDstMaskValue, rc)
        use ESMF_UtilTypesMod
        implicit none
        type(ESMF_DynamicMaskElementR8R8R8V), pointer       :: dynMaskList(:)
        real(ESMF_KIND_R8),            intent(in), optional :: dynamicSrcMaskValue
        real(ESMF_KIND_R8),            intent(in), optional :: dynamicDstMaskValue
        integer,                       intent(out)  :: rc
      end subroutine
    end interface

The difference compared to the previously used interface is that the first argument now is of type ESMF_DynamicMaskElementR8R8R8V. This type is declared as follows:

    type ESMF_DynamicMaskElementR8R8R8V
      real(ESMF_KIND_R8), pointer       :: dstElement(:)
      real(ESMF_KIND_R8), allocatable   :: factor(:)
      type(ESMF_PtrR8D1), allocatable   :: srcElement(:)
    end type

Here size(dstElement) for every element in dynMaskList is identical to the vector size, i.e. the number of undistributed elements to be handled. The same is true for size(srcElement(j)%ptr)), for every element j of the interpolation stencile.

  call ESMF_DynamicMaskSetR8R8R8V(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValue, &
    dynamicDstMaskValue=-2.d0, &
    dynamicMaskRoutine=simpleHandleAllProcV, &
    handleAllElements=.true., &
    rc=rc)

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, zeroregion=ESMF_REGION_EMPTY, &
    dynamicMask=dynamicMask, rc=rc)

Applying dynamic masking to source and destination fields of other typekind than R8 only requires that the correct DynamicMaskSet method is chosen. Here we create real(ESMF_KIND_R4) source and destination fields.

  srcField = ESMF_FieldCreate(srcGrid, ESMF_TYPEKIND_R4, rc=rc)

  dstField = ESMF_FieldCreate(dstGrid, ESMF_TYPEKIND_R4, rc=rc)

Computing a suitable RouteHandle is unchanged.

  srcTermProcessing=0

  call ESMF_FieldRegridStore(srcField=srcField, dstField=dstField, &
    srcTermProcessing=srcTermProcessing, routehandle=routehandle, rc=rc)

Now setting some source and destination elements to defined special values of the correct typekind.

  call ESMF_FieldGet(srcField, farrayPtr=farrayPtrR4, rc=rc)

  farrayPtrR4(lbound(farrayPtrR4,1)+3,lbound(farrayPtrR4,2)+3) = srcMaskValueR4

  call ESMF_FieldFill(dstField, dataFillScheme="const", const1=0.d0, rc=rc)

  call ESMF_FieldGet(dstField, farrayPtr=farrayPtrR4, rc=rc)

  farrayPtrR4(lbound(farrayPtrR4,1)+1,lbound(farrayPtrR4,2)+1) = dstMaskValueR4

Setting up the ESMF_DynamicMask object is practically the same as before, just that the correct typekind-triplet suffix for the DynamicMaskSet method must be selected, indicating that the destination data is of typekind R4, the factors are still of typekind R8, and the source data is of typekind R4.

  call ESMF_DynamicMaskSetR4R8R4(dynamicMask, &
    dynamicSrcMaskValue=srcMaskValueR4, &
    dynamicDstMaskValue=dstMaskValueR4, &
    dynamicMaskRoutine=simpleDynMaskProcR4R8R4, &
    rc=rc)

Finally calling into ESMF_FieldRegrid() with the dynamicMask object is unchanged.

  call ESMF_FieldRegrid(srcField=srcField, dstField=dstField, &
    routehandle=routehandle, zeroregion=ESMF_REGION_EMPTY, &
    dynamicMask=dynamicMask, rc=rc)

37.3 Restrictions and Future Work

37.4 Design and Implementation Notes

Internally all route-based communication calls are implemented as sparse matrix multiplications. The precompute step for all of the supported communication methods can be broken up into three steps:

  1. Construction of the sparse matrix for the specific communication method.
  2. Generation of the communication pattern according to the sparse matrix.
  3. Encoding of the communication pattern for each participating PET in form of an XXE stream.

37.5 Class API

37.5.1 ESMF_RouteHandleCreate - Create a new RouteHandle from RouteHandle


INTERFACE:

   ! Private name; call using ESMF_RouteHandleCreate()
   function ESMF_RouteHandleCreateRH(routehandle, &
     originPetList, targetPetList, rc)
RETURN VALUE:
     type(ESMF_RouteHandle) :: ESMF_RouteHandleCreateRH
ARGUMENTS:
     type(ESMF_RouteHandle), intent(in)            :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(in),  optional :: originPetList(:)
     integer,                intent(in),  optional :: targetPetList(:)
     integer,                intent(out), optional :: rc
DESCRIPTION:

Create a new ESMF_RouteHandle object from and existing RouteHandle. The new RouteHandle can be created to function on a different petList than the incoming RouteHandle.

The arguments are:

routehandle
The RouteHandle object to be duplicated.
[originPetList]
The petList on which the incoming routehandle is defined to operate. If present, then targetPetList must also be present and of the same size. The petLists are used to map origin PETs to target PETs. By convention the petLists are constructed to first list the PETs of the source component, followed by the PETs of the destination component. Defaults, to the petList of the current component context, meaning that the PETs in the RouteHandle are not modified.
[targetPetList]
The petList on which the newly created RouteHandle is defined to operate. If present, then originPetList must also be present and of the same size. The petLists are used to map origin PETs to target PETs. By convention the petLists are constructed to first list the PETs of the source component, followed by the PETs of the destination component. Defaults, to the petList of the current component context, meaning that the PETs in the RouteHandle are not modified.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.2 ESMF_RouteHandleCreate - Create a new RouteHandle from file


INTERFACE:

   ! Private name; call using ESMF_RouteHandleCreate()
   function ESMF_RouteHandleCreateFile(fileName, rc)
RETURN VALUE:
     type(ESMF_RouteHandle) :: ESMF_RouteHandleCreateFile
ARGUMENTS:
     character(*),           intent(in)            :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
DESCRIPTION:

Create a new ESMF_RouteHandle object from a file. This method must be called from a VM context that holds exactly as many PETs as were used when generating the file.

The arguments are:

fileName
The name of the RouteHandle file to be read in.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.3 ESMF_RouteHandleDestroy - Release resources associated with a RouteHandle


INTERFACE:

   subroutine ESMF_RouteHandleDestroy(routehandle, &
     noGarbage, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)          :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     logical,                intent(in),   optional :: noGarbage
     integer,                intent(out),  optional :: rc
DESCRIPTION:

Destroys an ESMF_RouteHandle, releasing the resources associated with the object.

The arguments are:

routehandle
The ESMF_RouteHandle to be destroyed.
[noGarbage]
If set to .TRUE. the object will be fully destroyed and removed from the ESMF garbage collection system. Note however that under this condition ESMF cannot protect against accessing the destroyed object through dangling aliases - a situation which may lead to hard to debug application crashes.

It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.

[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.4 ESMF_RouteHandleGet - Get values from a RouteHandle


INTERFACE:

   ! Private name; call using ESMF_RouteHandleGet()
   subroutine ESMF_RouteHandleGetP(routehandle, name, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(in)            :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(len=*),       intent(out), optional :: name
     integer,                intent(out), optional :: rc
DESCRIPTION:

Returns information about an ESMF_RouteHandle.

The arguments are:

routehandle
ESMF_RouteHandle to be queried.
[name]
Name of the RouteHandle object.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.5 ESMF_RouteHandleIsCreated - Check whether a RouteHandle object has been created


INTERFACE:

   function ESMF_RouteHandleIsCreated(routehandle, rc)
RETURN VALUE:
     logical :: ESMF_RouteHandleIsCreated
ARGUMENTS:
     type(ESMF_RouteHandle), intent(in)            :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
DESCRIPTION:

Return .true. if the routehandle has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..

The arguments are:

routehandle
ESMF_RouteHandle queried.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.6 ESMF_RouteHandlePrint - Print the contents of a RouteHandle


INTERFACE:

   subroutine ESMF_RouteHandlePrint(routehandle, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(in)            :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
DESCRIPTION:

Print information about an ESMF_RouteHandle.

The arguments are:

routehandle
ESMF_RouteHandle to print contents of.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.7 ESMF_RouteHandleSet - Set values in a RouteHandle


INTERFACE:

   ! Private name; call using ESMF_RouteHandleSet()
   subroutine ESMF_RouteHandleSetP(routehandle, name, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)         :: routehandle
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     character(len = *),     intent(in),  optional :: name
     integer,                intent(out), optional :: rc
DESCRIPTION:

Set an ESMF_RouteHandle attribute with the given value.

The arguments are:

routehandle
ESMF_RouteHandle to be modified.
[name]
The RouteHandle name.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

37.5.8 ESMF_RouteHandleWrite - Write the RouteHandle to file


INTERFACE:

   subroutine ESMF_RouteHandleWrite(routehandle, fileName, rc)
ARGUMENTS:
     type(ESMF_RouteHandle), intent(inout)         :: routehandle
     character(*),           intent(in)            :: fileName
 -- The following arguments require argument keyword syntax (e.g. rc=rc). --
     integer,                intent(out), optional :: rc
DESCRIPTION:

Write the RouteHandle to file. The generated file can then be used to re-create the same RouteHandle, on the same number of PETs, using the ESMF_RouteHandleCreate(fileName=...) method.

The arguments are:

routehandle
The ESMF_RouteHandle to be written.
fileName
The name of the output file to which the RouteHandle is written.
[rc]
Return code; equals ESMF_SUCCESS if there are no errors.

38 I/O Capability

38.1 Description

The ESMF I/O provides a unified interface for input and output of high level ESMF objects such as Fields. ESMF I/O capability is integrated with third-party software such as Parallel I/O (PIO) to read and write Fortran array data in NetCDF format, and JSON for Modern C++ Library to read Info attribute data in JSON format. Other file I/O functionalities, such as writing of error and log messages, input of configuration parameters from an ASCII file, and lower-level I/O utilities are covered in different sections of this document. See the Log Class 49.1, the Config Class 47.1, and the Fortran I/O Utilities, 53.1 respectively.


38.2 Data I/O

ESMF provides interfaces for high performance, parallel I/O using ESMF data objects such as Arrays and Fields. Currently ESMF only supports I/O of NetCDF files. The current ESMF implementation relies on the Parallel I/O (PIO) library developed as a collaboration between NCAR and DOE laboratories. PIO is built as part of the ESMF build when the environment variable ESMF_PIO is set to "internal", or is linked against when ESMF_PIO is set to "external"; by default ESMF_PIO is not set (which results in using the internal PIO if other aspects of the ESMF build configuration allow it). When PIO is built with ESMF, the ESMF methods internally call the PIO interfaces. When ESMF is not built with PIO, the ESMF methods are non-operable (no-op) stubs that simply return with a return code of ESMF_RC_LIB_NOT_PRESENT. Details about the environment variables can be found in ESMF User Guide, "Building and Installing the ESMF", "Third Party Libraries".

The following methods support parallel data I/O using PIO:

ESMF_FieldBundleRead(), section 25.5.20.
ESMF_FieldBundleWrite(), section 25.5.40.
ESMF_FieldRead(), section 26.6.62.
ESMF_FieldWrite(), section 26.6.84.
ESMF_ArrayBundleRead(), section 27.5.16.
ESMF_ArrayBundleWrite(), section 27.5.27.
ESMF_ArrayRead(), section 28.5.27.
ESMF_ArrayWrite(), section 28.5.47.

38.3 Data formats

The only format currently supported is NetCDF. The environment variables ESMF_NETCDF and/or ESMF_PNETCDF must be set to enable this NetCDF-based I/O. Details about the environment variables can be found in ESMF User Guide, "Building and Installing the ESMF", "Third Party Libraries".

NetCDF
Network Common Data Form (NetCDF) is an interface for array-oriented data access. The NetCDF library provides an implementation of the interface. It also defines a machine-independent format for representing scientific data. Together, the interface, library, and format support the creation, access, and sharing of scientific data. The NetCDF software was developed at the Unidata Program Center in Boulder, Colorado. See [16]. In geoscience, NetCDF can be naturally used to represent fields defined on logically rectangular grids. NetCDF use in geosciences is specified by CF conventions mentioned above [15].

To the extent that data on unstructured grids (or even observations) can be represented as one-dimensional arrays, NetCDF can also be used to store these data. However, it does not provide a high-level abstraction for this type of data.

38.4 Restrictions and Future Work

  1. Limited data formats supported. Currently a small fraction of the anticipated data formats is implemented by ESMF. The data I/O uses NetCDF format, and ESMF Info I/O uses JSON format. Different libraries are employed for these different formats. In future development, a more centralized I/O technique will likely be defined to provide efficient utilities with a set of standard APIs that will allow manipulation of multiple standard formats. Also, the ability to automatically detect file formats at runtime will be developed.

  2. Some limitations with multi-tile I/O. There are a few limitations when doing I/O on multi-tile Arrays and Fields (e.g., a cubed sphere grid represented as a six-tile grid): This I/O requires at least as many PETs as there are tiles, and for I/O of ArrayBundles and FieldBundles, all Arrays / Fields in the bundle must contain the same number of tiles.

  3. Replicated dimensions. I/O of Arrays / Fields with replicated dimensions (section 28.2.12) is only partially working. In most situations, replicated dimensions appear as dimensions in the output file; ideally, these replicated dimensions would be removed in the output file, and we plan to make that change in the future. Furthermore, slices of the replicated dimensions other than the first can have garbage values in the output file. In addition, there is an inconsistency when outputting Arrays / Fields that have a decomposition with more than one DE per PET: in this case, replicated dimensions are removed in the output file. Finally, I/O cannot be performed on multi-tile Arrays / Fields with replicated dimensions.

38.5 Design and Implementation Notes

For data I/O, the ESMF I/O capability relies on the PIO and NetCDF libraries, and optionally the PNetCDF library. For Info attribute I/O, the ESMF I/O capability uses the JSON for Modern C++ library to perform reading of JSON files. PIO and JSON for Modern C++ are included with the ESMF distribution; the other libraries must be installed on the machine of interest.

esmf_support@ucar.edu