The ESMF infrastructure data classes are part of the framework's hierarchy of structures for handling Earth system model data and metadata on parallel platforms. The hierarchy is in complexity; the simplest data class in the infrastructure represents a distributed data array and the most complex data class represents a bundle of physical fields that are discretized on the same grid. Data class methods are called both from user-written code and from other classes internal to the framework.
Data classes are distributed over DEs, or Decomposition Elements. A DE represents a piece of a decomposition. A DELayout is a collection of DEs with some associated connectivity that describes a specific distribution. For example, the distribution of a grid divided into four segments in the x-dimension would be expressed in ESMF as a DELayout with four DEs lying along an x-axis. This abstract concept enables a data decomposition to be defined in terms of threads, MPI processes, virtual decomposition elements, or combinations of these without changes to user code. This is a primary strategy for ensuring optimal performance and portability for codes using ESMF for communications.
ESMF data classes are useful because they provide a standard, convenient way for developers to collect together information related to model or observational data. The information assembled in a data class includes a data pointer, a set of attributes (e.g. units, although attributes can also be user-defined), and a description of an associated grid. The same set of information within an ESMF data object can be used by the framework to arrange intercomponent data transfers, to perform I/O, for communications such as gathers and scatters, for simplification of interfaces within user code, for debugging, and for other functions. This unifies and organizes codes overall so that the user need not define different representations of metadata for the same field for I/O and for component coupling.
Since it is critical that users be able to introduce ESMF into their codes easily and incrementally, ESMF data classes can be created based on native Fortran pointers. Likewise, there are methods for retrieving native Fortran pointers from within ESMF data objects. This allows the user to perform allocations using ESMF, and to retrieve Fortran arrays later for optimized model calculations. The ESMF data classes do not have associated differential operators or other mathematical methods.
For flexibility, it is not necessary to build an ESMF data object all at once. For example, it's possible to create a field but to defer allocation of the associated field data until a later time.
|
Key Features |
| Hierarchy of data structures designed specifically for the Earth system domain and high performance, parallel computing. |
| Multi-use ESMF structures simplify user code overall. |
| Data objects support incremental construction and deferred allocation. |
| Native Fortran arrays can be associated with or retrieved from ESMF data objects, for ease of adoption, convenience, and performance. |
| A variety of operations are provided for manipulating data in data objects such as regridding, redistribution, halo communication, and sparse matrix multiply. |
The main classes that are used for model and observational data manipulation are as follows:
Data elements in Arrays are partitioned into categories defined by the role the data element plays in distributed halo operations. Haloing - sometimes called ghosting - is the practice of copying portions of array data to multiple memory locations to ensure that data dependencies can be satisfied quickly when performing a calculation. ESMF Arrays contain an exclusive domain, which contains data elements updated exclusively and definitively by a given DE; a computational domain, which contains all data elements with values that are updated by the DE in computations; and a total domain, which includes both the computational domain and data elements from other DEs which may be read but are not updated in computations.
FieldBundle objects contain methods for setting and retrieving constituent fields, regridding, data I/O, and reordering of data in memory.
Bit-for-bit reproducibility is at the core of the regression testing schemes of many scientific model codes. The bit-for-bit requirement makes it easy to compare the numerical results of simulation runs using standard binary diff tools.
For the most part, ESMF methods do not modify user data numerically, and thus have no effect on the bit-for-bit characteristics of the model code. The exceptions are the regrid weight generation and the sparse matrix multiplication.
In the case of the regrid weight generation, user data is used to produce interpolation weights following specific numerical schemes. The bit-for-bit reproducibility of the generated weights depends on the implementation details. Section 24.2 provides more details about the bit-for-bit considerations with respect to the regrid weights generated by ESMF.
In the case of the sparse matrix multiplication, which is the typical method that is used to apply the regrid weights, user data is directly manipulated by ESMF. In order to help users with the implementation of their bit-for-bit requirements, while also considering the associated performance impact, the ESMF sparse matrix implementation provides three levels of bit-for-bit support. The strictest level ensures that the numerical results are bit-for-bit identical, even when executing across different numbers of PETs. In the relaxed level, bit-for-bit reproducibility is guaranteed when running across an unchanged number of PETs. The lowest level makes no guarantees about bit-for-bit reproducibility, however, it provides the greatest performance potential for those cases where numerical round-off differences are acceptable. An in-depth discussion of bit-for-bit reproducibility, and the performance aspects of route-based communication methods, such the sparse matrix multiplication, is given in section 36.2.1.
This section describes the regridding methods provided by ESMF. Regridding, also called remapping or interpolation, is the process of changing the grid that underlies data values while preserving qualities of the original data. Different kinds of transformations are appropriate for different problems. Regridding may be needed when communicating data between Earth system model components such as land and atmosphere, or between different data sets to support operations such as visualization.
Regridding can be broken into two stages. The first stage is generation of an interpolation weight matrix that describes how points in the source grid contribute to points in the destination grid. The second stage is the multiplication of values on the source grid by the interpolation weight matrix to produce values on the destination grid. This occurs through a parallel sparse matrix multiplication.
There are two options for accessing ESMF regridding functionality: offline and integrated. Offline regridding is a process whereby interpolation weights are generated by a separate ESMF application, not within the user code. The ESMF offline regridding application also only generates the interpolation matrix, the user is responsible for reading in this matrix and doing the actual interpolation (multiplication by the sparse matrix) in their code. Please see Section 12 for a description of the offline regridding application and the options it supports. For user convenience, there is also a method interface to the offline regrid application functionality which is described in Section 24.3.1. In contrast to offline regridding, integrated regridding is a process whereby interpolation weights are generated via subroutine calls during the execution of the user's code. In addition to generating the weights, integrated regridding can also produce a RouteHandle (described in Section 36.1) which allows the user to perform the parallel sparse matrix multiplication using ESMF methods. In other words, ESMF integrated regridding allows a user to perform the whole process of interpolation within their code.
To see what types of grids and other options are supported in the two types of regridding and their testing status, please see the ESMF Regridding Status webpage for this version of ESMF. Figure 24.2 shows a comparison of different regrid interfaces and where they can be found in the documentation.
The rest of this section further describes various options available in ESMF regridding. It starts with a description of the five interpolation methods available in regridding. Next, it describes how to remove (mask) pieces of a grid from the regridding. Following that, it describes options if a destination point can't be regridding. After that, is a section on spherical grids and controlling what happens at the pole. Finally, there is a section on troubleshooting regridding.
| ||||||||||||||||||||||||||||||||||||
In 2D, ESMF supports bilinear regridding between any combination of the following:
In 3D, ESMF supports bilinear regridding between any combination of the following:
Restrictions:
To use the bilinear method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates.
Patch (or higher-order) interpolation is the ESMF version of a technique called ``patch recovery'' commonly used in finite element modeling [19] [15]. It typically results in better approximations to values and derivatives when compared to bilinear interpolation. Patch interpolation works by constructing multiple polynomial patches to represent the data in a source cell. For 2D grids, these polynomials are currently 2nd degree 2D polynomials. One patch is constructed for each corner of the source cell, and the patch is constructed by doing a least squares fit through the data in the cells surrounding the corner. The interpolated value at the destination point is then a weighted average of the values of the patches at that point. The patch method has a larger stencil than the bilinear, for this reason the patch weight matrix can be correspondingly larger than the bilinear matrix (e.g. for a quadrilateral grid the patch matrix is around 4x the size of the bilinear matrix). This can be an issue when performing a regrid operation close to the memory limit on a machine.
In 2D, ESMF supports patch regridding between any combination of the following:
Patch regridding is currently not supported in 3D.
Restrictions:
To use the patch method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates.
In 2D, ESMF supports nearest source to destination regridding between any combination of the following:
In 3D, ESMF supports nearest source to destination regridding between any combination of the following:
Restrictions:
NONE
To use the nearest source to destination method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates.
In 2D, ESMF supports nearest destination to source regridding between any combination of the following:
In 3D, ESMF supports nearest destination to source regridding between any combination of the following:
Restrictions:
NONE
To use the nearest destination to source method the user may create their Fields on any stagger location (e.g. ESMF_STAGGERLOC_CENTER) for a Grid, or any Mesh location (e.g. ESMF_MESHLOC_NODE) for a Mesh. For either a Grid or a Mesh, the location upon which the Field is built must contain coordinates.
The user should be aware that because of the conservation relationship between the source and destination fields, the more the total source area differs from the total destination area the more the values of the source field will differ from the corresponding values of the destination field, likely giving a higher interpolation error. It is best to have the total source and destination areas the same (this will automatically be true if no user areas are specified). For source and destination grids that only partially overlap, the overlapping regions of the source and destination should be the same.
Note that for grids on a sphere the conservative interpolation assumes great circle edges to cells. This means that the edges of a cell won't necessarily be the same as a straight line in latitude longitude. For small edges, this difference will be small, but for long edges it could be significant. This means if the user expects cell edges as straight lines in latitude longitude space, they should avoid using one large cell with long edges to compute an average over a region (e.g. over an ocean basin). The user should also avoid using cells that contain one edge that runs half way or more around the earth, because the regrid weight calculation assumes the edge follows the shorter great circle path. Also, there isn't a unique great circle edge defined between points on the exact opposite side of the earth from one another (antipodal points). However, the user can work around both of these problem by breaking the long edge into two smaller edges by inserting an extra node, or by breaking the large target grid cells into two or more smaller grid cells. This allows the application to resolve the ambiguity in edge direction.
It is important to note that by default (i.e. using destination area normalization) conservative regridding doesn't normalize the interpolation weights by the destination fraction. This means that for a destination grid which only partially overlaps the source grid the destination field that is output from the regrid operation should be divided by the corresponding destination fraction to yield the true interpolated values for cells which are only partially covered by the source grid. The fraction also needs to be included when computing the total source and destination integrals. (To include the fraction in the conservative weights, the user can specify the fraction area normalization type. This can be done by specifying normType=ESMF_NORMTYPE_FRACAREA when invoking ESMF_FieldRegridStore().)
For weights generated using destination area normalization (either by not specifying any normalization type or by specifying normType=ESMF_NORMTYPE_DSTAREA), if a destination field extends outside the unmasked source field, then the values of the cells which extend partway outside the unmasked source field are decreased by the fraction they extend outside. To correct these values, the destination field (dst_field) resulting from the ESMF_FieldRegrid() call can be divided by the destination fraction dst_frac from the ESMF_FieldRegridStore() call. The following pseudocode demonstrates how to do this:
for each destination element i
if (dst_frac(i) not equal to 0.0) then
dst_field(i)=dst_field(i)/dst_frac(i)
end if
end for
For weights generated using destination area normalization (either by not specifying any normalization type or by specifying normType=ESMF_NORMTYPE_DSTAREA), the following pseudo-code shows how to compute the total destination integral (dst_total) given the destination field values (dst_field) resulting from the ESMF_FieldRegrid() call, the destination area (dst_area) from the ESMF_FieldRegridGetArea() call, and the destination fraction (dst_frac) from the ESMF_FieldRegridStore() call. As shown in the previous paragraph, it also shows how to adjust the destination field (dst_field) resulting from the ESMF_FieldRegrid() call by the fraction (dst_frac) from the ESMF_FieldRegridStore() call:
dst_total=0.0
for each destination element i
if (dst_frac(i) not equal to 0.0) then
dst_total=dst_total+dst_field(i)*dst_area(i)
dst_field(i)=dst_field(i)/dst_frac(i)
! If mass computed here after dst_field adjust, would need to be:
! dst_total=dst_total+dst_field(i)*dst_area(i)*dst_frac(i)
end if
end for
For weights generated using fraction area normalization (by specifying normType=ESMF_NORMTYPE_FRACAREA), no adjustment of the destination field is necessary. The following pseudo-code shows how to compute the total destination integral (dst_total) given the destination field values (dst_field) resulting from the ESMF_FieldRegrid() call, the destination area (dst_area) from the ESMF_FieldRegridGetArea() call, and the destination fraction (dst_frac) from the ESMF_FieldRegridStore() call:
dst_total=0.0
for each destination element i
dst_total=dst_total+dst_field(i)*dst_area(i)*dst_frac(i)
end for
For both normalization types, the following pseudo-code shows how to compute the total source integral (src_total) given the source field values (src_field), the source area (src_area) from the ESMF_FieldRegridGetArea() call, and the source fraction (src_frac) from the ESMF_FieldRegridStore() call:
src_total=0.0
for each source element i
src_total=src_total+src_field(i)*src_area(i)*src_frac(i)
end for
In 2D, ESMF supports conservative regridding between any combination of the following:
In 3D, ESMF supports conservative regridding between any combination of the following:
Restrictions:
To use the conservative method the user should create their Fields on the center stagger location (ESMF_STAGGERLOC_CENTER in 2D or ESMF_STAGGERLOC_CENTER_VCENTER in 3D) for Grids or the element location (ESMF_MESHLOC_ELEMENT) for Meshes. For Grids, the corner stagger location (ESMF_STAGGERLOC_CORNER in 2D or ESMF_STAGGERLOC_CORNER_VFACE in 3D) must contain coordinates describing the outer perimeter of the Grid cells.
The user may mask out points in the source Field or destination Field or both. To do masking the user sets mask information in the Grid (see 31.3.14), Mesh (see 33.3.8), or LocStream (see 32.2.2) upon which the Fields passed into the ESMF_FieldRegridStore() call are built. The srcMaskValues and dstMaskValues arguments to that call can then be used to specify which values in that mask information indicate that a location should be masked out. For example, if dstMaskValues is set to (/1,2/), then any location that has a value of 1 or 2 in the mask information of the Grid, Mesh or LocStream upon which the destination Field is built will be masked out.
Masking behavior differs slightly between regridding methods. For non-conservative regridding methods (e.g. bilinear or high-order patch), masking is done on points. For these methods, masking a destination point means that that point won't participate in regridding (e.g. won't be interpolated to). For these methods, masking a source point means that the entire source cell using that point is masked out. In other words, if any corner point making up a source cell is masked then the cell is masked. For conservative regridding methods (e.g. first-order conservative) masking is done on cells. Masking a destination cell means that the cell won't participate in regridding (e.g. won't be interpolated to). Similarly, masking a source cell means that the cell won't participate in regridding (e.g. won't be interpolated from). For any type of interpolation method (conservative or non-conservative) the masking is set on the location upon which the Fields passed into the regridding call are built. For example, if Fields built on ESMF_STAGGERLOC_CENTER are passed into the ESMF_FieldRegridStore() call then the masking should also be set on ESMF_STAGGERLOC_CENTER.
For Grids on a sphere, the regridding occurs in 3D Cartesian to avoid problems with periodicity and with the pole singularity. This library supports four options for handling the pole region (i.e. the empty area above the top row of the source grid or below the bottom row of the source grid). Note that all of these pole options currently only work for the Fields build on the Grid class and not for those built on the Mesh class.
The first option is to leave the pole region empty (polemethod=ESMF_POLEMETHOD_NONE), in this case if a destination point lies above or below the top row of the source grid, it will fail to map, yielding an error (unless unmappedaction=ESMF_UNMAPPEDACTION_IGNORE is specified).
With the next two options (ESMF_POLEMETHOD_ALLAVG and ESMF_POLEMETHOD_NPNTAVG), the pole region is handled by constructing an artificial pole in the center of the top and bottom row of grid points and then filling in the region from this pole to the edges of the source grid with triangles. The pole is located at the average of the position of the points surrounding it, but moved outward to be at the same radius as the rest of the points in the grid. The difference between the two artificial pole options is what value is used at the pole. The option (polemethod=ESMF_POLEMETHOD_ALLAVG) sets the value at the pole to be the average of the values of all of the grid points surrounding the pole. The option (polemethod=ESMF_POLEMETHOD_NPNTAVG) allows the user to choose a number N from 1 to the number of source grid points around the pole. The value N is set via the argument regridPoleNPnts. For each destination point, the value at the pole is then the average of the N source points surrounding that destination point.
The last option (polemethod=ESMF_POLEMETHOD_TEETH) does not construct an artificial pole, instead the pole region is covered by connecting points across the top and bottom row of the source Grid into triangles. As this makes the top and bottom of the source sphere flat, for a big enough difference between the size of the source and destination pole regions, this can still result in unmapped destination points. Only pole option ESMF_POLEMETHOD_NONE is currently supported with the conservative interpolation method (regridmethod=ESMF_REGRIDMETHOD_CONSERVE) and with the nearest neighbor interpolation options (regridmethod=ESMF_REGRIDMETHOD_NEAREST_STOD and regridmethod=ESMF_REGRIDMETHOD_NEAREST_DTOS).
Another variation in the regridding supported with spherical grids is line type. This is controlled in the ESMF_FieldRegridStore() method by the lineType argument. This argument allows the user to select the path of the line which connects two points on a sphere surface. This in turn controls the path along which distances are calculated and the shape of the edges that make up a cell. Both of these quantities can influence how interpolation weights are calculated, for example in bilinear interpolation the distances are used to calculate the weights and the cell edges are used to determine to which source cell a destination point should be mapped.
ESMF currently supports two line types: ESMF_LINETYPE_CART and ESMF_LINETYPE_GREAT_CIRCLE. The ESMF_LINETYPE_CART option specifies that the line between two points follows a straight path through the 3D Cartesian space in which the sphere is embedded. Distances are measured along this 3D Cartesian line. Under this option cells are approximated by planes in 3D space, and their boundaries are 3D Cartesian lines between their corner points. The ESMF_LINETYPE_GREAT_CIRCLE option specifies that the line between two points follows a great circle path along the sphere surface. (A great circle is the shortest path between two points on a sphere.) Distances are measured along the great circle path. Under this option cells are on the sphere surface, and their boundaries are great circle paths between their corner points.
Figure 24.2.8 shows which line types are supported for each regrid method as well as the defaults (indicated by *).
The below is a list of problems users commonly encounter with regridding and potential solutions. This is by no means an exhaustive list, so if none of these problems fit your case, or if the solutions don't fix your problem, please feel free to email esmf support (esmf_support@list.woc.noaa.gov).
Problem: Regridding is too slow.
Possible Cause: The ESMF_FieldRegridStore() method is called more than is necessary.
The ESMF_FieldRegridStore() operation is a complex one and can be
relatively slow for some cases (large Grids, 3D grids, etc.)
Solution: Reduce the number of ESMF_FieldRegridStore() calls to the minimum necessary. The routeHandle generated by the ESMF_FieldRegridStore() call depends on only four factors: the stagger locations that the input Fields are created on, the coordinates in the Grids the input Fields are built on at those stagger locations, the padding of the input Fields (specified by the totalWidth arguments in FieldCreate) and the size of the tensor dimensions in the input Fields (specified by the ungridded arguments in FieldCreate). For any pair of Fields which share these attributes with the Fields used in the ESMF_FieldRegridStore call the same routeHandle can be used. Note, that the data in the Fields does NOT matter, the same routeHandle can be used no matter how the data in the Fields changes.
In particular:
Problem: Distortions in destination Field at periodic boundary.
Possible Cause: The Grid overlaps itself. With a periodic Grid, the regrid system expects the first point to not be a repeat of the last point. In other words, regrid constructs its own connection and overlap between the first and last points of the periodic dimension and so the Grid doesn't need to contain these. If the Grid does, then this can cause problems.
Solution: Define the Grid so that it doesn't contain the overlap point. This typically means simply making the Grid one point smaller in the periodic dimension. If a Field constructed on the Grid needs to contain these overlap points then the user can use the totalWidth arguments to include this extra padding in the Field. Note, however, that the regrid won't update these extra points, so the user will have to do a copy to fill the points in the overlap region in the Field.
The ESMF regrid weight calculation functionality has been designed to enable it to support a wide range of grid and interpolation types without needing to support each individual combination of source grid type, destination grid type, and interpolation method. To avoid the quadratic growth of the number of pairs of grid types, all grids are converted to a common internal format and the regrid weight calculation is performed on that format. This vastly reduces the variety of grids that need to be supported in the weight calculations for each interpolation method. It also has the added benefit of making it straightforward to add new grid types and to allow them to work with all the existing grid types. To hook into the existing weight calculation code, the new type just needs to be converted to the internal format.
The internal grid format used by the ESMF regrid weight calculation is a finite element unstructured mesh. This was chosen because it was the most general format and all the others could be converted to it. The ESMF finite element unstructured mesh (ESMF FEM) is similar in some respects to the SIERRA [11] package developed at Sandia National Laboratory. The ESMF code relies on some of the same underlying toolkits (e.g. Zoltan [9] library for calculating mesh partitions) and adds a layer on top that allows the calculation of regrid weights and some mesh operations (e.g. mesh redistribution) that ESMF needs. The ESMF FEM has similar notions to SIERRA about the basic structure of the mesh entities, fields, iteration and a similar notion of parallel distribution.
Currently we use the ESMF FEM internal mesh to hold the structure of our Mesh class and in our regrid weight calculation. The parts of the internal FEM code that are used/tested by ESMF are the following:
INTERFACE:
! Private name; call using ESMF_RegridWeightGen()
subroutine ESMF_RegridWeightGenFile(srcFile, dstFile, weightFile, &
regridmethod, polemethod, regridPoleNPnts, lineType, normType, &
unmappedaction, ignoreDegenerate, srcFileType, dstFileType, &
srcRegionalFlag, dstRegionalFlag, srcMeshname, dstMeshname, &
srcMissingvalueFlag, srcMissingvalueVar, &
dstMissingvalueFlag, dstMissingvalueVar, &
useSrcCoordFlag, srcCoordinateVars, &
useDstCoordFlag, dstCoordinateVars, &
useSrcCornerFlag, useDstCornerFlag, &
useUserAreaFlag, largefileFlag, &
netcdf4fileFlag, verboseFlag, rc)
ARGUMENTS:
character(len=*), intent(in) :: srcFile character(len=*), intent(in) :: dstFile character(len=*), intent(in) :: weightFile -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_RegridMethod_Flag), intent(in), optional :: regridmethod type(ESMF_PoleMethod_Flag), intent(in), optional :: polemethod integer, intent(in), optional :: regridPoleNPnts type(ESMF_LineType_Flag), intent(in), optional :: lineType type(ESMF_NormType_Flag), intent(in), optional :: normType type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction logical, intent(in), optional :: ignoreDegenerate type(ESMF_FileFormat_Flag), intent(in), optional :: srcFileType type(ESMF_FileFormat_Flag), intent(in), optional :: dstFileType logical, intent(in), optional :: srcRegionalFlag logical, intent(in), optional :: dstRegionalFlag character(len=*), intent(in), optional :: srcMeshname character(len=*), intent(in), optional :: dstMeshname logical, intent(in), optional :: srcMissingValueFlag character(len=*), intent(in), optional :: srcMissingValueVar logical, intent(in), optional :: dstMissingValueFlag character(len=*), intent(in), optional :: dstMissingValueVar logical, intent(in), optional :: useSrcCoordFlag character(len=*), intent(in), optional :: srcCoordinateVars(:) logical, intent(in), optional :: useDstCoordFlag character(len=*), intent(in), optional :: dstCoordinateVars(:) logical, intent(in), optional :: useSrcCornerFlag logical, intent(in), optional :: useDstCornerFlag logical, intent(in), optional :: useUserAreaFlag logical, intent(in), optional :: largefileFlag logical, intent(in), optional :: netcdf4fileFlag logical, intent(in), optional :: verboseFlag integer, intent(out), optional :: rcDESCRIPTION:
This subroutine provides the same function as the ESMF_RegridWeightGen application described in Section 12. It takes two grid files in NetCDF format and writes out an interpolation weight file also in NetCDF format. The interpolation weights can be generated with the bilinear (24.2.1), higher-order patch (24.2.2), or first order conservative (24.2.5) methods. The grid files can be in one of the following four formats:
The arguments are:
INTERFACE:
! Private name; call using ESMF_RegridWeightGen()
subroutine ESMF_RegridWeightGenDG(srcFile, dstFile, regridRouteHandle, &
srcElementDistgrid, dstElementDistgrid, &
srcNodalDistgrid, dstNodalDistgrid, &
weightFile, regridmethod, lineType, normType, unmappedaction, &
ignoreDegenerate, useUserAreaFlag, &
largefileFlag, netcdf4fileFlag, verboseFlag, rc)
ARGUMENTS:
character(len=*), intent(in) :: srcFile character(len=*), intent(in) :: dstFile type(ESMF_RouteHandle), intent(out) :: regridRouteHandle -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DistGrid), intent(in), optional :: srcElementDistgrid type(ESMF_DistGrid), intent(in), optional :: dstElementDistgrid character(len=*), intent(in), optional :: weightFile type(ESMF_DistGrid), intent(in), optional :: srcNodalDistgrid type(ESMF_DistGrid), intent(in), optional :: dstNodalDistgrid type(ESMF_RegridMethod_Flag), intent(in), optional :: regridmethod type(ESMF_LineType_Flag), intent(in), optional :: lineType type(ESMF_NormType_Flag), intent(in), optional :: normType type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction logical, intent(in), optional :: ignoreDegenerate logical, intent(in), optional :: useUserAreaFlag logical, intent(in), optional :: largefileFlag logical, intent(in), optional :: netcdf4fileFlag logical, intent(in), optional :: verboseFlag integer, intent(out), optional :: rcDESCRIPTION:
This subroutine does online regridding weight generation from files with user specified distribution. The main differences between this API with the one in 24.3.1 are as follows:
The arguments are:
INTERFACE:
subroutine ESMF_FileRegrid(srcFile, dstFile, srcVarName, dstVarName, &
dstCoordVars, regridmethod, polemethod, regridPoleNPnts, &
unmappedaction, ignoreDegenerate, srcRegionalFlag, dstRegionalFlag, &
verboseFlag, rc)
ARGUMENTS:
character(len=*), intent(in) :: srcFile character(len=*), intent(in) :: dstFile character(len=*), intent(in) :: srcVarName character(len=*), intent(in) :: dstVarName -- The following arguments require argument keyword syntax (e.g. rc=rc). -- character(len=*), intent(in), optional :: dstCoordVars type(ESMF_RegridMethod_Flag), intent(in), optional :: regridmethod type(ESMF_PoleMethod_Flag), intent(in), optional :: polemethod integer, intent(in), optional :: regridPoleNPnts type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction logical, intent(in), optional :: ignoreDegenerate logical, intent(in), optional :: srcRegionalFlag logical, intent(in), optional :: dstRegionalFlag logical, intent(in), optional :: verboseFlag integer, intent(out), optional :: rcDESCRIPTION:
This subroutine provides the same function as the ESMF_Regrid application described in Section 13. It takes two grid files in NetCDF format and interpolate the variable defined in the source grid file to the destination variable using one of the ESMF supported regrid methods - bilinear (24.2.1), higher-order patch (24.2.2), first order conservative (24.2.5) or nearest neighbor methods. The grid files can be in one of the following two formats:
The arguments are:
elements. This limitation will be removed in future versions of ESMF by allowing 64-bit sequence indices.
A FieldBundle functions mainly as a convenient container for storing similar Fields. It represents ``bundles'' of Fields that are discretized on the same Grid, Mesh, LocStream, or XGrid and distributed in the same manner. The FieldBundle is an important data structure because it can be added to a State, which is used for sending and receiving data between Components.
In the common case where FieldBundle is built on top of a Grid, Fields within a FieldBundle may be located at different locations relative to the vertices of their common Grid. The Fields in a FieldBundle may be of different dimensions, as long as the Grid dimensions that are distributed are the same. For example, a surface Field on a distributed lat/lon Grid and a 3D Field with an added vertical dimension on the same distributed lat/lon Grid can be included in the same FieldBundle.
FieldBundles can be created and destroyed, can have Attributes added or retrieved, and can have Fields added, removed, replaced, or retrieved. Methods include queries that return information about the FieldBundle itself and about the Fields that it contains. The Fortran data pointer of a Field within a FieldBundle can be obtained by first retrieving the Field with a call to ESMF_FieldBundleGet(), and then using ESMF_FieldGet() to get the data.
In the future FieldBundles will serve as a mechanism for performance optimization. ESMF will take advantage of the similarities of the Fields within a FieldBundle to optimize collective communication, IO, and regridding. See Section 25.3 for a description of features that are scheduled for future work.
Examples of creating, accessing and destroying FieldBundles and their constituent Fields are provided in this section, along with some notes on FieldBundle methods.
!-------------------------------------------------------------------------
! ! Create several Fields and add them to a new FieldBundle.
grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/100,200/), &
regDecomp=(/2,2/), name="atmgrid", rc=rc)
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R8, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
field(1) = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
name="temperature", rc=rc)
field(2) = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
name="pressure", rc=rc)
field(3) = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
name="heat flux", rc=rc)
bundle1 = ESMF_FieldBundleCreate(fieldList=field(1:3), &
name="atmosphere data", rc=rc)
print *, "FieldBundle example 1 returned"
!-------------------------------------------------------------------------
! ! Create an empty FieldBundle and then add a single field to it.
simplefield = ESMF_FieldCreate(grid, arrayspec, &
staggerloc=ESMF_STAGGERLOC_CENTER, name="rh", rc=rc)
bundle2 = ESMF_FieldBundleCreate(name="time step 1", rc=rc)
call ESMF_FieldBundleAdd(bundle2, (/simplefield/), rc=rc)
call ESMF_FieldBundleGet(bundle2, fieldCount=fieldcount, rc=rc)
print *, "FieldBundle example 2 returned, fieldcount =", fieldcount
!-------------------------------------------------------------------------
! ! Create an empty FieldBundle and then add multiple fields to it.
bundle3 = ESMF_FieldBundleCreate(name="southern hemisphere", rc=rc)
call ESMF_FieldBundleAdd(bundle3, field(1:3), rc=rc)
call ESMF_FieldBundleGet(bundle3, fieldCount=fieldcount, rc=rc)
print *, "FieldBundle example 3 returned, fieldcount =", fieldcount
!-------------------------------------------------------------------------
! ! Get a Field back from a FieldBundle, first by name and then by index.
! ! Also get the FieldBundle name.
call ESMF_FieldBundleGet(bundle1, "pressure", field=returnedfield1, rc=rc)
call ESMF_FieldGet(returnedfield1, name=fname1, rc=rc)
call ESMF_FieldBundleGet(bundle1, 2, returnedfield2, rc=rc)
call ESMF_FieldGet(returnedfield2, name=fname2, rc=rc)
call ESMF_FieldBundleGet(bundle1, name=bname1, rc=rc)
print *, "FieldBundle example 4 returned, field names = ", &
trim(fname1), ", ", trim(fname2)
print *, "FieldBundle name = ", trim(bname1)
call ESMF_FieldBundleGet(bundle1, fieldList=r_fields, rc=rc)
do i = 1, 3
call ESMF_FieldGet(r_fields(i), name=fname1, rc=rc)
print *, fname1
enddo
call ESMF_FieldBundleGet(bundle1, fieldList=r_fields, &
itemorderflag=ESMF_ITEMORDER_ADDORDER, rc=rc)
do i = 1, 3
call ESMF_FieldGet(r_fields(i), name=fname1, rc=rc)
print *, fname1
enddo
The user must call ESMF_FieldBundleDestroy() before deleting any of the Fields it contains. Because Fields can be shared by multiple FieldBundles and States, they are not deleted by this call.
!-------------------------------------------------------------------------
call ESMF_FieldBundleDestroy(bundle1, rc=rc)
A user can use ESMF_FieldBundleRedist interface to redistribute data from source FieldBundle to destination FieldBundle. This interface is overloaded by type and kind; In the version of ESMF_FieldBundleRedist without factor argument, a default value of factor 1 is used.
In this example, we first create two FieldBundles, a source FieldBundle and a destination FieldBundle. Then we use ESMF_FieldBundleRedist to redistribute data from source FieldBundle to destination FieldBundle.
! perform redist
call ESMF_FieldBundleRedistStore(srcFieldBundle, dstFieldBundle, &
routehandle, rc=rc)
call ESMF_FieldBundleRedist(srcFieldBundle, dstFieldBundle, &
routehandle, rc=rc)
A user can use ESMF_FieldBundleSMM interface to perform SMM from source FieldBundle to destination FieldBundle. This interface is overloaded by type and kind;
In this example, we first create two FieldBundles, a source FieldBundle and a destination FieldBundle. Then we use ESMF_FieldBundleSMM to perform sparse matrix multiplication from source FieldBundle to destination FieldBundle.
The operation performed in this example is better illustrated in section 26.3.32.
Section 28.2.17 provides a detailed discussion of the sparse matrix mulitiplication operation implemented in ESMF.
call ESMF_VMGetCurrent(vm, rc=rc)
call ESMF_VMGet(vm, localPet=lpe, rc=rc)
! create distgrid and grid
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
regDecomp=(/4/), &
rc=rc)
grid = ESMF_GridCreate(distgrid=distgrid, &
gridEdgeLWidth=(/0/), gridEdgeUWidth=(/0/), &
name="grid", rc=rc)
call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_I4, rc=rc)
! create field bundles and fields
srcFieldBundle = ESMF_FieldBundleCreate(rc=rc)
dstFieldBundle = ESMF_FieldBundleCreate(rc=rc)
do i = 1, 3
srcField(i) = ESMF_FieldCreate(grid, arrayspec, &
totalLWidth=(/1/), totalUWidth=(/2/), &
rc=rc)
call ESMF_FieldGet(srcField(i), localDe=0, farrayPtr=srcfptr, rc=rc)
srcfptr = 1
call ESMF_FieldBundleAdd(srcFieldBundle, (/srcField(i)/), rc=rc)
dstField(i) = ESMF_FieldCreate(grid, arrayspec, &
totalLWidth=(/1/), totalUWidth=(/2/), &
rc=rc)
call ESMF_FieldGet(dstField(i), localDe=0, farrayPtr=dstfptr, rc=rc)
dstfptr = 0
call ESMF_FieldBundleAdd(dstFieldBundle, (/dstField(i)/), rc=rc)
enddo
! initialize factorList and factorIndexList
allocate(factorList(4))
allocate(factorIndexList(2,4))
factorList = (/1,2,3,4/)
factorIndexList(1,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
factorIndexList(2,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
call ESMF_FieldBundleSMMStore(srcFieldBundle, dstFieldBundle, &
routehandle, factorList, factorIndexList, rc=rc)
! perform smm
call ESMF_FieldBundleSMM(srcFieldBundle, dstFieldBundle, routehandle, &
rc=rc)
! release SMM route handle
call ESMF_FieldBundleSMMRelease(routehandle, rc=rc)
ESMF_FieldBundleHalo interface can be used to perform halo update of all the Fields contained in the ESMF_FieldBundle.
In this example, we will set up a FieldBundle for a 2D inviscid and compressible flow problem. We will illustrate the FieldBundle halo update operation but we will not solve the non-linear PDEs here. The emphasis here is to demonstrate how to set up halo regions, how a numerical scheme updates the exclusive regions, and how halo update communicates data in the halo regions. Here are the governing equations:
(conservation of momentum in x-direction)
(conservation of momentum in y-direction)
(conservation of mass)
(conservation of energy)
The four unknowns are pressure 
, density 
, velocity (
, 
). The grids
are set up using Arakawa D stagger ( on corner, at center, and on edges).
, , , and are bounded by necessary boundary conditions and initial conditions.
Section 28.2.14 provides a detailed discussion of the halo operation implemented in ESMF.
! create distgrid and grid according to the following decomposition
! and stagger pattern, r is density.
!
! p--------u-------+p+-------u--------p
! ! | |
! ! | |
! ! | |
! v r v r v
! ! PET 0 | PET 1 |
! ! | |
! ! | |
! p--------u-------+p+-------u--------p
! ! | |
! ! | |
! ! | |
! v r v r v
! ! PET 2 | PET 3 |
! ! | |
! ! | |
! p--------u-------+p+-------u--------p
!
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/256,256/), &
regDecomp=(/2,2/), &
rc=rc)
grid = ESMF_GridCreate(distgrid=distgrid, name="grid", rc=rc)
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc=rc)
! create field bundles and fields
fieldBundle = ESMF_FieldBundleCreate(rc=rc)
! set up exclusive/total region for the fields
!
! halo: L/U, nDim, nField, nPet
! halo configuration for pressure, and similarly for density, u, and v
halo(1,1,1,1) = 0
halo(2,1,1,1) = 0
halo(1,2,1,1) = 0
halo(2,2,1,1) = 0
halo(1,1,1,2) = 1 ! halo in x direction on left hand side of pet 1
halo(2,1,1,2) = 0
halo(1,2,1,2) = 0
halo(2,2,1,2) = 0
halo(1,1,1,3) = 0
halo(2,1,1,3) = 1 ! halo in y direction on upper side of pet 2
halo(1,2,1,3) = 0
halo(2,2,1,3) = 0
halo(1,1,1,4) = 1 ! halo in x direction on left hand side of pet 3
halo(2,1,1,4) = 1 ! halo in y direction on upper side of pet 3
halo(1,2,1,4) = 0
halo(2,2,1,4) = 0
! names and staggers of the 4 unknown fields
names(1) = "pressure"
names(2) = "density"
names(3) = "u"
names(4) = "v"
staggers(1) = ESMF_STAGGERLOC_CORNER
staggers(2) = ESMF_STAGGERLOC_CENTER
staggers(3) = ESMF_STAGGERLOC_EDGE2
staggers(4) = ESMF_STAGGERLOC_EDGE1
! create a FieldBundle
lpe = lpe + 1
do i = 1, 4
field(i) = ESMF_FieldCreate(grid, arrayspec, &
totalLWidth=(/halo(1,1,i,lpe), halo(1,2,i,lpe)/), &
totalUWidth=(/halo(2,1,i,lpe), halo(2,2,i,lpe)/), &
staggerloc=staggers(i), name=names(i), &
rc=rc)
call ESMF_FieldBundleAdd(fieldBundle, (/field(i)/), rc=rc)
enddo
! compute the routehandle
call ESMF_FieldBundleHaloStore(fieldBundle, routehandle=routehandle, &
rc=rc)
do iter = 1, 10
do i = 1, 4
call ESMF_FieldGet(field(i), farrayPtr=fptr, &
exclusiveLBound=excllb, exclusiveUBound=exclub, rc=rc)
sizes = exclub - excllb
! fill the total region with 0.
fptr = 0.
! only update the exclusive region on local PET
do j = excllb(1), exclub(1)
do k = excllb(2), exclub(2)
fptr(j,k) = iter * cos(2.*PI*j/sizes(1))*sin(2.*PI*k/sizes(2))
enddo
enddo
enddo
! call halo execution to update the data in the halo region,
! it can be verified that the halo regions change from 0.
! to non zero values.
call ESMF_FieldBundleHalo(fieldbundle, routehandle=routehandle, rc=rc)
enddo
! release halo route handle
call ESMF_FieldBundleHaloRelease(routehandle, rc=rc)
INTERFACE:
interface assignment(=)
fieldbundle1 = fieldbundle2
ARGUMENTS:
type(ESMF_FieldBundle) :: fieldbundle1
type(ESMF_FieldBundle) :: fieldbundle2
STATUS:
DESCRIPTION:
Assign fieldbundle1 as an alias to the same ESMF fieldbundle object in memory as fieldbundle2. If fieldbundle2 is invalid, then fieldbundle1 will be equally invalid after the assignment.
The arguments are:
INTERFACE:
interface operator(==)
if (fieldbundle1 == fieldbundle2) then ... endif
OR
result = (fieldbundle1 == fieldbundle2)
RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle1
type(ESMF_FieldBundle), intent(in) :: fieldbundle2
STATUS:
DESCRIPTION:
Test whether fieldbundle1 and fieldbundle2 are valid aliases to the same ESMF fieldbundle object in memory. For a more general comparison of two ESMF FieldBundles, going beyond the simple alias test, the ESMF_FieldBundleMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
interface operator(/=)
if (fieldbundle1 /= fieldbundle2) then ... endif
OR
result = (fieldbundle1 /= fieldbundle2)
RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle1
type(ESMF_FieldBundle), intent(in) :: fieldbundle2
STATUS:
DESCRIPTION:
Test whether fieldbundle1 and fieldbundle2 are not valid aliases to the same ESMF fieldbundle object in memory. For a more general comparison of two ESMF FieldBundles, going beyond the simple alias test, the ESMF_FieldBundleMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldBundleAdd()
subroutine ESMF_FieldBundleAddList(fieldbundle, fieldList, &
multiflag, relaxedflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_Field), intent(in) :: fieldList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Add Field(s) to a FieldBundle. It is an error if fieldList contains Fields that match by name Fields already contained in fieldbundle when multiflag is set to .false. and relaxedflag is set to .false..
INTERFACE:
subroutine ESMF_FieldBundleAddReplace(fieldbundle, fieldList, rc)ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_Field), intent(in) :: fieldList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Fields in fieldList that do not match any Fields by name in fieldbundle are added to the FieldBundle. Fields in fieldList that match any Fields by name in fieldbundle replace those Fields.
INTERFACE:
function ESMF_FieldBundleCreate(fieldList, &
multiflag, relaxedflag, name, rc)
RETURN VALUE:
type(ESMF_FieldBundle) :: ESMF_FieldBundleCreateARGUMENTS:
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Field), intent(in), optional :: fieldList(:)
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
character (len=*),intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_FieldBundle object from a list of existing Fields.
The creation of a FieldBundle leaves the bundled Fields unchanged, they remain valid individual objects. a FieldBundle is a light weight container of Field references. The actual data remains in place, there are no data movements or duplications associated with the creation of an FieldBundle.
INTERFACE:
subroutine ESMF_FieldBundleDestroy(fieldbundle, noGarbage, rc)ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: noGarbage
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Destroy an ESMF_FieldBundle object. The member Fields are not touched by this operation and remain valid objects that need to be destroyed individually if necessary.
The arguments are:
It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.
INTERFACE:
! Private name; call using ESMF_FieldBundleGet()
subroutine ESMF_FieldBundleGetListAll(fieldbundle, &
itemorderflag, geomtype, grid, locstream, mesh, xgrid, &
fieldCount, fieldList, fieldNameList, name, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_ItemOrder_Flag), intent(in), optional :: itemorderflag
type(ESMF_GeomType_Flag), intent(out), optional :: geomtype
type(ESMF_Grid), intent(out), optional :: grid
type(ESMF_LocStream), intent(out), optional :: locstream
type(ESMF_Mesh), intent(out), optional :: mesh
type(ESMF_XGrid), intent(out), optional :: xgrid
integer, intent(out), optional :: fieldCount
type(ESMF_Field), intent(out), optional :: fieldList(:)
character(len=*), intent(out), optional :: fieldNameList(:)
character(len=*), intent(out), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Get the list of all Fields and field names bundled in a FieldBundle.
INTERFACE:
! Private name; call using ESMF_FieldBundleGet()
subroutine ESMF_FieldBundleGetItem(fieldbundle, fieldName, &
field, fieldCount, isPresent, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
character(len=*), intent(in) :: fieldName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Field), intent(out), optional :: field
integer, intent(out), optional :: fieldCount
logical, intent(out), optional :: isPresent
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Get information about items that match fieldName in FieldBundle.
INTERFACE:
! Private name; call using ESMF_FieldBundleGet()
subroutine ESMF_FieldBundleGetList(fieldbundle, fieldName, fieldList, &
itemorderflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
character(len=*), intent(in) :: fieldName
type(ESMF_Field), intent(out) :: fieldList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_ItemOrder_Flag), intent(in), optional :: itemorderflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Get the list of Fields from fieldbundle that match fieldName.
INTERFACE:
subroutine ESMF_FieldBundleHalo(fieldbundle, routehandle, &
checkflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed FieldBundle halo operation for the Fields in FieldBundle. See ESMF_FieldBundleStore() on how to compute routehandle.
INTERFACE:
subroutine ESMF_FieldBundleHaloRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a FieldBundle halo operation. After this call routehandle becomes invalid.
INTERFACE:
subroutine ESMF_FieldBundleHaloStore(fieldbundle, routehandle, &
rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a FieldBundle halo operation over the data in fieldbundle. By definition, all elements in the total Field regions that lie outside the exclusive regions will be considered potential destination elements for halo. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleHalo() on any FieldBundle that is weakly congruent and typekind conform to fieldbundle. Congruency for FieldBundles is given by the congruency of its constituents. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
This call is collective across the current VM.
INTERFACE:
function ESMF_FieldBundleIsCreated(fieldbundle, rc)RETURN VALUE:
logical :: ESMF_FieldBundleIsCreatedARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Return .true. if the fieldbundle has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundlePrint(fieldbundle, rc)ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Print internal information of the specified fieldbundle object.
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundleRead(fieldbundle, fileName, &
singleFile, timeslice, iofmt, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
character(*), intent(in) :: fileName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: singleFile
integer, intent(in), optional :: timeslice
type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
integer, intent(out), optional :: rc
DESCRIPTION:
Read field data to a FieldBundle object from file(s). For this API to be functional, the environment variable ESMF_PIO should be set to "internal" when the ESMF library is built. Please see the section on Data I/O, 37.3.
Limitations:
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundleRedist(srcFieldBundle, dstFieldBundle, &
routehandle, checkflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in), optional :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout), optional :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed FieldBundle redistribution from srcFieldBundle to dstFieldBundle. Both srcFieldBundle and dstFieldBundle must be weakly congruent and typekind conform with the respective FieldBundles used during ESMF_FieldBundleRedistStore(). Congruency for FieldBundles is given by the congruency of its constituents. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
The srcFieldBundle and dstFieldBundle arguments are optional in support of the situation where srcFieldBundle and/or dstFieldBundle are not defined on all PETs. The srcFieldBundle and dstFieldBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
See ESMF_FieldBundleRedistStore() on how to precompute routehandle.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 25.2.7.
INTERFACE:
subroutine ESMF_FieldBundleRedistRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a FieldBundle redistribution. After this call routehandle becomes invalid.
INTERFACE:
! Private name; call using ESMF_FieldBundleRedistStore()
subroutine ESMF_FieldBundleRedistStore<type><kind>(srcFieldBundle, &
dstFieldBundle, routehandle, factor, &
srcToDstTransposeMap, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), intent(in) :: factor
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: srcToDstTransposeMap(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a FieldBundle redistribution operation from srcFieldBundle to dstFieldBundle. PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.
Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices. Redistribution corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.
Source and destination FieldBundles may be of different <type><kind>. Further source and destination FieldBundles may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleRedist() on any pair of FieldBundles that are congruent and typekind conform with the srcFieldBundle, dstFieldBundle pair. Congruency for FieldBundles is given by the congruency of its constituents. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 25.2.7.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldBundleRedistStore()
subroutine ESMF_FieldBundleRedistStoreNF(srcFieldBundle, dstFieldBundle, &
routehandle, factor, srcToDstTransposeMap, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: srcToDstTransposeMap(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a FieldBundle redistribution operation from srcFieldBundle to dstFieldBundle. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices. Redistribution corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.
Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleRedist() on any pair of Fields that are congruent and typekind conform with the srcFieldBundle, dstFieldBundle pair. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 25.2.7.
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundleRegrid(srcFieldBundle, dstFieldBundle, &
routehandle, zeroregion, termorderflag, checkflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in), optional :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout), optional :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Region_Flag), intent(in), optional :: zeroregion
type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag(:)
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed FieldBundle regrid from srcFieldBundle to dstFieldBundle. Both srcFieldBundle and dstFieldBundle must be congruent and typekind conform with the respective FieldBundles used during ESMF_FieldBundleRegridStore(). Congruency for FieldBundles is given by the congruency of its constituents. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
The srcFieldBundle and dstFieldBundle arguments are optional in support of the situation where srcFieldBundle and/or dstFieldBundle are not defined on all PETs. The srcFieldBundle and dstFieldBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
See ESMF_FieldBundleRegridStore() on how to precompute routehandle.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_FieldBundleRegridRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a FieldBundle regrid operation. After this call routehandle becomes invalid.
INTERFACE:
subroutine ESMF_FieldBundleRegridStore(srcFieldBundle, dstFieldBundle, &
srcMaskValues, dstMaskValues, &
regridmethod, polemethod, &
regridPoleNPnts, &
lineType, normType, &
unmappedaction, ignoreDegenerate, &
routehandle, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
integer(ESMF_KIND_I4), intent(in), optional :: srcMaskValues(:)
integer(ESMF_KIND_I4), intent(in), optional :: dstMaskValues(:)
type(ESMF_RegridMethod_Flag), intent(in), optional :: regridmethod
type(ESMF_PoleMethod_Flag), intent(in), optional :: polemethod
integer, intent(in), optional :: regridPoleNPnts
type(ESMF_LineType_Flag), intent(in), optional :: lineType
type(ESMF_NormType_Flag), intent(in), optional :: normType
type(ESMF_UnmappedAction_Flag),intent(in), optional :: unmappedaction
logical, intent(in), optional :: ignoreDegenerate
type(ESMF_RouteHandle), intent(inout),optional :: routehandle
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a FieldBundle regrid operation over the data in srcFieldBundle and dstFieldBundle pair.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleRegrid() on any FieldBundle pairs that are weakly congruent and typekind conform to the FieldBundle pair used here. Congruency for FieldBundles is given by the congruency of its constituents. Congruent Fields possess matching DistGrids, and the shape of the local array tiles matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Note ESMF_FieldBundleRegridStore() assumes the coordinates used in the Grids upon which the FieldBundles are built are in degrees.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_FieldBundleRemove(fieldbundle, fieldNameList, &
multiflag, relaxedflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
character(len=*), intent(in) :: fieldNameList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Remove field(s) by name from FieldBundle. In the relaxed setting it is not an error if fieldNameList contains names that are not found in fieldbundle.
INTERFACE:
subroutine ESMF_FieldBundleReplace(fieldbundle, fieldList, &
multiflag, relaxedflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_Field), intent(in) :: fieldList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Replace field(s) by name in FieldBundle. In the relaxed setting it is not an error if fieldList contains Fields that do not match by name any item in fieldbundle. These Fields are simply ignored in this case.
INTERFACE:
! Private name; call using ESMF_FieldBundleSet()
subroutine ESMF_FieldBundleSetGrid(fieldbundle, grid, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_Grid), intent(in) :: grid
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Sets the grid for a fieldbundle.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldBundleSet()
subroutine ESMF_FieldBundleSetMesh(fieldbundle, mesh, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_Mesh), intent(in) :: mesh
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Sets the mesh for a fieldbundle.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldBundleSet()
subroutine ESMF_FieldBundleSetLS(fieldbundle, locstream, &
rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_LocStream), intent(in) :: locstream
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Sets the locstream for a fieldbundle.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldBundleSet()
subroutine ESMF_FieldBundleSetXGrid(fieldbundle, xgrid, &
rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(inout) :: fieldbundle
type(ESMF_XGrid), intent(in) :: xgrid
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Sets the xgrid for a fieldbundle
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundleSMM(srcFieldBundle, dstFieldBundle, &
routehandle, zeroregion, termorderflag, checkflag, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in), optional :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout), optional :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Region_Flag), intent(in), optional :: zeroregion
type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag(:)
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed FieldBundle sparse matrix multiplication from srcFieldBundle to dstFieldBundle. Both srcFieldBundle and dstFieldBundle must be congruent and typekind conform with the respective FieldBundles used during ESMF_FieldBundleSMMStore(). Congruent FieldBundles possess matching DistGrids and the shape of the local array tiles matches between the FieldBundles for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
The srcFieldBundle and dstFieldBundle arguments are optional in support of the situation where srcFieldBundle and/or dstFieldBundle are not defined on all PETs. The srcFieldBundle and dstFieldBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
See ESMF_FieldBundleSMMStore() on how to precompute routehandle.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 25.2.8.
ESMF_REGION_SELECT will only zero out those elements in the destination FieldBundle that will be updated by the sparse matrix multiplication. See section 51.46 for a complete list of valid settings.
INTERFACE:
subroutine ESMF_FieldBundleSMMRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a FieldBundle sparse matrix multiplication. After this call routehandle becomes invalid.
INTERFACE:
! Private name; call using ESMF_FieldBundleSMMStore()
subroutine ESMF_FieldBundleSMMStore<type><kind>(srcFieldBundle, &
dstFieldBundle, routehandle, factorList, factorIndexList, &
rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), intent(in) :: factorList(:)
integer, intent(in), :: factorIndexList(:,:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a FieldBundle sparse matrix multiplication operation from srcFieldBundle to dstFieldBundle. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.
Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleSMM() on any pair of FieldBundles that are congruent and typekind conform with the srcFieldBundle, dstFieldBundle pair. Congruent FieldBundles possess matching DistGrids and the shape of the local array tiles matches between the FieldBundles for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 25.2.8.
The arguments are:
The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.
In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcFieldBundle while factorIndexList(2,:) specifies the sequence index of the destination element in dstFieldBundle. For this format to be a valid option source and destination FieldBundles must have matching number of tensor elements (the product of the sizes of all Field tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.
The size 4 format is more general and does not require a matching tensor element count. Here the
factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcFieldBundle. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstFieldBundle.
See section 28.2.17 for details on the definition of sequence indices and tensor sequence indices.
INTERFACE:
! Private name; call using ESMF_FieldBundleSMMStore()
subroutine ESMF_FieldBundleSMMStoreNF(srcFieldBundle, dstFieldBundle, &
routehandle, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: srcFieldBundle
type(ESMF_FieldBundle), intent(inout) :: dstFieldBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a FieldBundle sparse matrix multiplication operation from srcFieldBundle to dstFieldBundle. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcFieldBundle and dstFieldBundle are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source FieldBundle vector to the destination FieldBundle vector.
Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical FieldBundle object for srcFieldBundle and dstFieldBundle arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldBundleSMM() on any pair of FieldBundles that are congruent and typekind conform with the srcFieldBundle, dstFieldBundle pair. Congruent FieldBundles possess matching DistGrids and the shape of the local array tiles matches between the FieldBundles for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
This method is overloaded for ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8, ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 25.2.8.
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundleValidate(fieldbundle, rc)ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
integer, intent(out), optional :: rc
DESCRIPTION:
Validates that the fieldbundle is internally consistent. The method returns an error code if problems are found.
The arguments are:
INTERFACE:
subroutine ESMF_FieldBundleWrite(fieldbundle, fileName, &
singleFile, overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
type(ESMF_FieldBundle), intent(in) :: fieldbundle
character(*), intent(in) :: fileName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: singleFile
logical , intent(in), optional :: overwrite
type(ESMF_FileStatus_Flag), intent(in), optional :: status
integer, intent(in), optional :: timeslice
type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
integer, intent(out), optional :: rc
DESCRIPTION:
Write the Fields into a file. For this API to be functional, the environment variable ESMF_PIO should be set to "internal" when the ESMF library is built. Please see the section on Data I/O, 37.3.
Limitations:
The arguments are:
An ESMF Field represents a physical field, such as temperature. The motivation for including Fields in ESMF is that bundles of Fields are the entities that are normally exchanged when coupling Components.
The ESMF Field class contains distributed and discretized field data, a reference to its associated grid, and metadata. The Field class stores the grid staggering for that physical field. This is the relationship of how the data array of a field maps onto a grid (e.g. one item per cell located at the cell center, one item per cell located at the NW corner, one item per cell vertex, etc.). This means that different Fields which are on the same underlying ESMF Grid but have different staggerings can share the same Grid object without needing to replicate it multiple times.
Fields can be added to States for use in inter-Component data communications. Fields can also be added to FieldBundles, which are groups of Fields on the same underlying Grid. One motivation for packing Fields into FieldBundles is convenience; another is the ability to perform optimized collective data transfers.
Field communication capabilities include: data redistribution, regridding, scatter, gather, sparse-matrix multiplication, and halo update. These are discussed in more detail in the documentation for the specific method calls. ESMF does not currently support vector fields, so the components of a vector field must be stored as separate Field objects.
The Field class allows the user to easily perform a number of operations on the data stored in a Field. This section gives a brief summary of the different types of operations and the range of their capabilities. The operations covered here are: redistribution (ESMF_FieldRedistStore()), sparse matrix multiply (ESMF_FieldSMMStore()), and regridding (ESMF_FieldRegridStore()).
The redistribution operation (ESMF_FieldRedistStore()) allows the user to move data between two Fields with the same size, but different distribution. This operation is useful, for example, to move data between two components with different distributions. Please see Section 26.3.29 for an example of the redistribution capability.
The sparse matrix multiply operation (ESMF_FieldSMMStore()) allows the user to multiply the data in a Field by a sparse matrix. This operation is useful, for example, if the user has an interpolation matrix and wants to apply it to the data in a Field. Please see Section 26.3.32 for an example of the sparse matrix multiply capability.
The regridding operation (ESMF_FieldRegridStore()) allows the user to move data from one grid to another while maintaining certain properties of the data. Regridding is also called interpolation or remapping. In the Field regridding operation the grids the data is being moved between are the grids associated with the Fields storing the data. The regridding operation works on Fields built on 2D or 3D Meshes and 2D or 3D Grids. There are three regridding methods available: bilinear, higher-order patch, and first-order conservative. All three of these methods are supported on 2D Grids or 2D Meshes. In 3D the situation is more complicated. Bilinear is supported on 3D Grid or 3D Meshes composed of hexahedrons. Higher-order patch is not supported in 3D. First-order conservative is supported on 3D Grids or 3D Meshes composed of hexahdrons or tetrahedrons. There are also a range of options for what to do at the poles and what to do if a point does not map. Please see Section 26.3.24 for a full description of the regridding capability. Several sections following Section 26.3.24 contain examples of using regridding.
DESCRIPTION:
An ESMF_Field can be in different status after initialization. Field status can be queried using ESMF_FieldGet() method.
The type of this flag is:
type(ESMF_FieldStatus_Flag)
The valid values are:
A Field serves as an annotator of data, since it carries a description of the grid it is associated with and metadata such as name and units. Fields can be used in this capacity alone, as convenient, descriptive containers into which arrays can be placed and retrieved. However, for most codes the primary use of Fields is in the context of import and export States, which are the objects that carry coupling information between Components. Fields enable data to be self-describing, and a State holding ESMF Fields contains data in a standard format that can be queried and manipulated.
The sections below go into more detail about Field usage.
Fields can be created and destroyed at any time during application execution. However, these Field methods require some time to complete. We do not recommend that the user create or destroy Fields inside performance-critical computational loops.
All versions of the ESMF_FieldCreate() routines require a Grid object as input, or require a Grid be added before most operations involving Fields can be performed. The Grid contains the information needed to know which Decomposition Elements (DEs) are participating in the processing of this Field, and which subsets of the data are local to a particular DE.
The details of how the create process happens depends on which of the variants of the ESMF_FieldCreate() call is used. Some of the variants are discussed below.
There are versions of the ESMF_FieldCreate() interface which create the Field based on the input Grid. The ESMF can allocate the proper amount of space but not assign initial values. The user code can then get the pointer to the uninitialized buffer and set the initial data values.
Other versions of the ESMF_FieldCreate() interface allow user code to attach arrays that have already been allocated by the user. Empty Fields can also be created in which case the data can be added at some later time.
For versions of Create which do not specify data values, user code can create an ArraySpec object, which contains information about the typekind and rank of the data values in the array. Then at Field create time, the appropriate amount of memory is allocated to contain the data which is local to each DE.
When finished with a ESMF_Field, the ESMF_FieldDestroy method removes it. However, the objects inside the ESMF_Field created externally should be destroyed separately, since objects can be added to more than one ESMF_Field. For example, the same ESMF_Grid can be referenced by multiple ESMF_Fields. In this case the internal Grid is not deleted by the ESMF_FieldDestroy call.
A user can get bounds and counts information from an ESMF_Field through the ESMF_FieldGet() interface. Also available through this interface is the intrinsic Fortran data pointer contained in the internal ESMF_Array object of an ESMF_Field. The bounds and counts information are DE specific for the associated Fortran data pointer.
For a better discussion of the terminologies, bounds and widths in ESMF e.g. exclusive, computational, total bounds for the lower and upper corner of data region, etc.., user can refer to the explanation of these concepts for Grid and Array in their respective sections in the Reference Manual, e.g. Section 28.2.6 on Array and Section 31.3.16 on Grid.
In this example, we first create a 3D Field based on a 3D Grid and Array. Then we use the ESMF_FieldGet() interface to retrieve the data pointer, potentially updating or verifying its values. We also retrieve the bounds and counts information of the 3D Field to assist in data element iteration.
xdim = 180
ydim = 90
zdim = 50
! create a 3D data Field from a Grid and Array.
! first create a Grid
grid3d = ESMF_GridCreateNoPeriDim(minIndex=(/1,1,1/), &
maxIndex=(/xdim,ydim,zdim/), &
regDecomp=(/2,2,1/), name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridGet(grid=grid3d, staggerloc=ESMF_STAGGERLOC_CENTER, &
distgrid=distgrid3d, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridGetFieldBounds(grid=grid3d, localDe=0, &
staggerloc=ESMF_STAGGERLOC_CENTER, totalCount=fa_shape, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
allocate(farray(fa_shape(1), fa_shape(2), fa_shape(3)) )
! create an Array
array3d = ESMF_ArrayCreate(distgrid3d, farray, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create a Field
field = ESMF_FieldCreate(grid=grid3d, array=array3d, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! retrieve the Fortran data pointer from the Field
call ESMF_FieldGet(field=field, localDe=0, farrayPtr=farray1, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! retrieve the Fortran data pointer from the Field and bounds
call ESMF_FieldGet(field=field, localDe=0, farrayPtr=farray1, &
computationalLBound=compLBnd, computationalUBound=compUBnd, &
exclusiveLBound=exclLBnd, exclusiveUBound=exclUBnd, &
totalLBound=totalLBnd, totalUBound=totalUBnd, &
computationalCount=comp_count, &
exclusiveCount=excl_count, &
totalCount=total_count, &
rc=rc)
! iterate through the total bounds of the field data pointer
do k = totalLBnd(3), totalUBnd(3)
do j = totalLBnd(2), totalUBnd(2)
do i = totalLBnd(1), totalUBnd(1)
farray1(i, j, k) = sin(2*i/total_count(1)*PI) + &
sin(4*j/total_count(2)*PI) + &
sin(8*k/total_count(2)*PI)
enddo
enddo
enddo
A user can get the internal ESMF_Grid and ESMF_Array from a ESMF_Field. Note that the user should not issue any destroy command on the retrieved grid or array object since they are referenced from within the ESMF_Field. The retrieved objects should be used in a read-only fashion to query additional information not directly available through the ESMF_FieldGet() interface.
call ESMF_FieldGet(field, grid=grid, array=array, &
typekind=typekind, dimCount=dimCount, staggerloc=staggerloc, &
gridToFieldMap=gridToFieldMap, &
ungriddedLBound=ungriddedLBound, ungriddedUBound=ungriddedUBound, &
totalLWidth=totalLWidth, totalUWidth=totalUWidth, &
name=name, &
rc=rc)
A user can create an ESMF_Field from an ESMF_Grid and typekind/rank. This create method associates the two objects.
We first create a Grid with a regular distribution that is 10x20 index in 2x2 DEs. This version of Field create simply associates the data with the Grid. The data is referenced explicitly on a regular 2x2 uniform grid. Finally we create a Field from the Grid, typekind, rank, and a user specified StaggerLoc.
This example also illustrates a typical use of this Field creation method. By creating a Field from a Grid and typekind/rank, the user allows the ESMF library to create a internal Array in the Field. Then the user can use ESMF_FieldGet() to retrieve the Fortran data array and necessary bounds information to assign initial values to it.
! create a grid
grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), name="atmgrid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create a Field from the Grid and arrayspec
field1 = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_R4, &
indexflag=ESMF_INDEX_DELOCAL, &
staggerloc=ESMF_STAGGERLOC_CENTER, name="pressure", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field1, localDe=0, farrayPtr=farray2dd, &
totalLBound=ftlb, totalUBound=ftub, totalCount=ftc, rc=rc)
do i = ftlb(1), ftub(1)
do j = ftlb(2), ftub(2)
farray2dd(i, j) = sin(i/ftc(1)*PI) * cos(j/ftc(2)*PI)
enddo
enddo
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
A user can create an ESMF_Field from an ESMF_Grid and a ESMF_Arrayspec with corresponding rank and type. This create method associates the two objects.
We first create a Grid with a regular distribution that is 10x20 index in 2x2 DEs. This version of Field create simply associates the data with the Grid. The data is referenced explicitly on a regular 2x2 uniform grid. Then we create an ArraySpec. Finally we create a Field from the Grid, ArraySpec, and a user specified StaggerLoc.
This example also illustrates a typical use of this Field creation method. By creating a Field from a Grid and an ArraySpec, the user allows the ESMF library to create a internal Array in the Field. Then the user can use ESMF_FieldGet() to retrieve the Fortran data array and necessary bounds information to assign initial values to it.
! create a grid
grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), name="atmgrid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! setup arrayspec
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create a Field from the Grid and arrayspec
field1 = ESMF_FieldCreate(grid, arrayspec, &
indexflag=ESMF_INDEX_DELOCAL, &
staggerloc=ESMF_STAGGERLOC_CENTER, name="pressure", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field1, localDe=0, farrayPtr=farray2dd, &
totalLBound=ftlb, totalUBound=ftub, totalCount=ftc, rc=rc)
do i = ftlb(1), ftub(1)
do j = ftlb(2), ftub(2)
farray2dd(i, j) = sin(i/ftc(1)*PI) * cos(j/ftc(2)*PI)
enddo
enddo
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
A user can also create an ArraySpec that has a different rank from the Grid, For example, the following code shows creation of of 3D Field from a 2D Grid using a 3D ArraySpec.
This example also demonstrates the technique to create a typical 3D data Field that has 2 gridded dimensions and 1 ungridded dimension.
First we create a 2D grid with an index space of 180x360 equivalent to 180x360 Grid cells (note that for a distributed memory computer, this means each grid cell will be on a separate PE!). In the FieldCreate call, we use gridToFieldMap to indicate the mapping between Grid dimension and Field dimension. For the ungridded dimension (typically the altitude), we use ungriddedLBound and ungriddedUBound to describe its bounds. Internally the ungridded dimension has a stride of 1, so the number of elements of the ungridded dimension is ungriddedUBound - ungriddedLBound + 1.
Note that gridToFieldMap in this specific example is (/1,2/) which is the default value so the user can neglect this argument for the FieldCreate call.
grid2d = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), &
maxIndex=(/180,360/), regDecomp=(/2,2/), name="atmgrid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_ArraySpecSet(arrayspec, 3, ESMF_TYPEKIND_R4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
field1 = ESMF_FieldCreate(grid2d, arrayspec, &
indexflag=ESMF_INDEX_DELOCAL, &
staggerloc=ESMF_STAGGERLOC_CENTER, &
gridToFieldMap=(/1,2/), &
ungriddedLBound=(/1/), ungriddedUBound=(/50/), &
name="pressure", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
A user can create an ESMF_Field from an ESMF_Grid and a ESMF_Array. The Grid was created in the previous example.
This example creates a 2D ESMF_Field from a 2D ESMF_Grid and a 2D ESMF_Array.
! Get necessary information from the Grid
call ESMF_GridGet(grid, staggerloc=ESMF_STAGGERLOC_CENTER, &
distgrid=distgrid, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create a 2D ESMF_TYPEKIND_R4 arrayspec
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_R4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create a ESMF_Array from the arrayspec and distgrid
array2d = ESMF_ArrayCreate(arrayspec=arrayspec, &
distgrid=distgrid, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create a ESMF_Field from the grid and array
field4 = ESMF_FieldCreate(grid, array2d, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
A user can create an ESMF_Field in three steps: first create an empty ESMF_Field; then set a ESMF_Grid on the empty ESMF_Field; and finally complete the ESMF_Field by calling ESMF_FieldEmptyComplete.
! create an empty Field
field3 = ESMF_FieldEmptyCreate(name="precip", rc=rc)
! use FieldGet to retrieve the Field Status
call ESMF_FieldGet(field3, status=fstatus, rc=rc)
Once the Field is created, we can verify that the status of the Field is ESMF_FIELDSTATUS_EMPTY.
! Test the status of the Field
if (fstatus /= ESMF_FIELDSTATUS_EMPTY) then
call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
Next we set a Grid on the empty Field. We use the 2D grid created in a previous example simply to demonstrate the method. The Field data points will be on east edge of the Grid cells with the specified ESMF_STAGGERLOC_EDGE1.
! Set a grid on the Field
call ESMF_FieldEmptySet(field3, grid2d, &
staggerloc=ESMF_STAGGERLOC_EDGE1, rc=rc)
! use FieldGet to retrieve the Field Status again
call ESMF_FieldGet(field3, status=fstatus, rc=rc)
! Test the status of the Field
if (fstatus /= ESMF_FIELDSTATUS_GRIDSET) then
call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
The partially created Field is completed by specifying the typekind of its data storage. This method is overloaded with one of the following parameters, arrayspec, typekind, Fortran array, or Fortran array pointer. Additional optional arguments can be used to specify ungridded dimensions and halo regions similar to the other Field creation methods.
! Complete the Field by specifying the data typekind
! to be allocated internally.
call ESMF_FieldEmptyComplete(field3, typekind=ESMF_TYPEKIND_R8, &
ungriddedLBound=(/1/), ungriddedUBound=(/5/), rc=rc)
! use FieldGet to retrieve the Field Status again
call ESMF_FieldGet(field3, status=fstatus, rc=rc)
! Test the status of the Field
if (fstatus /= ESMF_FIELDSTATUS_COMPLETE) then
call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
A user can create an empty ESMF_Field. Then the user can finalize the empty ESMF_Field from a ESMF_Grid and a intrinsic Fortran data array. This interface is overloaded for typekind and rank of the Fortran data array.
In this example, both grid and Fortran array pointer are 2 dimensional and each dimension index maps in order, i.e. 1st dimension of grid maps to 1st dimension of Fortran array pointer, 2nd dimension of grid maps to 2nd dimension of Fortran array pointer, so on and so forth.
In order to create or complete a Field from a Grid and a Fortran array pointer, certain rules of the Fortran array bounds must be obeyed. We will discuss these rules as we progress in Field creation examples. We will make frequent reference to the terminologies for bounds and widths in ESMF. For a better discussion of these terminologies and concepts behind them, e.g. exclusive, computational, total bounds for the lower and upper corner of data region, etc.., users can refer to the explanation of these concepts for Grid and Array in their respective sections in the Reference Manual, e.g. Section 28.2.6 on Array and Section 31.3.16 on Grid. The examples here are designed to help a user to get up to speed with creating Fields for typical use.
This example introduces a helper method, the ESMF_GridGetFieldBounds interface that facilitates the computation of Fortran data array bounds and shape to assist ESMF_FieldEmptyComplete finalizing a Field from an intrinsic Fortran data array and a Grid.
! create an empty Field
field3 = ESMF_FieldEmptyCreate(name="precip", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! use FieldGet to retrieve total counts
call ESMF_GridGetFieldBounds(grid2d, localDe=0, &
staggerloc=ESMF_STAGGERLOC_CENTER, totalCount=ftc, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! allocate the 2d Fortran array based on retrieved total counts
allocate(farray2d(ftc(1), ftc(2)))
! finalize the Field
call ESMF_FieldEmptyComplete(field3, grid2d, farray2d, rc=rc)
In this example, we will show how to create a 7D Field from a 5D ESMF_Grid and 2D ungridded bounds with arbitrary halo widths and gridToFieldMap.
We first create a 5D DistGrid and a 5D Grid based on the DistGrid; then ESMF_GridGetFieldBounds computes the shape of a 7D array in fsize. We can then create a 7D Field from the 5D Grid and the 7D Fortran data array with other assimilating parameters.
! create a 5d distgrid
distgrid5d = ESMF_DistGridCreate(minIndex=(/1,1,1,1,1/), &
maxIndex=(/10,4,10,4,6/), regDecomp=(/2,1,2,1,1/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create a 5d Grid
grid5d = ESMF_GridCreate(distgrid=distgrid5d, name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! use FieldGet to retrieve total counts
call ESMF_GridGetFieldBounds(grid5d, localDe=0, ungriddedLBound=(/1,2/), &
ungriddedUBound=(/4,5/), &
totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
gridToFieldMap=(/3,2,5,4,1/), &
totalCount=fsize, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! allocate the 7d Fortran array based on retrieved total counts
allocate(farray7d(fsize(1), fsize(2), fsize(3), fsize(4), fsize(5), &
fsize(6), fsize(7)))
! create the Field
field7d = ESMF_FieldCreate(grid5d, farray7d, ESMF_INDEX_DELOCAL, &
ungriddedLBound=(/1,2/), ungriddedUBound=(/4,5/), &
totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
gridToFieldMap=(/3,2,5,4,1/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
A user can allocate the Fortran array in a different manner using the lower and upper bounds returned from FieldGet through the optional totalLBound and totalUBound arguments. In the following example, we create another 7D Field by retrieving the bounds and allocate the Fortran array with this approach. In this scheme, indexing the Fortran array is sometimes more convenient than using the shape directly.
call ESMF_GridGetFieldBounds(grid5d, localDe=0, ungriddedLBound=(/1,2/), &
ungriddedUBound=(/4,5/), &
totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
gridToFieldMap=(/3,2,5,4,1/), &
totalLBound=flbound, totalUBound=fubound, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
allocate(farray7d2(flbound(1):fubound(1), flbound(2):fubound(2), &
flbound(3):fubound(3), flbound(4):fubound(4), &
flbound(5):fubound(5), flbound(6):fubound(6), &
flbound(7):fubound(7)) )
field7d2 = ESMF_FieldCreate(grid5d, farray7d2, ESMF_INDEX_DELOCAL, &
ungriddedLBound=(/1,2/), ungriddedUBound=(/4,5/), &
totalLWidth=(/1,1,1,2,2/), totalUWidth=(/1,2,3,4,5/), &
gridToFieldMap=(/3,2,5,4,1/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
A user can create an ESMF_Field directly from an ESMF_Grid and an intrinsic Fortran data array. This interface is overloaded for typekind and rank of the Fortran data array.
In the following example, each dimension size of the Fortran array is equal to the exclusive bounds of its corresponding Grid dimension queried from the Grid through ESMF_GridGet() public interface.
Formally let fa_shape(i) be the shape of i-th dimension of user supplied Fortran array, then rule 1 states:
(1) fa_shape(i) = exclusiveCount(i)
i = 1...GridDimCount
fa_shape(i) defines the shape of i-th dimension of the Fortran array. ExclusiveCount are the number of data elements of i-th dimension in the exclusive region queried from ESMF_GridGet interface. Rule 1 assumes that the Grid and the Fortran intrinsic array have same number of dimensions; and optional arguments of FieldCreate from Fortran array are left unspecified using default setup. These assumptions are true for most typical use of FieldCreate from Fortran data array. This is the easiest way to create a Field from a Grid and Fortran intrinsic data array.
Fortran array dimension sizes (called shape in most Fortran language books) are equivalent to the bounds and counts used in this manual. The following equation holds:
fa_shape(i) = shape(i) = counts(i) = upper_bound(i) - lower_bound(i) + 1
These typically mean the same concept unless specifically explained to mean something else. For example, ESMF uses DimCount very often to mean number of dimensions instead of its meaning implied in the above equation. We'll clarify the meaning of a word when ambiguity could occur.
Rule 1 is most useful for a user working with Field creation from a Grid and a Fortran data array in most scenarios. It extends to higher dimension count, 3D, 4D, etc... Typically, as the code example demonstrates, a user first creates a Grid , then uses ESMF_GridGet() to retrieve the exclusive counts. Next the user calculates the shape of each Fortran array dimension according to rule 1. The Fortran data array is allocated and initialized based on the computed shape. A Field can either be created in one shot created empty and finished using ESMF_FieldEmptyComplete.
There are important details that can be skipped but are good to know for ESMF_FieldEmptyComplete
and ESMF_FieldCreate from a Fortran data array. 1) these methods require each PET contains
exactly one DE. This implies that a code using FieldCreate from a data array or FieldEmptyComplete must
have the same number of DEs and PETs, formally
. Violation of this condition
will cause run time failures. 2) the bounds and counts retrieved from GridGet are DE specific
or equivalently PET specific, which means that the Fortran array shape could be different from one
PET to another.
grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), name="atmgrid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridGet(grid, localDE=0, staggerloc=ESMF_STAGGERLOC_CENTER, &
exclusiveCount=gec, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
allocate(farray(gec(1), gec(2)) )
field = ESMF_FieldCreate(grid, farray, ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
The setup of this example is similar to the previous section except that the Field is created from a data pointer instead of a data array. We highlight the ability to deallocate the internal fortran data pointer queried from the Field. This gives a user more flexibility with memory management.
allocate(farrayPtr(gec(1), gec(2)) )
field = ESMF_FieldCreate(grid, farrayPtr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field, farrayPtr=farrayPtr2, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! deallocate the retrieved fortran array pointer
deallocate(farrayPtr2)
This example demonstrates a typical use of ESMF_Field combining a 2D grid and a 3D Fortran native data array. One immediate problem follows: how does one define the bounds of the ungridded dimension? This is solved by the optional arguments ungriddedLBound and ungriddedUBound of the ESMF_FieldCreate interface. By definition, ungriddedLBound and ungriddedUBound are both 1 dimensional integer Fortran arrays.
Formally, let fa_shape(j=1...FieldDimCount-GridDimCount) be the shape of the ungridded dimensions of a Field relative to the Grid used in Field creation. The Field dimension count is equal to the number of dimensions of the Fortran array, which equals the number of dimensions of the resultant Field. GridDimCount is the number of dimensions of the Grid.
fa_shape(j) is computed as:
fa_shape(j) = ungriddedUBound(j) - ungriddedLBound(j) + 1
fa_shape is easy to compute when the gridded and ungridded dimensions do not mix. However, it's conceivable that at higher dimension count, gridded and ungridded dimensions can interleave. To aid the computation of ungridded dimension shape we formally introduce the mapping concept.
Let
, and
![$i_B \in [\phi, 1...n_B]$](img25.png){kind=small}
. 
is the number
of elements in set A, 
is the number of elements in set B. 
defines
a mapping from i-th element of set A to 
-th element in set B. 
indicates there does not exist a mapping from i-th element of set A to set B.
Suppose we have a mapping from dimension index of ungriddedLBound (or
ungriddedUBound) to Fortran array dimension index, called ugb2fa.
By definition, equals to the dimension count of
ungriddedLBound (or ungriddedUBound), equals to the dimension count of
the Fortran array. We can now formulate the computation of ungridded
dimension shape as rule 2:
(2) fa_shape(ugb2fa(j)) = ungriddedUBound(j) - ungriddedLBound(j) + 1
j = 1..FortranArrayDimCount - GridDimCount
The mapping can be computed in linear time proportional to the Fortran array dimension count (or rank) using the following algorithm in pseudocode:
map_index = 1
do i = 1, farray_rank
if i-th dimension of farray is ungridded
ugb2fa(map_index) = i
map_index = map_index + 1
endif
enddo
Here we use rank and dimension count interchangeably. These 2 terminologies are typically equivalent. But there are subtle differences under certain conditions. Rank is the total number of dimensions of a tensor object. Dimension count allows a finer description of the heterogeneous dimensions in that object. For example, A Field of rank 5 can have 3 gridded dimensions and 2 ungridded dimensions. Rank is precisely the summation of dimension count of all types of dimensions.
For example, if a 5D array is used with a 3D Grid, there are 2 ungridded dimensions: ungriddedLBound=(/1,2/) and ungriddedUBound=(/5,7/). Suppose the distribution of dimensions look like (O, X, O, X, O), O means gridded, X means ungridded. Then the mapping from ungridded bounds to Fortran array is ugb2fa=(/2, 4/). The shape of 2nd and 4th dimension of Fortran array should equal (5, 8).
Back to our 3D Field created from a 2D Grid and 3D Fortran array example, suppose the 3rd Field dimension is ungridded, ungriddedLBound=(/3/), ungriddedUBound=(/9/). First we use rule 1 to compute shapes of the gridded Fortran array dimension, then we use rule 2 to compute shapes of the ungridded Fortran array dimension. In this example, we used the exclusive bounds obtained in the previous example.
fa_shape(1) = gec(1) ! rule 1
fa_shape(2) = gec(2)
fa_shape(3) = 7 ! rule 2 9-3+1
allocate(farray3d(fa_shape(1), fa_shape(2), fa_shape(3)))
field = ESMF_FieldCreate(grid, farray3d, ESMF_INDEX_DELOCAL, &
ungriddedLBound=(/3/), ungriddedUBound=(/9/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Building upon the previous example, we will create a 3D Field from a 2D grid and 3D array but with a slight twist. In this example, we introduce the gridToFieldMap argument that allows a user to map Grid dimension index to Field dimension index.
In this example, both dimensions of the Grid are distributed and the mapping from DistGrid to Grid is (/1,2/). We will introduce rule 3 assuming distgridToGridMap=(/1,2,3...gridDimCount/), and distgridDimCount equals to gridDimCount. This is a reasonable assumption in typical Field use.
We apply the mapping gridToFieldMap on rule 1 to create rule 3:
(3) fa_shape(gridToFieldMap(i)) = exclusiveCount(i)
i = 1,..GridDimCount.
Back to our example, suppose the 2nd Field dimension is ungridded, ungriddedLBound=(/3/), ungriddedUBound=(/9/). gridToFieldMap=(/3,1/), meaning the 1st Grid dimension maps to 3rd Field dimension, and 2nd Grid dimension maps to 1st Field dimension.
First we use rule 3 to compute shapes of the gridded Fortran array dimension, then we use rule 2 to compute shapes of the ungridded Fortran array dimension. In this example, we use the exclusive bounds obtained in the previous example.
gridToFieldMap2d(1) = 3
gridToFieldMap2d(2) = 1
do i = 1, 2
fa_shape(gridToFieldMap2d(i)) = gec(i)
end do
fa_shape(2) = 7
allocate(farray3d(fa_shape(1), fa_shape(2), fa_shape(3)))
field = ESMF_FieldCreate(grid, farray3d, ESMF_INDEX_DELOCAL, &
ungriddedLBound=(/3/), ungriddedUBound=(/9/), &
gridToFieldMap=gridToFieldMap2d, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
This example is similar to example 26.3.13, in addition we will show a user can associate different halo width to a Fortran array to create a Field through the totalLWidth and totalUWidth optional arguments. A diagram of the dimension configuration from Grid, halos, and Fortran data array is shown here.
The ESMF_FieldCreate() interface supports creating a Field from a Grid and a Fortran array padded with halos on the distributed dimensions of the Fortran array. Using this technique one can avoid passing non-contiguous Fortran array slice to FieldCreate. It guarantees the same exclusive region, and by using halos, it also defines a bigger total region to contain the entire contiguous memory block of the Fortran array.
The elements of totalLWidth and totalUWidth are applied in the order distributed dimensions appear in the Fortran array. By definition, totalLWidth and totalUWidth are 1 dimensional arrays of non-negative integer values. The size of haloWidth arrays is equal to the number of distributed dimensions of the Fortran array, which is also equal to the number of distributed dimensions of the Grid used in the Field creation.
Because the order of totalWidth (representing both totalLWidth and totalUWidth) element is applied to the order distributed dimensions appear in the Fortran array dimensions, it's quite simple to compute the shape of distributed dimensions of the Fortran array. They are done in a similar manner when applying ungriddedLBound and ungriddedUBound to ungridded dimensions of the Fortran array defined by rule 2.
Assume we have the mapping from the dimension index of totalWidth to the dimension index of Fortran array, called mhw2fa; and we also have the mapping from dimension index of Fortran array to dimension index of the Grid, called fa2g. The shape of distributed dimensions of a Fortran array can be computed by rule 4:
(4) fa_shape(mhw2fa(k)) = exclusiveCount(fa2g(mhw2fa(k)) +
totalUWidth(k) + totalLWidth(k)
k = 1...size(totalWidth)
This rule may seem confusing but algorithmically the computation can be done by the following pseudocode:
fa_index = 1
do i = 1, farray_rank
if i-th dimension of Fortran array is distributed
fa_shape(i) = exclusiveCount(fa2g(i)) +
totalUWidth(fa_index) + totalLWidth(fa_index)
fa_index = fa_index + 1
endif
enddo
The only complication then is to figure out the mapping from Fortran array dimension index to Grid dimension index. This process can be done by computing the reverse mapping from Field to Grid.
Typically, we don't have to consider these complications if the following conditions are met: 1) All Grid dimensions are distributed. 2) DistGrid in the Grid has a dimension index mapping to the Grid in the form of natural order (/1,2,3,.../). This natural order mapping is the default mapping between various objects throughout ESMF. 3) Grid to Field mapping is in the form of natural order, i.e. default mapping. These seem like a lot of conditions but they are the default case in the interaction among DistGrid, Grid, and Field. When these conditions are met, which is typically true, the shape of distributed dimensions of Fortran array follows rule 5 in a simple form:
(5) fa_shape(k) = exclusiveCount(k) +
totalUWidth(k) + totalLWidth(k)
k = 1...size(totalWidth)
Let's examine an example on how to apply rule 5. Suppose we have a 5D array and a 3D Grid that has its first 3 dimensions mapped to the first 3 dimensions of the Fortran array. totalLWidth=(/1,2,3/), totalUWidth=(/7,9,10/), then by rule 5, the following pseudo code can be used to compute the shape of the first 3 dimensions of the Fortran array. The shape of the remaining two ungridded dimensions can be computed according to rule 2.
do k = 1, 3
fa_shape(k) = exclusiveCount(k) +
totalUWidth(k) + totalLWidth(k))
enddo
Suppose now gridToFieldMap=(/2,3,4/) instead which says the first dimension of Grid maps to the 2nd dimension of Field (or Fortran array) and so on and so forth, we can obtain a more general form of rule 5 by introducing first_distdim_index shift when Grid to Field map (gridToFieldMap) is in the form of (/a,a+1,a+2.../).
(6) fa_shape(k+first_distdim_index-1) = exclusiveCount(k) +
totalUWidth(k) + totalLWidth(k)
k = 1...size(totalWidth)
It's obvious that first_distdim_index=a. If the first dimension of the Fortran array is distributed, then rule 6 degenerates into rule 5, which is the typical case.
Back to our example creating a 3D Field from a 2D Grid and a 3D intrinsic Fortran array, we will use the Grid created from previous example that satisfies condition 1 and 2. We'll also use a simple gridToFieldMap (1,2) which is the default mapping that satisfies condition 3. First we use rule 5 to compute the shape of distributed dimensions then we use rule 2 to compute the shape of the ungridded dimensions.
gridToFieldMap2d(1) = 1
gridToFieldMap2d(2) = 2
totalLWidth2d(1) = 3
totalLWidth2d(2) = 4
totalUWidth2d(1) = 3
totalUWidth2d(2) = 5
do k = 1, 2
fa_shape(k) = gec(k) + totalLWidth2d(k) + totalUWidth2d(k)
end do
fa_shape(3) = 7 ! 9-3+1
allocate(farray3d(fa_shape(1), fa_shape(2), fa_shape(3)))
field = ESMF_FieldCreate(grid, farray3d, ESMF_INDEX_DELOCAL, &
ungriddedLBound=(/3/), ungriddedUBound=(/9/), &
totalLWidth=totalLWidth2d, totalUWidth=totalUWidth2d, &
gridToFieldMap=gridToFieldMap2d, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, an ESMF_Field is created from an ESMF_LocStream and typekind/rank. The location stream object is uniformly distributed in a 1 dimensional space on 4 DEs. The rank is 1 dimensional. Please refer to LocStream examples section for more information on LocStream creation.
locs = ESMF_LocStreamCreate(minIndex=1, maxIndex=16, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
field = ESMF_FieldCreate(locs, typekind=ESMF_TYPEKIND_I4, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, an ESMF_Field is created from an ESMF_LocStream and an ESMF_Arrayspec. The location stream object is uniformly distributed in a 1 dimensional space on 4 DEs. The arrayspec is 1 dimensional. Please refer to LocStream examples section for more information on LocStream creation.
locs = ESMF_LocStreamCreate(minIndex=1, maxIndex=16, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_I4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
field = ESMF_FieldCreate(locs, arrayspec, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, an ESMF_Field is created from an ESMF_Mesh and typekind/rank. The mesh object is on a Euclidean surface that is partitioned to a 2x2 rectangular space with 4 elements and 9 nodes. The nodal space is represented by a distgrid with 9 indices. Field is created on locally owned nodes on each PET. Therefore, the created Field has 9 data points globally. The mesh object can be represented by the picture below. For more information on Mesh creation, please see Section 33.3.1.
Mesh Ids
2.0 7 ------- 8 -------- 9
| | |
| 3 | 4 |
| | |
1.0 4 ------- 5 -------- 6
| | |
| 1 | 2 |
| | |
0.0 1 ------- 2 -------- 3
0.0 1.0 2.0
Node Ids at corners
Element Ids in centers
Mesh Owners
2.0 2 ------- 2 -------- 3
| | |
| 2 | 3 |
| | |
1.0 0 ------- 0 -------- 1
| | |
| 0 | 1 |
| | |
0.0 0 ------- 0 -------- 1
0.0 1.0 2.0
Node Owners at corners
Element Owners in centers
! Create Mesh structure in 1 step
mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
nodeIds=nodeIds, nodeCoords=nodeCoords, &
nodeOwners=nodeOwners, elementIds=elemIds,&
elementTypes=elemTypes, elementConn=elemConn, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Field is created on the 1 dimensional nodal distgrid. On
! each PET, Field is created on the locally owned nodes.
field = ESMF_FieldCreate(mesh, typekind=ESMF_TYPEKIND_I4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, an ESMF_Field is created from an ESMF_Mesh and an ESMF_Arrayspec. The mesh object is on a Euclidean surface that is partitioned to a 2x2 rectangular space with 4 elements and 9 nodes. The nodal space is represented by a distgrid with 9 indices. Field is created on locally owned nodes on each PET. Therefore, the created Field has 9 data points globally. The mesh object can be represented by the picture below. For more information on Mesh creation, please see Section 33.3.1.
! Create Mesh structure in 1 step
mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
nodeIds=nodeIds, nodeCoords=nodeCoords, &
nodeOwners=nodeOwners, elementIds=elemIds,&
elementTypes=elemTypes, elementConn=elemConn, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_I4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Field is created on the 1 dimensional nodal distgrid. On
! each PET, Field is created on the locally owned nodes.
field = ESMF_FieldCreate(mesh, arrayspec, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, an ESMF_Field is created from an ESMF_Mesh and an ESMF_Array. The mesh object is created in the previous example and the array object is retrieved from the field created in the previous example too.
call ESMF_MeshGet(mesh, nodalDistgrid=distgrid, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
array = ESMF_ArrayCreate(distgrid=distgrid, arrayspec=arrayspec, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! query the array from the previous example
call ESMF_FieldGet(field, array=array, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create a Field from a mesh and an array
field1 = ESMF_FieldCreate(mesh, array, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, an ESMF_Field is created from an ESMF_Mesh and an ESMF_ArraySpec. The mesh object is created in the previous example. The Field is also created with optional arguments such as ungridded dimensions and dimension mapping.
In this example, the mesh is mapped to the 2nd dimension of the ESMF_Field, with its first dimension being the ungridded dimension with bounds 1,3.
call ESMF_ArraySpecSet(arrayspec, 2, ESMF_TYPEKIND_I4, rc=rc)
field = ESMF_FieldCreate(mesh, arrayspec=arrayspec, gridToFieldMap=(/2/), &
ungriddedLBound=(/1/), ungriddedUBound=(/3/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example an ESMF_Field with replicated dimension is created from an ESMF_Grid and an ESMF_Arrayspec. A user can also use other ESMF_FieldCreate() methods to create replicated dimension Field, this example illustrates the key concepts and use of a replicated dimension Field.
Normally gridToFieldMap argument in ESMF_FieldCreate() should not contain 0 value entries. However, for Field with replicated dimension, a 0 entry in gridToFieldMap indicates the corresponding Grid dimension is replicated in the Field. In such a Field, the rank of the Field is no longer necessarily greater than its Grid rank. An example will make this clear. We will start by creating Distgrid and Grid.
! create 4D distgrid
distgrid = ESMF_DistGridCreate(minIndex=(/1,1,1,1/), &
maxIndex=(/6,4,6,4/), regDecomp=(/2,1,2,1/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create 4D grid on top of the 4D distgrid
grid = ESMF_GridCreate(distgrid=distgrid, name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create 3D arrayspec
call ESMF_ArraySpecSet(arrayspec, 3, ESMF_TYPEKIND_R8, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
In this example, a user creates a 3D Field with replicated dimension replicated along the 2nd and 4th dimension of its underlying 4D Grid. In addition, the 2nd dimension of the Field is ungridded (why?). The 1st and 3rd dimensions of the Field have halos.
! create field, 2nd and 4th dimensions of the Grid are replicated
field = ESMF_FieldCreate(grid, arrayspec, indexflag=ESMF_INDEX_DELOCAL, &
gridToFieldMap=(/1,0,2,0/), &
ungriddedLBound=(/1/), ungriddedUBound=(/4/), &
totalLWidth=(/1,1/), totalUWidth=(/4,5/), &
staggerloc=ESMF_STAGGERLOC_CORNER, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! get basic information from the field
call ESMF_FieldGet(field, grid=grid1, array=array, typekind=typekind, &
dimCount=dimCount, staggerloc=lstaggerloc, &
gridToFieldMap=lgridToFieldMap, ungriddedLBound=lungriddedLBound, &
ungriddedUBound=lungriddedUBound, totalLWidth=ltotalLWidth, &
totalUWidth=ltotalUWidth, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! get bounds information from the field
call ESMF_FieldGet(field, localDe=0, farrayPtr=farray, &
exclusiveLBound=felb, exclusiveUBound=feub, exclusiveCount=fec, &
computationalLBound=fclb, computationalUBound=fcub, &
computationalCount=fcc, totalLBound=ftlb, totalUBound=ftub, &
totalCount=ftc, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Next we verify that the field and array bounds agree with each other
call ESMF_ArrayGet(array, rank=arank, dimCount=adimCount, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
gridrank_repdim = 0
do i = 1, size(gridToFieldMap)
if(gridToFieldMap(i) == 0) gridrank_repdim = gridrank_repdim + 1
enddo
Number of undistributed dimension of the array X is computed from total rank of the array A, the dimension count of its underlying distgrid B and number of replicated dimension in the distgrid C. We have the following formula: X = A - (B - C)
allocate(audlb(arank-adimCount+gridrank_repdim), &
audub(arank-adimCount+gridrank_repdim))
call ESMF_ArrayGet(array, exclusiveLBound=aelb, exclusiveUBound=aeub, &
computationalLBound=aclb, computationalUBound=acub, &
totalLBound=atlb, totalUBound=atub, &
undistLBound=audlb, undistUBound=audub, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! verify the ungridded bounds from field match
! undistributed bounds from its underlying array
do i = 1, arank-adimCount
if(lungriddedLBound(i) .ne. audlb(i) ) &
rc = ESMF_FAILURE
enddo
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
do i = 1, arank-adimCount
if(lungriddedUBound(i) .ne. audub(i) ) &
rc = ESMF_FAILURE
enddo
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
We then verify the data in the replicated dimension Field can be updated and accessed.
do ik = ftlb(3), ftub(3)
do ij = ftlb(2), ftub(2)
do ii = ftlb(1), ftub(1)
farray(ii,ij,ik) = ii+ij*2+ik
enddo
enddo
enddo
! access and verify
call ESMF_FieldGet(field, localDe=0, farrayPtr=farray1, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
do ik = ftlb(3), ftub(3)
do ij = ftlb(2), ftub(2)
do ii = ftlb(1), ftub(1)
n = ii+ij*2+ik
if(farray1(ii,ij,ik) .ne. n ) rc = ESMF_FAILURE
enddo
enddo
enddo
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! release resources
call ESMF_FieldDestroy(field)
call ESMF_GridDestroy(grid)
call ESMF_DistGridDestroy(distgrid)
With the introduction of Field on arbitrarily distributed Grid, Field has two kinds of dimension count: one associated geometrical (or physical) dimensionality, the other one associated with its memory index space representation. Field and Grid dimCount reflect the physical index space of the objects. A new type of dimCount rank should be added to both of these entities. rank gives the number of dimensions of the memory index space of the objects. This would be the dimension of the pointer pulled out of Field and the size of the bounds vector, for example.
For non-arbitrary Grids rank=dimCount, but for grids and fields with arbitrary dimensions rank = dimCount - (number of Arb dims) + 1 (Internally Field can use the Arb info from the grid to create the mapping from the Field Array to the DistGrid)
When creating a Field size(GridToFieldMap)=dimCount for both Arb and Non-arb grids This array specifies the mapping of Field to Grid identically for both Arb and Nonarb grids If a zero occurs in an entry corresponding to any arbitrary dimension, then a zero must occur in every entry corresponding to an arbitrary dimension (i.e. all arbitrary dimensions must either be all replicated or all not replicated, they can't be broken apart).
In this example an ESMF_Field is created from an arbitrarily distributed ESMF_Grid and an ESMF_Arrayspec. A user can also use other ESMF_FieldCreate() methods to create such a Field, this example illustrates the key concepts and use of Field on arbitrary distributed Grid.
The Grid is 3 dimensional in physics index space but the first two dimension are collapsed into a single memory index space. Thus the result Field is 3D in physics index space and 2D in memory index space. This is made obvious with the 2D arrayspec used to create this Field.
! create a 3D grid with the first 2 dimensions collapsed
! and arbitrarily distributed
grid3d = ESMF_GridCreateNoPeriDim(coordTypeKind=ESMF_TYPEKIND_R8, &
minIndex=(/1,1,1/), maxIndex=(/xdim, ydim,zdim/), &
arbIndexList=localArbIndex,arbIndexCount=localArbIndexCount, &
name="arb3dgrid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create a 2D arrayspec
call ESMF_ArraySpecSet(arrayspec2D, rank=2, typekind=ESMF_TYPEKIND_R4, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create a 2D Field using the Grid and the arrayspec
field = ESMF_FieldCreate(grid3d, arrayspec2D, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field, rank=rank, dimCount=dimCount, &
rc=rc)
if (myPet .eq. 0) print *, 'Field rank, dimCount', &
rank, dimCount
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! verify that the dimension counts are correct
if (rank .ne. 2) correct = .false.
if (dimCount .ne. 3) correct = .false.
The next example is slightly more complicated in that the Field also contains ungridded dimension and its gridded dimension is replicated on the arbitrarily distributed dimension of the Grid.
The same 3D Grid and 2D arrayspec in the previous example are used but a gridToFieldMap argument is supplied to the ESMF_FieldCreate() call. The first 2 entries of the map are 0, the last (3rd) entry is 1. The 3rd dimension of the Grid is mapped to the first dimension of the Field, this dimension is then replicated on the arbitrarily distributed dimensions of the Grid. In addition, the Field also has one ungridded dimension. Thus the final dimension count of the Field is 2 in both physics and memory index space.
field = ESMF_FieldCreate(grid3d, arrayspec2D,gridToFieldMap=(/0,0,1/), &
ungriddedLBound=(/1/), ungriddedUBound=(/10/),rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field, rank=rank, dimCount=dimCount, &
rc=rc)
if (myPet .eq. 0) print *, 'Field rank, dimCount', &
rank, dimCount
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
if (rank .ne. 2) correct = .false.
if (dimCount .ne. 2) correct = .false.
This section describes the Field regrid methods. For an in depth description of ESMF regridding and the options available please see Section 24.2.
The basic flow of ESMF Field regridding is as follows. First a source and destination geometry object are created, depending on the regrid method they can be either a Grid, a Mesh, or a LocStream. Next Fields are built on the source and destination grid objects. These Fields are then passed into ESMF_FieldRegridStore(). The user can either get a sparse matrix from this call and/or a routeHandle. If the user gets the sparse matrix then they are responsible for deallocating it, but other than that can use it as they wish. The routeHandle can be used in the ESMF_FieldRegrid() call to perform the actual interpolation of data from the source to the destination field. This interpolation can be repeated for the same set of Fields as long as the coordinates at the staggerloc involved in the regridding in the associated grid object don't change. The same routeHandle can also be used between any pair of Fields which is weakly congruent to the pair used to create the routeHandle. Congruent Fields possess matching DistGrids and the shape of the local array tiles matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. You can apply the routehandle between any set of Fields weakly congruent to the original Fields used to create the routehandle without incurring an error. However, if you want the routehandle to be the same interpolation between the grid objects upon which the Fields are built as was calculated with the original ESMF_FieldRegridStore() call, then there are additional constraints on the grid objects. To be the same interpolation, the grid objects upon which the Fields are build must contain the same coordinates at the stagger locations involved in the regridding as the original source and destination Fields used in the ESMF_FieldRegridStore() call. The routehandle represents the interpolation between the grid objects as they were during the ESMF_FieldRegridStore() call. So if the coordinates at the stagger location in the grid objects change, a new call to ESMF_FieldRegridStore() is necessary to compute the interpolation between that new set of coordinates. When finished with the routeHandle ESMF_FieldRegridRelease() should be used to free the associated memory.
The following example demonstrates doing a regrid operation between two Fields.
! (Create source Grid, Mesh, or LocStream.)
! (Create srcField on the above.)
! (Create destination Grid, Mesh, or LocStream.)
! (Create dstField on the above.)
! Create the routeHandle which encodes the communication and
! information necessary for the regrid sparse matrix multiply.
call ESMF_FieldRegridStore(srcField=srcField, dstField=dstField, &
routeHandle=routeHandle, rc=localrc)
! Can loop here regridding from srcField to dstField
! do i=1,....
! (Put data into srcField)
! Use the routeHandle to regrid data from srcField to dstField.
! As described above, the same routeHandle can be used to
! regrid any source and destination Fields which are weakly
! congruent to the original srcField and dstField.
call ESMF_FieldRegrid(srcField, dstField, routeHandle, rc=localrc)
! (Use data in dstField) ! enddo ! Free the buffers and data associated with the routeHandle. call ESMF_FieldRegridRelease(routeHandle, rc=localrc)
call ESMF_FieldRegridStore(srcField=srcField, srcMaskValues=(/1/), &
dstField=dstField, dstMaskValues=(/1/), &
unmappedaction=ESMF_UNMAPPEDACTION_IGNORE, &
routeHandle=routeHandle, &
regridmethod=ESMF_REGRIDMETHOD_BILINEAR, &
rc=localrc)
The ESMF_FieldRegrid and ESMF_FieldRegridRelease calls may then be applied as in the previous example.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Create Source Mesh
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Create the Mesh structure.
! For brevity's sake, the code to fill the Mesh creation
! arrays is omitted from this example. However, here
! is a brief description of the arrays:
! srcNodeIds - the global ids for the src nodes
! srcNodeCoords - the coordinates for the src nodes
! srcNodeOwners - which PET owns each src node
! srcElemIds - the global ids of the src elements
! srcElemTypes - the topological shape of each src element
! srcElemConn - how to connect the nodes to form the elements
! in the source mesh
! Several examples of setting up these arrays can be seen in
! the Mesh Section "Mesh Creation".
srcMesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
nodeIds=srcNodeIds, nodeCoords=srcNodeCoords, &
nodeOwners=srcNodeOwners, elementIds=srcElemIds,&
elementTypes=srcElemTypes, elementConn=srcElemConn, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Create and Fill Source Field
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Set description of source Field
call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_R8, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create source Field
srcField = ESMF_FieldCreate(srcMesh, arrayspec, &
name="source", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Get source Field data pointer to put data into
call ESMF_FieldGet(srcField, 0, fptr1D, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Get number of local nodes to allocate space
! to hold local node coordinates
call ESMF_MeshGet(srcMesh, &
numOwnedNodes=numOwnedNodes, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Allocate space to hold local node coordinates
! (spatial dimension of Mesh*number of local nodes)
allocate(ownedNodeCoords(2*numOwnedNodes))
! Get local node coordinates
call ESMF_MeshGet(srcMesh, &
ownedNodeCoords=ownedNodeCoords, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Set the source Field to the function 20.0+x+y
do i=1,numOwnedNodes
! Get coordinates
x=ownedNodeCoords(2*i-1)
y=ownedNodeCoords(2*i)
! Set source function
fptr1D(i) = 20.0+x+y
enddo
! Deallocate local node coordinates
deallocate(ownedNodeCoords)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Create Destination Mesh
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Create the Mesh structure.
! For brevity's sake, the code to fill the Mesh creation
! arrays is omitted from this example. However, here
! is a brief description of the arrays:
! dstNodeIds - the global ids for the dst nodes
! dstNodeCoords - the coordinates for the dst nodes
! dstNodeOwners - which PET owns each dst node
! dstElemIds - the global ids of the dst elements
! dstElemTypes - the topological shape of each dst element
! dstElemConn - how to connect the nodes to form the elements
! in the destination mesh
! Several examples of setting up these arrays can be seen in
! the Mesh Section "Mesh Creation".
dstMesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
nodeIds=dstNodeIds, nodeCoords=dstNodeCoords, &
nodeOwners=dstNodeOwners, elementIds=dstElemIds,&
elementTypes=dstElemTypes, elementConn=dstElemConn, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Create Destination Field
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Set description of source Field
call ESMF_ArraySpecSet(arrayspec, 1, ESMF_TYPEKIND_R8, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create destination Field
dstField = ESMF_FieldCreate(dstMesh, arrayspec, &
name="destination", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Do Regrid
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Compute RouteHandle which contains the regrid operation
call ESMF_FieldRegridStore( &
srcField, &
dstField=dstField, &
routeHandle=routeHandle, &
regridmethod=ESMF_REGRIDMETHOD_BILINEAR, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Perform Regrid operation moving data from srcField to dstField
call ESMF_FieldRegrid(srcField, dstField, routeHandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! dstField now contains the interpolated data.
! If the Meshes don't change, then routeHandle
! may be used repeatedly to interpolate from
! srcField to dstField.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! User code to use the routeHandle, Fields, and
! Meshes goes here before they are freed below.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Free the objects created in the example.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Free the RouteHandle
call ESMF_FieldRegridRelease(routeHandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Free the Fields
call ESMF_FieldDestroy(srcField, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldDestroy(dstField, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Free the Meshes
call ESMF_MeshDestroy(dstMesh, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_MeshDestroy(srcMesh, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
User can use ESMF_FieldGather interface to gather Field data from multiple PETS onto a single root PET. This interface is overloaded by type, kind, and rank.
Note that the implementation of Scatter and Gather is not sequence index based. If the Field is built on arbitrarily distributed Grid, Mesh, LocStream or XGrid, Gather will not gather data to rootPet from source data points corresponding to the sequence index on the rootPet. Instead Gather will gather a contiguous memory range from source PET to rootPet. The size of the memory range is equal to the number of data elements on the source PET. Vice versa for the Scatter operation. In this case, the user should use ESMF_FieldRedist to achieve the same data operation result. For examples how to use ESMF_FieldRedist to perform Gather and Scatter, please refer to 26.3.31 and 26.3.30.
In this example, we first create a 2D Field, then use ESMF_FieldGather to collect all the data in this Field into a data pointer on PET 0.
! Get current VM and pet number
call ESMF_VMGetCurrent(vm, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_VMGet(vm, localPet=lpe, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Create a 2D Grid and use this grid to create a Field
! farray is the Fortran data array that contains data on each PET.
grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), &
name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
field = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field, farrayPtr=fptr, rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
!---------Initialize pet specific field data----------------
! 1 5 10
! 1 +--------+---------+
! | | |
! | 0 | 1 |
! | | |
! 10 +--------+---------+
! | | |
! | 2 | 3 |
! | | |
! 20 +--------+---------+
fptr = lpe
! allocate the Fortran data array on PET 0 to store gathered data
if(lpe .eq. 0) then
allocate (farrayDst(10,20))
else
allocate (farrayDst(0,0))
end if
call ESMF_FieldGather(field, farrayDst, rootPet=0, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! check that the values gathered on rootPet are correct
if(lpe .eq. 0) then
do i = 1, 5
do j = 1, 10
if(farrayDst(i, j) .ne. 0) localrc=ESMF_FAILURE
enddo
enddo
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
do i = 6, 10
do j = 1, 10
if(farrayDst(i, j) .ne. 1) localrc=ESMF_FAILURE
enddo
enddo
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
do i = 1, 5
do j = 11, 20
if(farrayDst(i, j) .ne. 2) localrc=ESMF_FAILURE
enddo
enddo
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
do i = 6, 10
do j = 11, 20
if(farrayDst(i, j) .ne. 3) localrc=ESMF_FAILURE
enddo
enddo
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
! destroy all objects created in this example to prevent memory leak
call ESMF_FieldDestroy(field, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridDestroy(grid, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
if(lpe .eq. 0) deallocate(farrayDst)
User can use ESMF_FieldScatter interface to scatter Field data from root PET onto its set of joint PETs. This interface is overloaded by type, kind, and rank.
In this example, we first create a 2D Field, then use ESMF_FieldScatter to scatter the data from a data array located on PET 0 onto this Field.
! Create a 2D Grid and use this grid to create a Field
! farray is the Fortran data array that contains data on each PET.
grid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), &
name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
field = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! initialize values to be scattered
! 1 5 10
! 1 +--------+---------+
! | | |
! | 0 | 1 |
! | | |
! 10 +--------+---------+
! | | |
! | 2 | 3 |
! | | |
! 20 +--------+---------+
if(lpe .eq. 0) then
allocate(farraySrc(10,20))
farraySrc(1:5,1:10) = 0
farraySrc(6:10,1:10) = 1
farraySrc(1:5,11:20) = 2
farraySrc(6:10,11:20) = 3
else
allocate (farraySrc(0,0))
endif
! scatter the data onto individual PETs of the Field
call ESMF_FieldScatter(field, farraySrc, rootPet=0, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(field, localDe=0, farrayPtr=fptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! verify that the scattered data is properly distributed
do i = lbound(fptr, 1), ubound(fptr, 1)
do j = lbound(fptr, 2), ubound(fptr, 2)
if(fptr(i, j) .ne. lpe) localrc = ESMF_FAILURE
enddo
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
enddo
! destroy all objects created in this example to prevent memory leak
call ESMF_FieldDestroy(field, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridDestroy(grid, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
if(lpe .eq. 0) deallocate(farraySrc)
User can use ESMF_FieldRedist interface to redistribute data from source Field to destination Field. This interface is overloaded by type and kind; In the version of ESMF_FieldRedist without factor argument, a default value of 1 is used.
In this example, we first create two 1D Fields, a source Field and a destination Field. Then we use ESMF_FieldRedist to redistribute data from source Field to destination Field.
! Get current VM and pet number
call ESMF_VMGetCurrent(vm, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_VMGet(vm, localPet=localPet, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create grid
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
regDecomp=(/4/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
grid = ESMF_GridCreate(distgrid=distgrid, &
name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create srcField
! +--------+--------+--------+--------+
! 0 1 2 3 ! value
! 1 4 8 12 16 ! bounds
srcField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(srcField, farrayPtr=srcfptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
srcfptr(:) = localPet
! create dstField
! +--------+--------+--------+--------+
! 0 0 0 0 ! value
! 1 4 8 12 16 ! bounds
dstField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldGet(dstField, farrayPtr=dstfptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
dstfptr(:) = 0
! perform redist
! 1. setup routehandle from source Field to destination Field
call ESMF_FieldRedistStore(srcField, dstField, routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! 2. use precomputed routehandle to redistribute data
call ESMF_FieldRedist(srcfield, dstField, routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! verify redist
call ESMF_FieldGet(dstField, localDe=0, farrayPtr=fptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Verify that the redistributed data in dstField is correct.
! Before the redist op, the dst Field contains all 0.
! The redist op reset the values to the PE value, verify this is the case.
do i = lbound(fptr, 1), ubound(fptr, 1)
if(fptr(i) .ne. localPet) localrc = ESMF_FAILURE
enddo
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Field redistribution can also be performed between weakly congruent Fields. In this case, source and destination Fields can have ungridded dimensions with size different from the Field pair used to compute the routehandle.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=2, rc=rc)
Create two fields with ungridded dimensions using the Grid created previously. The new Field pair has matching number of elements. The ungridded dimension is mapped to the first dimension of either Field.
srcFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)
dstFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)
Using the previously computed routehandle, weakly congruent Fields can be redistributed.
call ESMF_FieldRedist(srcfieldA, dstFieldA, routehandle, rc=rc)
call ESMF_FieldRedistRelease(routehandle, rc=rc)
User can use ESMF_FieldRedist interface to redistribute data from source Field to destination Field, where the destination Field is built on an arbitrarily distributed structure, e.g. ESMF_Mesh. The underlying mechanism is explained in section 28.2.18.
In this example, we will create 2 one dimensional Fields, the src Field has a regular decomposition and holds all its data on a single PET, in this case PET 0. The destination Field is built on a Mesh which is itself built on an arbitrarily distributed distgrid. Then we use ESMF_FieldRedist to redistribute data from source Field to destination Field, similar to a traditional scatter operation.
The src Field only has data on PET 0 where it is sequentially initialized, i.e. 1,2,3...This data will be redistributed (or scattered) from PET 0 to the destination Field arbitrarily distributed on all the PETs.
! a one dimensional grid whose elements are all located on PET 0
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/9/), &
regDecomp=(/1/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
grid = ESMF_GridCreate(distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
srcField = ESMF_FieldCreate(grid, typekind=ESMF_TYPEKIND_I4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! initialize the source data
if (localPet == 0) then
call ESMF_FieldGet(srcField, farrayPtr=srcfptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
do i = 1, 9
srcfptr(i) = i
enddo
endif
For more information on Mesh creation, user can refer to Mesh examples section or Field creation on Mesh example for more details.
! Create Mesh structure
mesh=ESMF_MeshCreate(parametricDim=2,spatialDim=2, &
nodeIds=nodeIds, nodeCoords=nodeCoords, &
nodeOwners=nodeOwners, elementIds=elemIds,&
elementTypes=elemTypes, elementConn=elemConn, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Create the destination Field on the Mesh that is arbitrarily distributed on all the PETs.
dstField = ESMF_FieldCreate(mesh, typekind=ESMF_TYPEKIND_I4, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Perform the redistribution from source Field to destination Field.
call ESMF_FieldRedistStore(srcField, dstField, &
routehandle=routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_FieldRedist(srcField, dstField, routehandle=routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
We can now verify that the sequentially initialized source data is scattered on to the destination Field. The data has been scattered onto the destination Field with the following distribution.
4 elements on PET 0: 1 2 4 5 2 elements on PET 1: 3 6 2 elements on PET 2: 7 8 1 element on PET 3: 9Because the redistribution is index based, the elements also corresponds to the index space of Mesh in the destination Field.
call ESMF_FieldGet(dstField, farrayPtr=dstfptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
The scatter operation is successful. Since the routehandle computed with ESMF_FieldRedistStore can be reused, user can use the same routehandle to scatter multiple source Fields from a single PET to multiple destination Fields distributed on all PETs. The gathering operation is just the opposite of the demonstrated scattering operation, where a user would redist from a source Field distributed on multiple PETs to a destination Field that only has data storage on a single PET.
Now it's time to release all the resources.
call ESMF_FieldRedistRelease(routehandle=routehandle, rc=rc)
Similarly, one can use the same approach to gather the data from an arbitrary distribution to a non-arbitrary distribution. This concept is demonstrated by using the previous Fields but the data operation is reversed. This time data is gathered from the Field built on the mesh to the Field that has only data allocation on rootPet.
First a FieldRedist routehandle is created from the Field built on Mesh to the Field that has only data allocation on rootPet.
call ESMF_FieldRedistStore(dstField, srcField, routehandle=routehandle, &
rc=rc)
Perform FieldRedist, this will gather the data points from the Field built on mesh to the data pointer on the rootPet (default to 0) stored in the srcField.
call ESMF_FieldRedist(dstField, srcField, routehandle=routehandle, rc=rc)
Release the routehandle used for the gather operation.
call ESMF_FieldRedistRelease(routehandle=routehandle, rc=rc)
A user can use ESMF_FieldSMM() interface to perform sparse matrix multiplication from source Field to destination Field. This interface is overloaded by type and kind;
In this example, we first create two 1D Fields, a source Field and a destination Field. Then we use ESMF_FieldSMM to perform sparse matrix multiplication from source Field to destination Field.
The source and destination Field data are arranged such that each of the 4 PETs has 4
data elements. Moreover, the source Field has all its data elements initialized to a linear
function based on local PET number.
Then collectively on each PET, a SMM according to the following formula
is preformed:
Because source Field data are initialized to a linear function based on local PET number, the formula predicts that the result destination Field data on each PET is 1,2,3,4. This is verified in the example.
Section 28.2.17 provides a detailed discussion of the sparse matrix multiplication operation implemented in ESMF.
! Get current VM and pet number
call ESMF_VMGetCurrent(vm, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_VMGet(vm, localPet=lpe, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create distgrid and grid
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
regDecomp=(/4/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
grid = ESMF_GridCreate(distgrid=distgrid, &
name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridGetFieldBounds(grid, localDe=0, totalCount=fa_shape, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create src\_farray, srcArray, and srcField
! +--------+--------+--------+--------+
! 1 2 3 4 ! value
! 1 4 8 12 16 ! bounds
allocate(src_farray(fa_shape(1)) )
src_farray = lpe+1
srcArray = ESMF_ArrayCreate(distgrid, src_farray, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
srcField = ESMF_FieldCreate(grid, srcArray, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! create dst_farray, dstArray, and dstField
! +--------+--------+--------+--------+
! 0 0 0 0 ! value
! 1 4 8 12 16 ! bounds
allocate(dst_farray(fa_shape(1)) )
dst_farray = 0
dstArray = ESMF_ArrayCreate(distgrid, dst_farray, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
dstField = ESMF_FieldCreate(grid, dstArray, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! perform sparse matrix multiplication
! 1. setup routehandle from source Field to destination Field
! initialize factorList and factorIndexList
allocate(factorList(4))
allocate(factorIndexList(2,4))
factorList = (/1,2,3,4/)
factorIndexList(1,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
factorIndexList(2,:) = (/lpe*4+1,lpe*4+2,lpe*4+3,lpe*4+4/)
call ESMF_FieldSMMStore(srcField, dstField, routehandle, &
factorList, factorIndexList, rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! 2. use precomputed routehandle to perform SMM
call ESMF_FieldSMM(srcfield, dstField, routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! verify sparse matrix multiplication
call ESMF_FieldGet(dstField, localDe=0, farrayPtr=fptr, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! Verify that the result data in dstField is correct.
! Before the SMM op, the dst Field contains all 0.
! The SMM op reset the values to the index value, verify this is the case.
! +--------+--------+--------+--------+
! 1 2 3 4 2 4 6 8 3 6 9 12 4 8 12 16 ! value
! 1 4 8 12 16 ! bounds
do i = lbound(fptr, 1), ubound(fptr, 1)
if(fptr(i) /= i*(lpe+1)) rc = ESMF_FAILURE
enddo
Field sparse matrix matmul can also be performed between weakly congruent Fields. In this case, source and destination Fields can have ungridded dimensions with size different from the Field pair used to compute the routehandle.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=2, rc=rc)
Create two fields with ungridded dimensions using the Grid created previously. The new Field pair has matching number of elements. The ungridded dimension is mapped to the first dimension of either Field.
srcFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)
dstFieldA = ESMF_FieldCreate(grid, arrayspec, gridToFieldMap=(/2/), &
ungriddedLBound=(/1/), ungriddedUBound=(/10/), rc=rc)
Using the previously computed routehandle, weakly congruent Fields can perform sparse matrix matmul.
call ESMF_FieldSMM(srcfieldA, dstFieldA, routehandle, rc=rc)
! release route handle
call ESMF_FieldSMMRelease(routehandle, rc=rc)
In the following discussion, we demonstrate how to set up a SMM routehandle between a pair of Fields that are different in number of gridded dimensions and the size of those gridded dimensions. The source Field has a 1D decomposition with 16 total elements; the destination Field has a 2D decomposition with 12 total elements. For ease of understanding of the actual matrix calculation, a global indexing scheme is used.
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/16/), &
indexflag=ESMF_INDEX_GLOBAL, &
regDecomp=(/4/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
grid = ESMF_GridCreate(distgrid=distgrid, &
indexflag=ESMF_INDEX_GLOBAL, &
name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_GridGetFieldBounds(grid, localDe=0, totalLBound=tlb, &
totalUBound=tub, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
create 1D src_farray, srcArray, and srcField
+ PET0 + PET1 + PET2 + PET3 +
+--------+--------+--------+--------+
1 2 3 4 ! value
1 4 8 12 16 ! bounds of seq indices
allocate(src_farray2(tlb(1):tub(1)) )
src_farray2 = lpe+1
srcArray = ESMF_ArrayCreate(distgrid, src_farray2, &
indexflag=ESMF_INDEX_GLOBAL, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
!print *, lpe, '+', tlb, tub, '+', src_farray2
srcField = ESMF_FieldCreate(grid, srcArray, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Create 2D dstField on the following distribution (numbers are the sequence indices):
+ PET0 + PET1 + PET2 + PET3 + +--------+--------+--------+--------+ | | | | | | 1 | 4 | 7 | 10 | | | | | | +--------+--------+--------+--------+ | | | | | | 2 | 5 | 8 | 11 | | | | | | +--------+--------+--------+--------+ | | | | | | 3 | 6 | 9 | 12 | | | | | | +--------+--------+--------+--------+
! Create the destination Grid
dstGrid = ESMF_GridCreateNoPeriDim(minIndex=(/1,1/), maxIndex=(/3,4/), &
indexflag = ESMF_INDEX_GLOBAL, &
regDecomp = (/1,4/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
dstField = ESMF_FieldCreate(dstGrid, typekind=ESMF_TYPEKIND_R4, &
indexflag=ESMF_INDEX_GLOBAL, &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Perform sparse matrix multiplication 
= 
* 
First setup routehandle from source Field to destination Field
with prescribed factorList and factorIndexList.
The sparse matrix is of size 12x16, however only the following entries are filled:
M(3,1) = 0.1 M(3,10) = 0.4 M(8,2) = 0.25 M(8,16) = 0.5 M(12,1) = 0.3 M(12,16) = 0.7
By the definition of matrix calculation, the 8th element on PET2 in the dstField equals to 0.25*srcField(2) + 0.5*srcField(16) = 0.25*1+0.5*4=2.25 For simplicity, we will load the factorList and factorIndexList on PET 0 and 1, the SMMStore engine will load balance the parameters on all 4 PETs internally for optimal performance.
if(lpe == 0) then
allocate(factorList(3), factorIndexList(2,3))
factorList=(/0.1,0.4,0.25/)
factorIndexList(1,:)=(/1,10,2/)
factorIndexList(2,:)=(/3,3,8/)
call ESMF_FieldSMMStore(srcField, dstField, routehandle=routehandle, &
factorList=factorList, factorIndexList=factorIndexList, rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
else if(lpe == 1) then
allocate(factorList(3), factorIndexList(2,3))
factorList=(/0.5,0.3,0.7/)
factorIndexList(1,:)=(/16,1,16/)
factorIndexList(2,:)=(/8,12,12/)
call ESMF_FieldSMMStore(srcField, dstField, routehandle=routehandle, &
factorList=factorList, factorIndexList=factorIndexList, rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
else
call ESMF_FieldSMMStore(srcField, dstField, routehandle=routehandle, &
rc=localrc)
if (localrc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
! 2. use precomputed routehandle to perform SMM
call ESMF_FieldSMM(srcfield, dstField, routehandle=routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
ESMF_FieldHalo() interface can be used to perform halo update of a Field. This
eases communication programming from a user perspective. By definition, user
program only needs to update locally owned exclusive region in each domain, then call
FieldHalo to communicate the values in the halo region from/to neighboring domain elements.
In this example, we solve a 1D heat transfer problem:
with the
initial condition 
and boundary conditions
.
The temperature field
is represented by a ESMF_Field. A finite difference explicit time stepping scheme is employed.
During each time step, FieldHalo update is called to communicate values in the halo region
to neighboring domain elements. The steady state (as
) solution
is a linear temperature profile along 
. The numerical solution is an approximation of
the steady state solution. It can be verified to represent a linear temperature profile.
Section 28.2.14 provides a discussion of the halo operation implemented in ESMF_Array.
! create 1D distgrid and grid decomposed according to the following diagram:
! +------------+ +----------------+ +---------------+ +--------------+
! | DE 0 | | | | DE 1 | | | | DE 2 | | | | DE 3 |
! | 1 x 16 | | | | 1 x 16 | | | | 1 x 16 | | | | 1 x 16 |
! | | 1|<->|1 | | 1|<->|1 | | 1|<->|1 | |
! | | | | | | | | | | | | | |
! +------------+ +----------------+ +---------------+ +--------------+
distgrid = ESMF_DistGridCreate(minIndex=(/1/), maxIndex=(/npx/), &
regDecomp=(/4/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
grid = ESMF_GridCreate(distgrid=distgrid, name="grid", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
! set up initial condition and boundary conditions of the
! temperature Field
if(lpe == 0) then
allocate(fptr(17), tmp_farray(17))
fptr = 20.
fptr(1) = 10.
tmp_farray(1) = 10.
startx = 2
endx = 16
field = ESMF_FieldCreate(grid, fptr, totalUWidth=(/1/), &
name="temperature", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
else if(lpe == 3) then
allocate(fptr(17), tmp_farray(17))
fptr = 20.
fptr(17) = 40.
tmp_farray(17) = 40.
startx = 2
endx = 16
field = ESMF_FieldCreate(grid, fptr, totalLWidth=(/1/), &
name="temperature", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
else
allocate(fptr(18), tmp_farray(18))
fptr = 20.
startx = 2
endx = 17
field = ESMF_FieldCreate(grid, fptr, &
totalLWidth=(/1/), totalUWidth=(/1/), name="temperature", rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
! compute the halo update routehandle of the decomposed temperature Field
call ESMF_FieldHaloStore(field, routehandle=routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
dt = 0.01
dx = 1./npx
alpha = 0.1
! Employ explicit time stepping
! Solution converges after about 9000 steps based on apriori knowledge.
! The result is a linear temperature profile stored in field.
do iter = 1, 9000
! only elements in the exclusive region are updated locally
! in each domain
do i = startx, endx
tmp_farray(i) = &
fptr(i)+alpha*alpha*dt/dx/dx*(fptr(i+1)-2.*fptr(i)+fptr(i-1))
enddo
fptr = tmp_farray
! call halo update to communicate the values in the halo region to
! neighboring domains
call ESMF_FieldHalo(field, routehandle=routehandle, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
enddo
! release the halo routehandle
call ESMF_FieldHaloRelease(routehandle, rc=rc)
INTERFACE:
interface assignment(=) field1 = field2ARGUMENTS:
type(ESMF_Field) :: field1 type(ESMF_Field) :: field2STATUS:
DESCRIPTION:
Assign field1 as an alias to the same ESMF Field object in memory as field2. If field2 is invalid, then field1 will be equally invalid after the assignment.
The arguments are:
INTERFACE:
interface operator(==) if (field1 == field2) then ... endif OR result = (field1 == field2)RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_Field), intent(in) :: field1 type(ESMF_Field), intent(in) :: field2STATUS:
DESCRIPTION:
Test whether field1 and field2 are valid aliases to the same ESMF Field object in memory. For a more general comparison of two ESMF Fields, going beyond the simple alias test, the ESMF_FieldMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
interface operator(/=) if (field1 /= field2) then ... endif OR result = (field1 /= field2)RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_Field), intent(in) :: field1 type(ESMF_Field), intent(in) :: field2STATUS:
DESCRIPTION:
Test whether field1 and field2 are not valid aliases to the same ESMF Field object in memory. For a more general comparison of two ESMF Fields, going beyond the simple alias test, the ESMF_FieldMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
subroutine ESMF_FieldCopy(fieldOut, fieldIn, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: fieldOut
type(ESMF_Field), intent(in) :: fieldIn
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Copy data from one ESMF_Field object to another.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateGridTKR(grid, typekind, &
indexflag, staggerloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateGridTKRARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type(ESMF_TypeKind_Flag),intent(in) :: typekind
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Index_Flag), intent(in), optional :: indexflag
type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.4.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateGridArraySpec(grid, arrayspec, &
indexflag, staggerloc, gridToFieldMap, ungriddedLBound, &
ungriddedUBound, totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateGridArraySpecARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type(ESMF_ArraySpec), intent(in) :: arrayspec
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Index_Flag), intent(in), optional :: indexflag
type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.5.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateGridArray(grid, array, datacopyflag, &
staggerloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
totalLWidth, totalUWidth, name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateGridArrayARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
type(ESMF_Array), intent(in) :: array
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
character (len = *), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateGridData<rank><type><kind>(grid, & farray, indexflag, datacopyflag, staggerloc, & gridToFieldMap, ungriddedLBound, ungriddedUBound, & totalLWidth, totalUWidth, name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateGridData<rank><type><kind>ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_StaggerLoc), intent(in), optional :: staggerloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Create an ESMF_Field from a fortran data array and ESMF_Grid. The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed. For examples and associated documentations using this method see section 26.3.10, 26.3.12, 26.3.13, 26.3.14, and 26.3.9.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateGridDataPtr<rank><type><kind>(grid, & farrayPtr, datacopyflag, staggerloc, gridToFieldMap, & totalLWidth, totalUWidth, name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateGridDataPtr<rank><type><kind>ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_StaggerLoc), intent(in), optional :: staggerloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Create an ESMF_Field from a fortran data pointer and ESMF_Grid. The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
For examples and associated documentations using this method see section 26.3.11, 26.3.12, 26.3.13, 26.3.14, and 26.3.9.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateLSTKR(locstream, typekind, &
gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateLSTKRARGUMENTS:
type(ESMF_LocStream), intent(in) :: locstream
type(ESMF_TypeKind_Flag),intent(in) :: typekind
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.15.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateLSArraySpec(locstream, arrayspec, &
gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateLSArraySpecARGUMENTS:
type(ESMF_LocStream), intent(in) :: locstream
type(ESMF_ArraySpec), intent(in) :: arrayspec
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.16.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateLSArray(locstream, array, &
datacopyflag, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateLSArrayARGUMENTS:
type(ESMF_LocStream), intent(in) :: locstream
type(ESMF_Array), intent(in) :: array
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len = *), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateLSData<rank><type><kind>(locstream, farray, & indexflag, datacopyflag, gridToFieldMap, ungriddedLBound, & ungriddedUBound, name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateLSData<rank><type><kind>ARGUMENTS:
type(ESMF_LocStream), intent(in) :: locstream <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcDESCRIPTION:
Create an ESMF_Field from a fortran data array and ESMF_LocStream. The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateLSDataPtr<rank><type><kind>(locstream, & farrayPtr, datacopyflag, gridToFieldMap, & name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateLSDataPtr<rank><type><kind>ARGUMENTS:
type(ESMF_LocStream), intent(in) :: locstream <type> (ESMF_KIND_<kind>),pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: gridToFieldMap(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcDESCRIPTION:
Create an ESMF_Field from a fortran data pointer and ESMF_LocStream. The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateMeshTKR(mesh, typekind, &
meshloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateMeshTKRARGUMENTS:
type(ESMF_Mesh), intent(in) :: mesh
type(ESMF_TypeKind_Flag), intent(in) :: typekind
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_MeshLoc), intent(in), optional :: meshloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.17.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateMeshArraySpec(mesh, arrayspec, &
meshloc, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateMeshArraySpecARGUMENTS:
type(ESMF_Mesh), intent(in) :: mesh
type(ESMF_ArraySpec), intent(in) :: arrayspec
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_MeshLoc), intent(in), optional :: meshloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.18 and 26.3.20.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateMeshArray(mesh, array, &
datacopyflag, meshloc, &
gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateMeshArrayARGUMENTS:
type(ESMF_Mesh), intent(in) :: mesh
type(ESMF_Array), intent(in) :: array
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
type(ESMF_MeshLoc), intent(in), optional :: meshloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len = *), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.19.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateMeshData<rank><type><kind>(mesh, & farray, indexflag, datacopyflag, meshloc, & gridToFieldMap, ungriddedLBound, ungriddedUBound, name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateMeshData<rank><type><kind>ARGUMENTS:
type(ESMF_Mesh), intent(in) :: mesh <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_MeshLoc), intent(in), optional :: meshloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcDESCRIPTION:
Create an ESMF_Field from a fortran data array and ESMF_Mesh. The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateMeshDataPtr<rank><type><kind>(mesh, & farrayPtr, datacopyflag, meshloc, gridToFieldMap, & name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateMeshDataPtr<rank><type><kind>ARGUMENTS:
type(ESMF_Mesh), intent(in) :: mesh <type> (ESMF_KIND_<kind>),pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_MeshLoc), intent(in), optional :: meshloc integer, intent(in), optional :: gridToFieldMap(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcDESCRIPTION:
Create an ESMF_Field from a fortran data pointer and ESMF_Mesh. The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateXGTKR(xgrid, typekind, &
xgridside, gridindex, &
gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateXGTKRARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid
type(ESMF_TypeKind_Flag), intent(in) :: typekind
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside
integer, intent(in), optional :: gridindex
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.15.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateXGArraySpec(xgrid, arrayspec, &
xgridside, gridindex, &
gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateXGArraySpecARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid
type(ESMF_ArraySpec), intent(in) :: arrayspec
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_XGridSide_Flag), intent(in), optional :: xgridSide
integer, intent(in), optional :: gridIndex
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field and allocate space internally for an ESMF_Array. Return a new ESMF_Field. For an example and associated documentation using this method see section 26.3.16.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate()
function ESMF_FieldCreateXGArray(xgrid, array, &
datacopyflag, xgridside, gridindex, &
gridToFieldMap, ungriddedLBound, ungriddedUBound, &
name, rc)
RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateXGArrayARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid
type(ESMF_Array), intent(in) :: array
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag
type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside
integer, intent(in), optional :: gridindex
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
character (len = *), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Create an ESMF_Field. This version of creation assumes the data exists already and is being passed in through an ESMF_Array. For an example and associated documentation using this method see section 26.3.6.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateXGData<rank><type><kind>(xgrid, & farray, indexflag, datacopyflag, xgridside, gridindex, & gridToFieldMap, ungriddedLBound, ungriddedUBound, name,& rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateXGData<rank><type><kind>ARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside integer, intent(in), optional :: gridindex integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcDESCRIPTION:
Create an ESMF_Field from a fortran data array and ESMF_Xgrid. The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldCreate() function ESMF_FieldCreateXGDataPtr<rank><type><kind>(xgrid, farrayPtr, & datacopyflag, xgridside, & gridindex, gridToFieldMap, name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldCreateXGDataPtr<rank><type><kind>ARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside integer, intent(in), optional :: gridindex integer, intent(in), optional :: gridToFieldMap(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcDESCRIPTION:
Create an ESMF_Field from a fortran data pointer and ESMF_Xgrid. The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
subroutine ESMF_FieldDestroy(field, noGarbage, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: noGarbage
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Destroys the ESMF_Field, releasing the resources associated with the object.
If an ESMF_Grid is associated with field, it will not be released.
By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.
The arguments are:
It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompAS(field, arrayspec, & indexflag, gridToFieldMap, & ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_ArraySpec), intent(in) :: arrayspec -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_Index_Flag), intent(in), optional :: indexflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Complete an ESMF_Field and allocate space internally for an ESMF_Array based on arrayspec. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompTK(field, typekind, & indexflag, gridToFieldMap, & ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_TypeKind_Flag), intent(in) :: typekind -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_Index_Flag), intent(in), optional :: indexflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Complete an ESMF_Field and allocate space internally for an ESMF_Array based on typekind. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.
For an example and associated documentation using this method see section 26.3.7.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyComp<rank><type><kind>(field, & farray, indexflag, datacopyflag, gridToFieldMap, & ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Complete an ESMF_Field and allocate space internally for an ESMF_Array based on typekind. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.
The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
For an example and associated documentation using this method see section 26.3.8.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompPtr<rank><type><kind>(field, & farrayPtr, datacopyflag, gridToFieldMap, & totalLWidth, totalUWidth, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Complete an ESMF_Field and allocate space internally for an ESMF_Array based on typekind. The input ESMF_Field must have a status of ESMF_FIELDSTATUS_GRIDSET. After this call the completed ESMF_Field has a status of ESMF_FIELDSTATUS_COMPLETE.
The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompGrid<rank><type><kind>(field, grid, & farray, indexflag, datacopyflag, staggerloc, gridToFieldMap, & ungriddedLBound, ungriddedUBound, totalLWidth, totalUWidth, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_Grid), intent(in) :: grid <type> (ESMF_KIND_<kind>),intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_STAGGERLOC), intent(in), optional :: staggerloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompGridPtr<rank><type><kind>(field, grid, & farrayPtr, datacopyflag, staggerloc, gridToFieldMap, & totalLWidth, totalUWidth, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_Grid), intent(in) :: grid <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_STAGGERLOC), intent(in), optional :: staggerloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management. The fortran data pointer inside ESMF_Field can be queried and deallocated when
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompLS<rank><type><kind>(field, locstream, & farray, indexflag, datacopyflag, gridToFieldMap, & ungriddedLBound, ungriddedUBound, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_LocStream), intent(in) :: locstream <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(out), optional :: rcDESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompLSPtr<rank><type><kind>(field, locstream, & farrayPtr, datacopyflag, gridToFieldMap, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_LocStream), intent(in) :: locstream <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: gridToFieldMap(:) integer, intent(out), optional :: rcDESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompMesh<rank><type><kind>(field, mesh, & farray, indexflag, datacopyflag, meshloc, & gridToFieldMap, ungriddedLBound, ungriddedUBound, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_Mesh), intent(in) :: mesh <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_MeshLoc), intent(in), optional :: meshloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(out), optional :: rcDESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompMeshPtr<rank><type><kind>(field, mesh, & farrayPtr, datacopyflag, meshloc, gridToFieldMap, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_Mesh), intent(in) :: mesh <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_MeshLoc), intent(in), optional :: meshloc integer, intent(in), optional :: gridToFieldMap(:) integer, intent(out), optional :: rcDESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompXG<rank><type><kind>(field, xgrid, & farray, indexflag, datacopyflag, xgridside, gridindex, & gridToFieldMap, & ungriddedLBound, ungriddedUBound, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_XGrid), intent(in) :: xgrid <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside integer, intent(in), optional :: gridindex integer, intent(in), optional :: gridToFieldMap(:) integer, intent(in), optional :: ungriddedLBound(:) integer, intent(in), optional :: ungriddedUBound(:) integer, intent(out), optional :: rcDESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried but deallocating the retrieved data pointer is not allowed.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptyComplete() subroutine ESMF_FieldEmptyCompXGPtr<rank><type><kind>(field, xgrid, & farrayPtr, xgridside, gridindex, & datacopyflag, gridToFieldMap, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_XGrid), intent(in) :: xgrid <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside integer, intent(in), optional :: gridindex integer, intent(in), optional :: gridToFieldMap(:) integer, intent(out), optional :: rcDESCRIPTION:
This call completes an ESMF_Field allocated with the ESMF_FieldEmptyCreate() call.
The fortran data pointer inside ESMF_Field can be queried and deallocated when datacopyflag is ESMF_DATACOPY_REFERENCE. Note that the ESMF_FieldDestroy call does not deallocate the fortran data pointer in this case. This gives user more flexibility over memory management.
The arguments are:
INTERFACE:
function ESMF_FieldEmptyCreate(name, rc)RETURN VALUE:
type(ESMF_Field) :: ESMF_FieldEmptyCreateARGUMENTS:
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
character (len = *), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
This version of ESMF_FieldCreate builds an empty ESMF_Field and depends on later calls to add an ESMF_Grid and ESMF_Array to it. The empty ESMF_Field can be completed in one more step or two more steps by the ESMF_FieldEmptySet and ESMF_FieldEmptyComplete methods. Attributes can be added to an empty Field object. For an example and associated documentation using this method see section 26.3.8 and 26.3.7.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptySet() subroutine ESMF_FieldEmptySetGrid(field, grid, StaggerLoc, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_Grid), intent(in) :: grid -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_STAGGERLOC), intent(in), optional :: StaggerLoc integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Set a grid and an optional staggerloc (default to center stagger ESMF_STAGGERLOC_CENTER) in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_Grid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.
For an example and associated documentation using this method see section 26.3.7.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptySet() subroutine ESMF_FieldEmptySetMesh(field, mesh, meshloc, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_Mesh), intent(in) :: mesh -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_MeshLoc), intent(in), optional :: meshloc integer, intent(out), optional :: rcDESCRIPTION:
Set a mesh and an optional meshloc (default to center stagger ESMF_MESHLOC_NODE) in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_Mesh internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptySet() subroutine ESMF_FieldEmptySetLocStream(field, locstream, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_LocStream), intent(in) :: locstream -- The following arguments require argument keyword syntax (e.g. rc=rc). -- integer, intent(out), optional :: rcDESCRIPTION:
Set a ESMF_LocStream in a non-completed ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_LocStream internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldEmptySet() subroutine ESMF_FieldEmptySetXGrid(field, xgrid, xgridside, gridindex, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field type(ESMF_XGrid), intent(in) :: xgrid -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside integer, intent(in), optional :: gridindex integer, intent(out), optional :: rcDESCRIPTION:
Set a xgrid and optional xgridside (default to balanced side ESMF_XGRIDSIDE_Balanced) and gridindex (default to 1) in a non-complete ESMF_Field. The ESMF_Field must not be completed for this to succeed. After this operation, the ESMF_Field contains the ESMF_XGrid internally but holds no data. The status of the field changes from ESMF_FIELDSTATUS_EMPTY to ESMF_FIELDSTATUS_GRIDSET or stays ESMF_FIELDSTATUS_GRIDSET.
The arguments are:
INTERFACE:
subroutine ESMF_FieldFill(field, &
dataFillScheme, member, step, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
character(len=*), intent(in), optional :: dataFillScheme
integer, intent(in), optional :: member
integer, intent(in), optional :: step
integer, intent(out), optional :: rc
DESCRIPTION:
Fill field with data according to dataFillScheme. Depending on the chosen fill scheme, the member and step arguments are used to provide differing fill data patterns.
The arguments are:
INTERFACE:
subroutine ESMF_FieldGather<rank><type><kind>(field, farray, & rootPet, tile, vm, rc)ARGUMENTS:
type(ESMF_Field), intent(in) :: field <type>(ESMF_KIND_<kind>), intent(out), target :: farray(<rank>) integer, intent(in) :: rootPet -- The following arguments require argument keyword syntax (e.g. rc=rc). -- integer, intent(in), optional :: tile type(ESMF_VM), intent(in), optional :: vm integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Gather the data of an ESMF_Field object into the farray located on rootPET. A single DistGrid tile of array must be gathered into farray. The optional tile argument allows selection of the tile. For Fields defined on a single tile DistGrid the default selection (tile 1) will be correct. The shape of farray must match the shape of the tile in Field.
If the Field contains replicating DistGrid dimensions data will be gathered from the numerically higher DEs. Replicated data elements in numericaly lower DEs will be ignored.
The implementation of Scatter and Gather is not sequence index based. If the Field is built on arbitrarily distributed Grid, Mesh, LocStream or XGrid, Gather will not gather data to rootPet from source data points corresponding to the sequence index on rootPet. Instead Gather will gather a contiguous memory range from source PET to rootPet. The size of the memory range is equal to the number of data elements on the source PET. Vice versa for the Scatter operation. In this case, the user should use ESMF_FieldRedist to achieve the same data operation result. For examples how to use ESMF_FieldRedist to perform Gather and Scatter, please refer to 26.3.31 and 26.3.30.
This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.
For examples and associated documentations using this method see Section 26.3.27.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldGet()
subroutine ESMF_FieldGetDefault(field, arrayspec, &
status, geomtype, grid, mesh, locstream, xgrid, array, &
typekind, dimCount, rank, staggerloc, meshloc, xgridside, &
gridindex, gridToFieldMap, ungriddedLBound, ungriddedUBound, &
totalLWidth, totalUWidth, localDeCount, name, vm, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_ArraySpec), intent(out), optional :: arrayspec
type(ESMF_FieldStatus_Flag),intent(out), optional :: status
type(ESMF_GeomType_Flag), intent(out), optional :: geomtype
type(ESMF_Grid), intent(out), optional :: grid
type(ESMF_Mesh), intent(out), optional :: mesh
type(ESMF_LocStream), intent(out), optional :: locstream
type(ESMF_XGrid), intent(out), optional :: xgrid
type(ESMF_Array), intent(out), optional :: array
type(ESMF_TypeKind_Flag), intent(out), optional :: typekind
integer, intent(out), optional :: dimCount
integer, intent(out), optional :: rank
type(ESMF_StaggerLoc), intent(out), optional :: staggerloc
type(ESMF_MeshLoc), intent(out), optional :: meshloc
type(ESMF_XGridSide_Flag), intent(out), optional :: xgridside
integer, intent(out), optional :: gridindex
integer, intent(out), optional :: gridToFieldMap(:)
integer, intent(out), optional :: ungriddedLBound(:)
integer, intent(out), optional :: ungriddedUBound(:)
integer, intent(out), optional :: totalLWidth(:,:)
integer, intent(out), optional :: totalUWidth(:,:)
integer, intent(out), optional :: localDeCount
character(len=*), intent(out), optional :: name
type(ESMF_VM), intent(out), optional :: vm
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Query an ESMF_Field for various things. All arguments after the field are optional. To select individual items use the named_argument=value syntax. For an example and associated documentation using this method see section 26.3.3.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldGet() subroutine ESMF_FieldGetDataPtr<rank><type><kind>(field, localDe, & farrayPtr, exclusiveLBound, exclusiveUBound, exclusiveCount, & computationalLBound, computationalUBound, computationalCount, & totalLBound, totalUBound, totalCount, rc)ARGUMENTS:
type(ESMF_Field), intent(in) :: field -- The following arguments require argument keyword syntax (e.g. rc=rc). -- integer, intent(in), optional :: localDe <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) integer, intent(out), optional :: exclusiveLBound(:) integer, intent(out), optional :: exclusiveUBound(:) integer, intent(out), optional :: exclusiveCount(:) integer, intent(out), optional :: computationalLBound(:) integer, intent(out), optional :: computationalUBound(:) integer, intent(out), optional :: computationalCount(:) integer, intent(out), optional :: totalLBound(:) integer, intent(out), optional :: totalUBound(:) integer, intent(out), optional :: totalCount(:) integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Get a Fortran pointer to DE-local memory allocation within field. For convenience DE-local bounds can be queried at the same time. For an example and associated documentation using this method see section 26.3.2.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldGetBounds()
subroutine ESMF_FieldGetBounds(field, localDe, &
exclusiveLBound, exclusiveUBound, exclusiveCount, computationalLBound, &
computationalUBound, computationalCount, totalLBound, &
totalUBound, totalCount, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: localDe
integer, intent(out), optional :: exclusiveLBound(:)
integer, intent(out), optional :: exclusiveUBound(:)
integer, intent(out), optional :: exclusiveCount(:)
integer, intent(out), optional :: computationalLBound(:)
integer, intent(out), optional :: computationalUBound(:)
integer, intent(out), optional :: computationalCount(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
This method returns the bounds information of a field that consists of a internal grid and a internal array. The exclusive and computational bounds are shared between the grid and the array but the total bounds are the array bounds plus the halo width. The count is the number of elements between each bound pair.
The arguments are:
INTERFACE:
subroutine ESMF_FieldHalo(field, routehandle, &
routesyncflag, finishedflag, checkflag, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_RouteSync_Flag), intent(in), optional :: routesyncflag
logical, intent(out), optional :: finishedflag
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed Field halo operation for field. The field argument must be weakly congruent and typekind conform to the Field used during ESMF_FieldHaloStore(). Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions.
See ESMF_FieldHaloStore() on how to precompute routehandle.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_FieldHaloRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a Field halo operation. After this call routehandle becomes invalid.
INTERFACE:
subroutine ESMF_FieldHaloStore(field, routehandle, &
startregion, haloLDepth, haloUDepth, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_StartRegion_Flag), intent(in), optional :: startregion
integer, intent(in), optional :: haloLDepth(:)
integer, intent(in), optional :: haloUDepth(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a Field halo operation over the data in field. By default, i.e. without specifying startregion, haloLDepth and haloUDepth, all elements in the total Field region that lie outside the exclusive region will be considered potential destination elements for halo. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.
Specifying startregion allows to change the shape of the effective halo region from the inside. Setting this flag to ESMF_STARTREGION_COMPUTATIONAL means that only elements outside the computational region of the Field are considered for potential destination elements for halo. The default is ESMF_STARTREGION_EXCLUSIVE.
The haloLDepth and haloUDepth arguments allow to reduce the extent of the effective halo region. Starting at the region specified by startregion, the haloLDepth and haloUDepth define a halo depth in each direction. Note that the maximum halo region is limited by the total Field region, independent of the actual haloLDepth and haloUDepth setting. The total Field region is local DE specific. The haloLDepth and haloUDepth are interpreted as the maximum desired extent, reducing the potentially larger region available for halo.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldHalo() on any Field that is weakly congruent and typekind conform to field. Congruent Fields possess matching DistGrids and the shape of the local field tiles, i.e. the memory allocation, matches between the Fieldss for every DE. For weakly congruent Fieldss the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fieldss that differ in the number of elements in the left most undistributed dimensions.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_FieldPrint(field, rc)ARGUMENTS:
type(ESMF_Field), intent(in) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Prints information about the field to stdout.
This subroutine goes through the internal data members of a field
data type and prints information of each data member.
The arguments are:
INTERFACE:
subroutine ESMF_FieldRead(field, fileName, &
variableName, timeslice, iofmt, rc)
ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
character(*), intent(in) :: fileName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
character(*), intent(in), optional :: variableName
integer, intent(in), optional :: timeslice
type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
integer, intent(out), optional :: rc
DESCRIPTION:
Read Field data from a file and put it into an ESMF_Field object. For this API to be functional, the environment variable ESMF_PIO should be set to "internal" when the ESMF library is built. Please see the section on Data I/O, 37.3.
Limitations:
The arguments are:
INTERFACE:
subroutine ESMF_FieldRedist(srcField, dstField, routehandle, &
checkflag, rc)
ARGUMENTS:
type(ESMF_Field), intent(in),optional :: srcField
type(ESMF_Field), intent(inout),optional :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed Field redistribution from srcField to dstField. Both srcField and dstField must be congruent and typekind conform with the respective Fields used during ESMF_FieldRedistStore(). Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Because Grid dimensions are mapped to Field in a sequence order, it's necessary to map the ungridded dimensions to the first set of dimensions in order to use the weakly congruent Field redist feature. Not providing a non-default gridToFieldMap during Field creation and then using such Fields in a weakly congruent manner in Field communication methods leads to undefined behavior.
The srcField and dstField arguments are optional in support of the situation where srcField and/or dstField are not defined on all PETs. The srcField and dstField must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
See ESMF_FieldRedistStore() on how to precompute routehandle.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 26.3.29.
INTERFACE:
subroutine ESMF_FieldRedistRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a Field redistribution. After this call routehandle becomes invalid.
INTERFACE:
! Private name; call using ESMF_FieldRedistStore()
subroutine ESMF_FieldRedistStore<type><kind>(srcField, dstField, &
routehandle, factor, srcToDstTransposeMap, &
ignoreUnmatchedIndices, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: srcField
type(ESMF_Field), intent(inout) :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), intent(in) :: factor
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: srcToDstTransposeMap(:)
logical, intent(in), optional :: ignoreUnmatchedIndices
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
ESMF_FieldRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_FieldRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_FieldRedistStore() method, as provided through the separate entry points shown in 26.6.55 and 26.6.56, is described in the following paragraphs as a whole.
Store a Field redistribution operation from srcField to dstField. Interface 26.6.55 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 26.6.56. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.
Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices.
Source Field, destination Field, and the factor may be of different <type><kind>. Further, source and destination Fields may differ in shape, however, the number of elements must match.
If srcToDstTransposeMap is not specified the redistribution corresponds to an identity mapping of the sequentialized source Field to the sequentialized destination Field. If the srcToDstTransposeMap argument is provided it must be identical on all PETs. The srcToDstTransposeMap allows source and destination Field dimensions to be transposed during the redistribution. The number of source and destination Field dimensions must be equal under this condition and the size of mapped dimensions must match.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldRedist() on any pair of Fields that are weakly congruent and typekind conform with the srcField, dstField pair. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Because Grid dimensions are mapped to Field in a sequence order, it's necessary to map the ungridded dimensions to the first set of dimensions in order to use the weakly congruent Field redist feature. Not providing a non-default gridToFieldMap during Field creation and then using such Fields in a weakly congruent manner in Field communication methods leads to undefined behavior.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 26.3.29.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldRedistStore()
subroutine ESMF_FieldRedistStoreNF(srcField, dstField, &
routehandle, srcToDstTransposeMap, &
ignoreUnmatchedIndices, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: srcField
type(ESMF_Field), intent(inout) :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: srcToDstTransposeMap(:)
logical, intent(in), optional :: ignoreUnmatchedIndices
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
ESMF_FieldRedistStore() is a collective method across all PETs of the current Component. The interface of the method is overloaded, allowing - in principle - each PET to call into ESMF_FieldRedistStore() through a different entry point. Restrictions apply as to which combinations are sensible. All other combinations result in ESMF run time errors. The complete semantics of the ESMF_FieldRedistStore() method, as provided through the separate entry points shown in 26.6.55 and 26.6.56, is described in the following paragraphs as a whole.
Store a Field redistribution operation from srcField to dstField. Interface 26.6.55 allows PETs to specify a factor argument. PETs not specifying a factor argument call into interface 26.6.56. If multiple PETs specify the factor argument, its type and kind, as well as its value must match across all PETs. If none of the PETs specify a factor argument the default will be a factor of 1. The resulting factor is applied to all of the source data during redistribution, allowing scaling of the data, e.g. for unit transformation.
Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices.
Source Field, destination Field, and the factor may be of different <type><kind>. Further, source and destination Fields may differ in shape, however, the number of elements must match.
If srcToDstTransposeMap is not specified the redistribution corresponds to an identity mapping of the sequentialized source Field to the sequentialized destination Field. If the srcToDstTransposeMap argument is provided it must be identical on all PETs. The srcToDstTransposeMap allows source and destination Field dimensions to be transposed during the redistribution. The number of source and destination Field dimensions must be equal under this condition and the size of mapped dimensions must match.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldRedist() on any pair of Fields that are weakly congruent and typekind conform with the srcField, dstField pair. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Because Grid dimensions are mapped to Field in a sequence order, it's necessary to map the ungridded dimensions to the first set of dimensions in order to use the weakly congruent Field redist feature. Not providing a non-default gridToFieldMap during Field creation and then using such Fields in a weakly congruent manner in Field communication methods leads to undefined behavior.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 26.3.29.
The arguments are:
INTERFACE:
subroutine ESMF_FieldRegrid(srcField, dstField, &
routehandle, zeroregion, &
termorderflag, checkflag, rc)
ARGUMENTS:
type(ESMF_Field), intent(in), optional :: srcField
type(ESMF_Field), intent(inout), optional :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Region_Flag), intent(in), optional :: zeroregion
type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute the precomputed regrid operation stored in routehandle to interpolate from srcField to dstField. See ESMF_FieldRegridStore() on how to precompute the routehandle.
Both srcField and dstField must be weakly congruent with the respective Fields used during ESMF_FieldRegridStore(). Congruent Fields possess matching DistGrids and the shape of the local array tiles matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. You can apply the routehandle between any set of Fields weakly congruent to the original Fields used to create the routehandle without incurring an error. However, if you want the routehandle to be the same interpolation between the grid objects upon which the Fields are build as was calculated with the original ESMF_FieldRegridStore() call, then there are additional constraints on the grid objects. To be the same interpolation, the grid objects upon which the Fields are build must contain the same coordinates at the stagger locations involved in the regridding as the original source and destination Fields used in the ESMF_FieldRegridStore() call. The routehandle represents the interpolation between the grid objects as they were during the ESMF_FieldRegridStore() call. So if the coordinates at the stagger location in the grid objects change, a new call to ESMF_FieldRegridStore() is necessary to compute the interpolation between that new set of coordinates.
The srcField and dstField arguments are optional in support of the situation where srcField and/or dstField are not defined on all PETs. The srcField and dstField must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_FieldRegridRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Free resources used by regrid objec
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldRegridStore()
subroutine ESMF_FieldRegridStoreNX(srcField, dstField, &
srcMaskValues, dstMaskValues, &
regridmethod, &
polemethod, regridPoleNPnts, &
lineType, &
normType, &
unmappedaction, ignoreDegenerate, &
srcTermProcessing, &
pipeLineDepth, &
routehandle, &
factorList, factorIndexList, &
weights, indices, & ! DEPRECATED ARGUMENTS
srcFracField, dstFracField, unmappedDstList, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: srcField
type(ESMF_Field), intent(inout) :: dstField
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer(ESMF_KIND_I4), intent(in), optional :: srcMaskValues(:)
integer(ESMF_KIND_I4), intent(in), optional :: dstMaskValues(:)
type(ESMF_RegridMethod_Flag), intent(in), optional :: regridmethod
type(ESMF_PoleMethod_Flag), intent(in), optional :: polemethod
integer, intent(in), optional :: regridPoleNPnts
type(ESMF_LineType_Flag), intent(in), optional :: lineType
type(ESMF_NormType_Flag), intent(in), optional :: normType
type(ESMF_UnmappedAction_Flag), intent(in), optional :: unmappedaction
logical, intent(in), optional :: ignoreDegenerate
integer, intent(inout), optional :: srcTermProcessing
integer, intent(inout), optional :: pipeLineDepth
type(ESMF_RouteHandle), intent(inout), optional :: routehandle
real(ESMF_KIND_R8), pointer, optional :: factorList(:)
integer(ESMF_KIND_I4), pointer, optional :: factorIndexList(:,:)
real(ESMF_KIND_R8), pointer, optional :: weights(:) ! DEPRECATED ARG
integer(ESMF_KIND_I4), pointer, optional :: indices(:,:) ! DEPRECATED ARG
type(ESMF_Field), intent(inout), optional :: srcFracField
type(ESMF_Field), intent(inout), optional :: dstFracField
integer(ESMF_KIND_I4), pointer, optional :: unmappedDstList(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Creates a sparse matrix operation (stored in routehandle) that contains the calculations and communications necessary to interpolate from srcField to dstField. The routehandle can then be used in the call ESMF_FieldRegrid() to interpolate between the Fields. The user may also get the interpolation matrix in sparse matrix form via the optional arguments factorList and factorIndexList.
The routehandle generated by this call is based just on the coordinates in the Grids or Meshes contained in the Fields. If those coordinates don't change the routehandle can be used repeatedly to interpolate from the source Field to the destination Field. This is true even if the data in the Fields changes. The routehandle may also be used to interpolate between any source and destination Field which are created on the same location in the same Grid, LocStream, or Mesh as the original Fields.
When it's no longer needed the routehandle should be destroyed by using ESMF_FieldRegridRelease() to free the memory it's using.
Note, as a side effect, that this call may change the data in dstField. If this is undesirable, then an easy work around is to create a second temporary field with the same structure as dstField and pass that in instead.
The arguments are:
Note that partial sums may lead to bit-for-bit differences in the results. See section 36.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.
The ESMF_FieldRegridStore() method implements an auto-tuning scheme
for the srcTermProcessing parameter. The intent on the
srcTermProcessing argument is "inout" in order to
support both overriding and accessing the auto-tuning parameter.
If an argument 
is specified, it is used for the
srcTermProcessing parameter, and the auto-tuning phase is skipped.
In this case the srcTermProcessing argument is not modified on
return. If the provided argument is 
, the srcTermProcessing
parameter is determined internally using the auto-tuning scheme. In this
case the srcTermProcessing argument is re-set to the internally
determined value on return. Auto-tuning is also used if the optional
srcTermProcessing argument is omitted.
Note that the pipeline depth has no affect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.
The ESMF_FieldRegridStore() method implements an auto-tuning scheme
for the pipelineDepth parameter. The intent on the
pipelineDepth argument is "inout" in order to
support both overriding and accessing the auto-tuning parameter.
If an argument is specified, it is used for the
pipelineDepth parameter, and the auto-tuning phase is skipped.
In this case the pipelineDepth argument is not modified on
return. If the provided argument is , the pipelineDepth
parameter is determined internally using the auto-tuning scheme. In this
case the pipelineDepth argument is re-set to the internally
determined value on return. Auto-tuning is also used if the optional
pipelineDepth argument is omitted.
INTERFACE:
! Private name; call using ESMF_FieldRegridStore()
subroutine ESMF_FieldRegridStoreX(xgrid, srcField, dstField, &
routehandle, srcFracField, dstFracField, &
srcMergeFracField, dstMergeFracField, rc)
ARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid
type(ESMF_Field), intent(in) :: srcField
type(ESMF_Field), intent(inout) :: dstField
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_RouteHandle), intent(inout), optional :: routehandle
type(ESMF_Field), intent(inout), optional :: srcFracField
type(ESMF_Field), intent(inout), optional :: dstFracField
type(ESMF_Field), intent(inout), optional :: srcMergeFracField
type(ESMF_Field), intent(inout), optional :: dstMergeFracField
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Creates a sparse matrix operation (stored in routehandle) that contains the calculations and communications necessary to interpolate from srcField to dstField. The routehandle can then be used in the call ESMF_FieldRegrid() to interpolate between the ESMF_Fields. Information such as index mapping and weights are obtained from the XGrid by matching the Field Grids or Meshes in the XGrid. It's erroneous to have matching Grid or Mesh objects in the srcField and dstField. They must be different in either topological or geometric characteristics. For ESMF_Fields built on identical ESMF_Grid or ESMF_Mesh on different VM, user can use ESMF_FieldRedistStore() and ESMF_FieldRedist() methods to communicate data directly without interpolation.
The routehandle generated by this call is subsequently computed based on these information. If those information don't change the routehandle can be used repeatedly to interpolate from the source Field to the destination Field. This is true even if the data in the Fields changes. The routehandle may also be used to interpolate between any source and destination Field which are created on the same stagger location and Grid or on the same mesh location and Mesh as the original Fields.
When it's no longer needed the routehandle should be destroyed by using ESMF_FieldRegridRelease() to free the memory it's using. Note ESMF_FieldRegridStore() assumes the coordinates used in the Grids upon which the Fields are built are in degrees.
The arguments are:
INTERFACE:
subroutine ESMF_FieldRegridGetArea(areaField, rc)RETURN VALUE:
ARGUMENTS:
type(ESMF_Field), intent(inout) :: areaField
integer, intent(out), optional :: rc
DESCRIPTION:
This subroutine gets the area of the cells used for conservative interpolation for the grid object associated with areaField and puts them into areaField. If created on a 2D Grid, it must be built on the ESMF_STAGGERLOC_CENTER stagger location. If created on a 3D Grid, it must be built on the ESMF_STAGGERLOC_CENTER_VCENTER stagger location. If created on a Mesh, it must be built on the ESMF_MESHLOC_ELEMENT mesh location.
The arguments are:
INTERFACE:
subroutine ESMF_FieldScatter<rank><type><kind>(field, farray, & rootPet, tile, vm, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field mtype (ESMF_KIND_mtypekind),intent(in), target :: farray(mdim) integer, intent(in) :: rootPet -- The following arguments require argument keyword syntax (e.g. rc=rc). -- integer, intent(in), optional :: tile type(ESMF_VM), intent(in), optional :: vm integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Scatter the data of farray located on rootPET across an ESMF_Field object. A single farray must be scattered across a single DistGrid tile in Field. The optional tile argument allows selection of the tile. For Fields defined on a single tile DistGrid the default selection (tile 1) will be correct. The shape of farray must match the shape of the tile in Field.
If the Field contains replicating DistGrid dimensions data will be scattered across all of the replicated pieces.
The implementation of Scatter and Gather is not sequence index based. If the Field is built on arbitrarily distributed Grid, Mesh, LocStream or XGrid, Scatter will not scatter data from rootPet to the destination data points corresponding to the sequence index on the rootPet. Instead Scatter will scatter a contiguous memory range from rootPet to destination PET. The size of the memory range is equal to the number of data elements on the destination PET. Vice versa for the Gather operation. In this case, the user should use ESMF_FieldRedist to achieve the same data operation result. For examples how to use ESMF_FieldRedist to perform Gather and Scatter, please refer to 26.3.31 and 26.3.30.
This version of the interface implements the PET-based blocking paradigm: Each PET of the VM must issue this call exactly once for all of its DEs. The call will block until all PET-local data objects are accessible.
For examples and associated documentations using this method see Section 26.3.28.
The arguments are:
INTERFACE:
subroutine ESMF_FieldSMM(srcField, dstField, routehandle, &
zeroregion, termorderflag, checkflag, rc)
ARGUMENTS:
type(ESMF_Field), intent(in), optional :: srcField
type(ESMF_Field), intent(inout), optional :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Region_Flag), intent(in), optional :: zeroregion
type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed Field sparse matrix multiplication from srcField to dstField. Both srcField and dstField must be congruent and typekind conform with the respective Fields used during ESMF_FieldSMMStore(). Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Because Grid dimensions are mapped to Field in a sequence order, it's necessary to map the ungridded dimensions to the first set of dimensions in order to use the weakly congruent Field SMM feature. Not providing a non-default gridToFieldMap during Field creation and then using such Fields in a weakly congruent manner in Field communication methods leads to undefined behavior.
The srcField and dstField arguments are optional in support of the situation where srcField and/or dstField are not defined on all PETs. The srcField and dstField must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
See ESMF_FieldSMMStore() on how to precompute routehandle.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 26.3.32.
sparse matrix multiplication
INTERFACE:
subroutine ESMF_FieldSMMRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with a Field sparse matrix multiplication. After this call routehandle becomes invalid.
INTERFACE:
! Private name; call using ESMF_FieldSMMStore()
subroutine ESMF_FieldSMMStore<type><kind>(srcField, dstField, &
routehandle, factorList, factorIndexList, &
ignoreUnmatchedIndices, srcTermProcessing, pipelineDepth, &
transposeRoutehandle, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: srcField
type(ESMF_Field), intent(inout) :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), intent(in) :: factorList(:)
integer, intent(in), :: factorIndexList(:,:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: ignoreUnmatchedIndices
integer, intent(inout), optional :: srcTermProcessing
integer, intent(inout), optional :: pipeLineDepth
type(ESMF_RouteHandle), intent(inout), optional :: transposeRoutehandle
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a Field sparse matrix multiplication operation from srcField to dstField. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source Field vector to the destination Field vector.
Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldSMM() on any pair of Fields that are weakly congruent and typekind conform with the srcField, dstField pair. Congruent Fields possess matching DistGrids, and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Because Grid dimensions are mapped to Field in a sequence order, it's necessary to map the ungridded dimensions to the first set of dimensions in order to use the weakly congruent Field SMM feature. Not providing a non-default gridToFieldMap during Field creation and then using such Fields in a weakly congruent manner in Field communication methods leads to undefined behavior.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 26.3.32.
The arguments are:
The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4. The second dimension of factorIndexList steps through the list of
In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the srcField while factorIndexList(2,:) specifies the sequence index of the destination element in dstField. For this format to be a valid option source and destination Fields must have matching number of tensor elements (the product of the sizes of all Field tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.
The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the srcField. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the dstField.
See section 28.2.17 for details on the definition of Field sequence indices and tensor sequence indices.
Note that partial sums may lead to bit-for-bit differences in the results. See section 36.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.
The ESMF_FieldSMMStore() method implements an auto-tuning scheme
for the srcTermProcessing parameter. The intent on the
srcTermProcessing argument is "inout" in order to
support both overriding and accessing the auto-tuning parameter.
If an argument is specified, it is used for the
srcTermProcessing parameter, and the auto-tuning phase is skipped.
In this case the srcTermProcessing argument is not modified on
return. If the provided argument is , the srcTermProcessing
parameter is determined internally using the auto-tuning scheme. In this
case the srcTermProcessing argument is re-set to the internally
determined value on return. Auto-tuning is also used if the optional
srcTermProcessing argument is omitted.
Note that the pipeline depth has no affect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.
The ESMF_FieldSMMStore() method implements an auto-tuning scheme
for the pipelineDepth parameter. The intent on the
pipelineDepth argument is "inout" in order to
support both overriding and accessing the auto-tuning parameter.
If an argument is specified, it is used for the
pipelineDepth parameter, and the auto-tuning phase is skipped.
In this case the pipelineDepth argument is not modified on
return. If the provided argument is , the pipelineDepth
parameter is determined internally using the auto-tuning scheme. In this
case the pipelineDepth argument is re-set to the internally
determined value on return. Auto-tuning is also used if the optional
pipelineDepth argument is omitted.
INTERFACE:
! Private name; call using ESMF_FieldSMMStore()
subroutine ESMF_FieldSMMStoreNF(srcField, dstField, &
routehandle, ignoreUnmatchedIndices, &
srcTermProcessing, pipelineDepth, transposeRoutehandle, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: srcField
type(ESMF_Field), intent(inout) :: dstField
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: ignoreUnmatchedIndices
integer, intent(inout), optional :: srcTermProcessing
integer, intent(inout), optional :: pipeLineDepth
type(ESMF_RouteHandle), intent(inout), optional :: transposeRoutehandle
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store a Field sparse matrix multiplication operation from srcField to dstField. PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
Both srcField and dstField are interpreted as sequentialized vectors. The sequence is defined by the order of DistGrid dimensions and the order of tiles within the DistGrid or by user-supplied arbitrary sequence indices. See section 28.2.17 for details on the definition of sequence indices. SMM corresponds to an identity mapping of the source Field vector to the destination Field vector.
Source and destination Fields may be of different <type><kind>. Further source and destination Fields may differ in shape, however, the number of elements must match.
It is erroneous to specify the identical Field object for srcField and dstField arguments.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_FieldSMM() on any pair of Fields that are weakly congruent and typekind conform with the srcField, dstField pair. Congruent Fields possess matching DistGrids and the shape of the local array tiles, i.e. the memory allocation, matches between the Fields for every DE. For weakly congruent Fields the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Fields that differ in the number of elements in the left most undistributed dimensions. Because Grid dimensions are mapped to Field in a sequence order, it's necessary to map the ungridded dimensions to the first set of dimensions in order to use the weakly congruent Field SMM feature. Not providing a non-default gridToFieldMap during Field creation and then using such Fields in a weakly congruent manner in Field communication methods leads to undefined behavior.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
For examples and associated documentations using this method see Section 26.3.32.
The arguments are:
Note that partial sums may lead to bit-for-bit differences in the results. See section 36.2.1 for an in-depth discussion of all bit-for-bit reproducibility aspects related to route-based communication methods.
The ESMF_FieldSMMStore() method implements an auto-tuning scheme
for the srcTermProcessing parameter. The intent on the
srcTermProcessing argument is "inout" in order to
support both overriding and accessing the auto-tuning parameter.
If an argument is specified, it is used for the
srcTermProcessing parameter, and the auto-tuning phase is skipped.
In this case the srcTermProcessing argument is not modified on
return. If the provided argument is , the srcTermProcessing
parameter is determined internally using the auto-tuning scheme. In this
case the srcTermProcessing argument is re-set to the internally
determined value on return. Auto-tuning is also used if the optional
srcTermProcessing argument is omitted.
Note that the pipeline depth has no affect on the bit-for-bit reproducibility of the results. However, it may affect the performance reproducibility of the exchange.
The ESMF_FieldSMMStore() method implements an auto-tuning scheme
for the pipelineDepth parameter. The intent on the
pipelineDepth argument is "inout" in order to
support both overriding and accessing the auto-tuning parameter.
If an argument is specified, it is used for the
pipelineDepth parameter, and the auto-tuning phase is skipped.
In this case the pipelineDepth argument is not modified on
return. If the provided argument is , the pipelineDepth
parameter is determined internally using the auto-tuning scheme. In this
case the pipelineDepth argument is re-set to the internally
determined value on return. Auto-tuning is also used if the optional
pipelineDepth argument is omitted.
INTERFACE:
function ESMF_FieldIsCreated(field, rc)RETURN VALUE:
logical :: ESMF_FieldIsCreatedARGUMENTS:
type(ESMF_Field), intent(in) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Return .true. if the field has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..
The arguments are:
INTERFACE:
subroutine ESMF_FieldValidate(field, rc)ARGUMENTS:
type(ESMF_Field), intent(in) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Validates that the field is internally consistent. Currently this method determines if the field is uninitialized or already destroyed. It validates the contained array and grid objects. The code also checks if the array and grid sizes agree. This check compares the distgrid contained in array and grid; then it proceeds to compare the computational bounds contained in array and grid.
The method returns an error code if problems are found.
The arguments are:
INTERFACE:
! Private name; call using ESMF_FieldSet() subroutine ESMF_FieldSet(field, name, rc)ARGUMENTS:
type(ESMF_Field), intent(inout) :: field
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
character(len = *), intent(in), optional :: name
integer, intent(out), optional :: rc
DESCRIPTION:
Sets adjustable settings in an ESMF_Field object.
The arguments are:
INTERFACE:
subroutine ESMF_FieldWrite(field, fileName, &
variableName, overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
type(ESMF_Field), intent(in) :: field
character(*), intent(in) :: fileName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
character(*), intent(in), optional :: variableName
logical, intent(in), optional :: overwrite
type(ESMF_FileStatus_Flag), intent(in), optional :: status
integer, intent(in), optional :: timeslice
type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
integer, intent(out), optional :: rc
DESCRIPTION:
Write Field data into a file. For this API to be functional, the environment variable ESMF_PIO should be set to "internal" when the ESMF library is built. Please see the section on Data I/O, 37.3.
Limitations:
The arguments are:
INTERFACE:
subroutine ESMF_GridGetFieldBounds(grid, &
localDe, staggerloc, gridToFieldMap, &
ungriddedLBound, ungriddedUBound, &
totalLWidth, totalUWidth, &
totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
type(ESMF_Grid), intent(in) :: grid
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: localDe
type(ESMF_StaggerLoc), intent(in), optional :: staggerloc
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_Grid and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.
The arguments are:
INTERFACE:
subroutine ESMF_LocStreamGetFieldBounds(locstream, &
localDe, gridToFieldMap, &
ungriddedLBound, ungriddedUBound, &
totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
type(ESMF_LocStream), intent(in) :: locstream
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: localDe
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_LocStream and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.
The arguments are:
INTERFACE:
subroutine ESMF_MeshGetFieldBounds(mesh, &
localDe, gridToFieldMap, &
ungriddedLBound, ungriddedUBound, &
totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
type(ESMF_Mesh), intent(in) :: mesh
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: localDe
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_Mesh and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.
The arguments are:
INTERFACE:
subroutine ESMF_XGridGetFieldBounds(xgrid, &
xgridside, gridindex, localDe, gridToFieldMap, &
ungriddedLBound, ungriddedUBound, &
totalLBound, totalUBound, totalCount, rc)
ARGUMENTS:
type(ESMF_XGrid), intent(in) :: xgrid
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_XGridSide_Flag), intent(in), optional :: xgridside
integer, intent(in), optional :: gridindex
integer, intent(in), optional :: localDe
integer, intent(in), optional :: gridToFieldMap(:)
integer, intent(in), optional :: ungriddedLBound(:)
integer, intent(in), optional :: ungriddedUBound(:)
integer, intent(out), optional :: totalLBound(:)
integer, intent(out), optional :: totalUBound(:)
integer, intent(out), optional :: totalCount(:)
integer, intent(out), optional :: rc
DESCRIPTION:
Compute the lower and upper bounds of Fortran data array that can later be used in FieldCreate interface to create a ESMF_Field from a ESMF_XGrid and the Fortran data array. For an example and associated documentation using this method see section 26.3.9.
The arguments are:
The ESMF_ArrayBundle class allows a set of Arrays to be bundled into a single object. The Arrays in an ArrayBundle may be of different type, kind, rank and distribution. Besides ease of use resulting from bundling, the ArrayBundle class offers the opportunity for performance optimization when operating on a bundle of Arrays as a single entity. Communication methods are especially good candidates for performance optimization. Best optimization results are expected for ArrayBundles that contain Arrays that share a common distribution, i.e. DistGrid, and are of same type, kind and rank.
ArrayBundles are one of the data objects that can be added to States, which are used for providing to or receiving data from other Components.
Examples of creating, destroying and accessing ArrayBundles and their constituent Arrays are provided in this section, along with some notes on ArrayBundle methods.
An ArrayBundle is created from a list of ESMF_Array objects.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
allocate(arrayList(2))
arrayList(1) = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
rc=rc)
arrayList(2) = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
rc=rc)
Now arrayList is used to create an ArrayBundle object.
arraybundle = ESMF_ArrayBundleCreate(arrayList=arrayList, &
name="MyArrayBundle", rc=rc)
Here the temporary arrayList can be deallocated. This will not affect the ESMF Array objects inside the ArrayBundle. However, the Array objects must not be deallocated while the ArrayBundle references them.
deallocate(arrayList)
Individual Arrays can be added using the Fortran array constructor syntax (/ ... /). Here an ESMF_Array is created on the fly and immediately added to the ArrayBundle.
call ESMF_ArrayBundleAdd(arraybundle, arrayList=(/ &
ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, name="AonFly")/), &
rc=rc)
Items in the ArrayBundle can be replaced by items with the same name.
call ESMF_ArraySpecSet(arrayspec2, typekind=ESMF_TYPEKIND_R4, rank=2, rc=rc)
call ESMF_ArrayBundleReplace(arraybundle, arrayList=(/ &
ESMF_ArrayCreate(arrayspec=arrayspec2, distgrid=distgrid, name="AonFly")/), &
rc=rc)
Items can be removed from the ArrayBundle by providing their name.
call ESMF_ArrayBundleRemove(arraybundle, arrayNameList=(/"AonFly"/), rc=rc)
The ArrayBundle AddReplace() method can be used to conveniently add an item to the ArrayBundle, or replacing an existing item of the same name.
call ESMF_ArrayBundleAddReplace(arraybundle, arrayList=(/ &
ESMF_ArrayCreate(arrayspec=arrayspec2, distgrid=distgrid, name="AonFly")/), &
rc=rc)
The ArrayBundle object can be printed at any time to list its contents by name.
call ESMF_ArrayBundlePrint(arraybundle, rc=rc)
Individual items in the ArrayBundle can be accessed directly by their name.
call ESMF_ArrayBundleGet(arraybundle, arrayName="AonFly", array=arrayOut, &
rc=rc)
A list containing all of the Arrays in the ArrayBundle can also be requested in a single call. This requires that a large enough list argument is passed into the ESMF_ArrayBundleGet() method. The exact number of items in the ArrayBundle can be queried using the arrayCount argument first.
call ESMF_ArrayBundleGet(arraybundle, arrayCount=arrayCount, rc=rc)
Then use arrayCount to correctly allocate the arrayList variable for a second call to ESMF_ArrayBundleGet().
allocate(arrayList(arrayCount)) call ESMF_ArrayBundleGet(arraybundle, arrayList=arrayList, rc=rc)
Now the arrayList variable can be used to access the individual Arrays, e.g. to print them.
do i=1, arrayCount
call ESMF_ArrayPrint(arrayList(i), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
enddo
By default the arrayList returned by ESMF_ArrayBundleGet() contains the items in alphabetical order. To instead return the items in the same order in which they were added to the ArrayBundle, the itemorderflag argument is passed with a value of ESMF_ITEMORDER_ADDORDER.
call ESMF_ArrayBundleGet(arraybundle, arrayList=arrayList, &
itemorderflag=ESMF_ITEMORDER_ADDORDER, rc=rc)
Destroying an ArrayBundle does not destroy the Arrays. In fact, it leaves the Arrays totally unchanged.
call ESMF_ArrayBundleDestroy(arraybundle, rc=rc)
The Arrays must be destroyed separately.
call ESMF_ArrayDestroy(arrayList(1), rc=rc)
call ESMF_ArrayDestroy(arrayList(2), rc=rc)
deallocate(arrayList) call ESMF_DistGridDestroy(distgrid, rc=rc)
One of the most fundamental communication pattern in domain decomposition codes is the halo operation. The ESMF Array class supports halos by allowing memory for extra elements to be allocated on each DE. See section 28.2.14 for a discussion of the Array level halo operation. The ArrayBundle level extents the Array halo operation to bundles of Arrays.
First create an ESMF_ArrayBundle object containing a set of ESMF Arrays.
arraybundle = ESMF_ArrayBundleCreate(arrayList=arrayList, &
name="MyArrayBundle", rc=rc)
The ArrayBundle object can be treated as a single entity. The ESMF_ArrayBundleHaloStore() call determines the most efficient halo exchange pattern for all Arrays that are part of arraybundle.
call ESMF_ArrayBundleHaloStore(arraybundle=arraybundle, &
routehandle=haloHandle, rc=rc)
The halo exchange pattern stored in haloHandle can now be applied to the arraybundle object, or any other ArrayBundle that is weakly congruent to the one used during the ESMF_ArrayBundleHaloStore() call.
call ESMF_ArrayBundleHalo(arraybundle=arraybundle, routehandle=haloHandle, &
rc=rc)
Finally, when no longer needed, the resources held by haloHandle need to be returned to the system by calling ESMF_ArrayBundleHaloRelease().
call ESMF_ArrayBundleHaloRelease(routehandle=haloHandle, rc=rc)
Finally the ArrayBundle object can be destroyed.
call ESMF_ArrayBundleDestroy(arraybundle, rc=rc)
The following is a list of implementation specific details about the current ESMF ArrayBundle.
INTERFACE:
interface assignment(=)
arraybundle1 = arraybundle2
ARGUMENTS:
type(ESMF_ArrayBundle) :: arraybundle1
type(ESMF_ArrayBundle) :: arraybundle2
STATUS:
DESCRIPTION:
Assign arraybundle1 as an alias to the same ESMF ArrayBundle object in memory as arraybundle2. If arraybundle2 is invalid, then arraybundle1 will be equally invalid after the assignment.
The arguments are:
INTERFACE:
interface operator(==)
if (arraybundle1 == arraybundle2) then ... endif
OR
result = (arraybundle1 == arraybundle2)
RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle1
type(ESMF_ArrayBundle), intent(in) :: arraybundle2
STATUS:
DESCRIPTION:
Test whether arraybundle1 and arraybundle2 are valid aliases to the same ESMF ArrayBundle object in memory. For a more general comparison of two ESMF ArrayBundles, going beyond the simple alias test, the ESMF_ArrayBundleMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
interface operator(/=)
if (arraybundle1 /= arraybundle2) then ... endif
OR
result = (arraybundle1 /= arraybundle2)
RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle1
type(ESMF_ArrayBundle), intent(in) :: arraybundle2
STATUS:
DESCRIPTION:
Test whether arraybundle1 and arraybundle2 are not valid aliases to the same ESMF ArrayBundle object in memory. For a more general comparison of two ESMF ArrayBundles, going beyond the simple alias test, the ESMF_ArrayBundleMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayBundleAdd(arraybundle, arrayList, &
multiflag, relaxedflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
type(ESMF_Array), intent(in) :: arrayList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Add Array(s) to an ArrayBundle. It is an error if arrayList contains Arrays that match by name Arrays already contained in arraybundle.
INTERFACE:
subroutine ESMF_ArrayBundleAddReplace(arraybundle, arrayList, rc)ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
type(ESMF_Array), intent(in) :: arrayList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Arrays in arrayList that do not match any Arrays by name in arraybundle are added to the ArrayBundle. Arrays in arraybundle that match by name Arrays in arrayList are replaced by those Arrays.
INTERFACE:
function ESMF_ArrayBundleCreate(arrayList, multiflag, &
relaxedflag, name, rc)
RETURN VALUE:
type(ESMF_ArrayBundle) :: ESMF_ArrayBundleCreateARGUMENTS:
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Array), intent(in), optional :: arrayList(:)
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
character(len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_ArrayBundle object from a list of existing Arrays.
The creation of an ArrayBundle leaves the bundled Arrays unchanged, they remain valid individual objects. An ArrayBundle is a light weight container of Array references. The actual data remains in place, there are no data movements or duplications associated with the creation of an ArrayBundle.
INTERFACE:
subroutine ESMF_ArrayBundleDestroy(arraybundle, noGarbage, rc)ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: noGarbage
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Destroys an ESMF_ArrayBundle object. The member Arrays are not touched by this operation and remain valid objects that need to be destroyed individually if necessary.
By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argument.
The arguments are:
It is generally recommended to leave the noGarbage argument set to .FALSE. (the default), and to take advantage of the ESMF garbage collection system which will prevent problems with dangling aliases or incorrect sequences of destroy calls. However this level of support requires that a small remnant of the object is kept in memory past the destroy call. This can lead to an unexpected increase in memory consumption over the course of execution in applications that use temporary ESMF objects. For situations where the repeated creation and destruction of temporary objects leads to memory issues, it is recommended to call with noGarbage set to .TRUE., fully removing the entire temporary object from memory.
INTERFACE:
! Private name; call using ESMF_ArrayBundleGet()
subroutine ESMF_ArrayBundleGetListAll(arraybundle, &
itemorderflag, arrayCount, arrayList, arrayNameList, name, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_ItemOrder_Flag), intent(in), optional :: itemorderflag
integer, intent(out), optional :: arrayCount
type(ESMF_Array), intent(out), optional :: arrayList(:)
character(len=*), intent(out), optional :: arrayNameList(:)
character(len=*), intent(out), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Get general, i.e. not Array name specific information from the ArrayBundle.
INTERFACE:
! Private name; call using ESMF_ArrayBundleGet()
subroutine ESMF_ArrayBundleGetItem(arraybundle, arrayName, &
array, arrayCount, isPresent, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle
character(len=*), intent(in) :: arrayName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Array), intent(out), optional :: array
integer, intent(out), optional :: arrayCount
logical, intent(out), optional :: isPresent
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Get information about items that match arrayName in ArrayBundle.
INTERFACE:
! Private name; call using ESMF_ArrayBundleGet()
subroutine ESMF_ArrayBundleGetList(arraybundle, arrayName, arrayList, &
itemorderflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle
character(len=*), intent(in) :: arrayName
type(ESMF_Array), intent(out) :: arrayList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_ItemOrder_Flag), intent(in), optional :: itemorderflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Get the list of Arrays from ArrayBundle that match arrayName.
INTERFACE:
subroutine ESMF_ArrayBundleHalo(arraybundle, routehandle, &
checkflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed ArrayBundle halo operation for the Arrays in arrayBundle.
See ESMF_ArrayBundleHaloStore() on how to precompute routehandle.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayBundleHaloRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with an ArrayBundle halo operation. After this call routehandle becomes invalid.
INTERFACE:
subroutine ESMF_ArrayBundleHaloStore(arraybundle, routehandle, &
startregion, haloLDepth, haloUDepth, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_StartRegion_Flag),intent(in), optional :: startregion
integer, intent(in), optional :: haloLDepth(:)
integer, intent(in), optional :: haloUDepth(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store an ArrayBundle halo operation over the data in arraybundle. By default, i.e. without specifying startregion, haloLDepth and haloUDepth, all elements in the total Array regions that lie outside the exclusive regions will be considered potential destination elements for halo. However, only those elements that have a corresponding halo source element, i.e. an exclusive element on one of the DEs, will be updated under the halo operation. Elements that have no associated source remain unchanged under halo.
Specifying startregion allows to change the shape of the effective halo region from the inside. Setting this flag to ESMF_STARTREGION_COMPUTATIONAL means that only elements outside the computational region for each Array are considered for potential destination elements for halo. The default is ESMF_STARTREGION_EXCLUSIVE.
The haloLDepth and haloUDepth arguments allow to reduce the extent of the effective halo region. Starting at the region specified by startregion, the haloLDepth and haloUDepth define a halo depth in each direction. Note that the maximum halo region is limited by the total region for each Array, independent of the actual haloLDepth and haloUDepth setting. The total Array regions are local DE specific. The haloLDepth and haloUDepth are interpreted as the maximum desired extent, reducing the potentially larger region available for halo.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleHalo() on any ArrayBundle that is weakly congruent and typekind conform to arraybundle. Congruency for ArrayBundles is given by the congruency of its constituents. Congruent Arrays possess matching DistGrids, and the shape of the local array tiles, i.e. the memory allocation, matches between the Arrays for every DE. For weakly congruent Arrays the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Arrays that differ in the number of elements in the left most undistributed dimensions.
This call is collective across the current VM.
INTERFACE:
function ESMF_ArrayBundleIsCreated(arraybundle, rc)RETURN VALUE:
logical :: ESMF_ArrayBundleIsCreatedARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Return .true. if the arraybundle has been created. Otherwise return .false.. If an error occurs, i.e. rc /= ESMF_SUCCESS is returned, the return value of the function will also be .false..
The arguments are:
INTERFACE:
subroutine ESMF_ArrayBundlePrint(arraybundle, rc)ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Print internal information of the specified ESMF_ArrayBundle
object to stdout.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayBundleRead(arraybundle, fileName, &
singleFile, timeslice, iofmt, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
character(*), intent(in) :: fileName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: singleFile
integer, intent(in), optional :: timeslice
type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
integer, intent(out), optional :: rc
DESCRIPTION:
Read Array data to an ArrayBundle object from file(s). For this API to be functional, the environment variable ESMF_PIO should be set to "internal" when the ESMF library is built. Please see the section on Data I/O, 37.3.
Limitations:
The arguments are:
INTERFACE:
subroutine ESMF_ArrayBundleRedist(srcArrayBundle, dstArrayBundle, &
routehandle, checkflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in), optional :: srcArrayBundle
type(ESMF_ArrayBundle), intent(inout), optional :: dstArrayBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed ArrayBundle redistribution from the Arrays in srcArrayBundle to the Arrays in dstArrayBundle.
The srcArrayBundle and dstArrayBundle arguments are optional in support of the situation where srcArrayBundle and/or dstArrayBundle are not defined on all PETs. The srcArrayBundle and dstArrayBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayBundleRedistRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with an ArrayBundle redistribution. After this call routehandle becomes invalid.
INTERFACE:
! Private name; call using ESMF_ArrayBundleRedistStore()
subroutine ESMF_ArrayBundleRedistStore<type><kind>(srcArrayBundle, &
dstArrayBundle, routehandle, factor, srcToDstTransposeMap, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: srcArrayBundle
type(ESMF_ArrayBundle), intent(inout) :: dstArrayBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), intent(in) :: factor
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: srcToDstTransposeMap(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store an ArrayBundle redistribution operation from srcArrayBundle to dstArrayBundle. The redistribution between ArrayBundles is defined as the sequence of individual Array redistributions over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.
The effect of this method on ArrayBundles that contain aliased members is undefined.
PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.
See the description of method ESMF_ArrayRedistStore() for the definition of the Array based operation.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleRedist() on any pair of ArrayBundles that are weakly congruent and typekind conform with the Arrays contained in srcArrayBundle and dstArrayBundle. Congruent Arrays possess matching DistGrids, and the shape of the local array tiles, i.e. the memory allocation, matches between the Arrays for every DE. For weakly congruent Arrays the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Arrays that differ in the number of elements in the left most undistributed dimensions.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
INTERFACE:
! Private name; call using ESMF_ArrayBundleRedistStore()
subroutine ESMF_ArrayBundleRedistStoreNF(srcArrayBundle, dstArrayBundle, &
routehandle, srcToDstTransposeMap, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: srcArrayBundle
type(ESMF_ArrayBundle), intent(inout) :: dstArrayBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: srcToDstTransposeMap(:)
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store an ArrayBundle redistribution operation from srcArrayBundle to dstArrayBundle. The redistribution between ArrayBundles is defined as the sequence of individual Array redistributions over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.
The effect of this method on ArrayBundles that contain aliased members is undefined.
PETs that specify a factor argument must use the <type><kind> overloaded interface. Other PETs call into the interface without factor argument. If multiple PETs specify the factor argument its type and kind as well as its value must match across all PETs. If none of the PETs specifies a factor argument the default will be a factor of 1.
See the description of method ESMF_ArrayRedistStore() for the definition of the Array based operation.
The routine returns an ESMF_RouteHandle that can be used to call
ESMF_ArrayBundleRedist() on any pair of ArrayBundles that
are weakly congruent and typekind conform with the Arrays contained in
srcArrayBundle and dstArrayBundle.
Congruent Arrays possess matching DistGrids, and the shape of the local
array tiles, i.e. the memory allocation, matches between the Arrays for
every DE. For weakly congruent
Arrays the sizes of the undistributed dimensions, that vary faster with
memory than the first distributed dimension, are permitted to be different.
This means that the same routehandle can be applied to a large class
of similar Arrays that differ in the number of elements in the left most
undistributed dimensions.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayBundleRemove(arraybundle, arrayNameList, &
multiflag, relaxedflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
character(len=*), intent(in) :: arrayNameList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Remove Array(s) by name from ArrayBundle. In the relaxed setting it is not an error if arrayNameList contains names that are not found in arraybundle.
INTERFACE:
subroutine ESMF_ArrayBundleReplace(arraybundle, arrayList, &
multiflag, relaxedflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(inout) :: arraybundle
type(ESMF_Array), intent(in) :: arrayList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: multiflag
logical, intent(in), optional :: relaxedflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Replace Array(s) by name in ArrayBundle. In the relaxed setting it is not an error if arrayList contains Arrays that do not match by name any item in arraybundle. These Arrays are simply ignored in this case.
INTERFACE:
subroutine ESMF_ArrayBundleSMM(srcArrayBundle, dstArrayBundle, &
routehandle, zeroregion, termorderflag, checkflag, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in), optional :: srcArrayBundle
type(ESMF_ArrayBundle), intent(inout), optional :: dstArrayBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Region_Flag), intent(in), optional :: zeroregion
type(ESMF_TermOrder_Flag), intent(in), optional :: termorderflag(:)
logical, intent(in), optional :: checkflag
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Execute a precomputed ArrayBundle sparse matrix multiplication from the Arrays in srcArrayBundle to the Arrays in dstArrayBundle.
The srcArrayBundle and dstArrayBundle arguments are optional in support of the situation where srcArrayBundle and/or dstArrayBundle are not defined on all PETs. The srcArrayBundle and dstArrayBundle must be specified on those PETs that hold source or destination DEs, respectively, but may be omitted on all other PETs. PETs that hold neither source nor destination DEs may omit both arguments.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayBundleSMMRelease(routehandle, rc)ARGUMENTS:
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Release resources associated with an ArrayBundle sparse matrix multiplication. After this call routehandle becomes invalid.
INTERFACE:
! Private name; call using ESMF_ArrayBundleSMMStore()
subroutine ESMF_ArrayBundleSMMStore<type><kind>(srcArrayBundle, &
dstArrayBundle, routehandle, factorList, factorIndexList, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: srcArrayBundle
type(ESMF_ArrayBundle), intent(inout) :: dstArrayBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
<type>(ESMF_KIND_<kind>), target, intent(in) :: factorList(:)
integer, intent(in) :: factorIndexList(:,:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store an ArrayBundle sparse matrix multiplication operation from srcArrayBundle to dstArrayBundle. The sparse matrix multiplication between ArrayBundles is defined as the sequence of individual Array sparse matrix multiplications over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.
The effect of this method on ArrayBundles that contain aliased members is undefined.
PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
See the description of method ESMF_ArraySMMStore() for the definition of the Array based operation.
The routine returns an ESMF_RouteHandle that can be used to call ESMF_ArrayBundleSMM() on any pair of ArrayBundles that are weakly congruent and typekind conform with the Arrays contained in srcArrayBundle and dstArrayBundle. Congruent Arrays possess matching DistGrids, and the shape of the local array tiles, i.e. the memory allocation, matches between the Arrays for every DE. For weakly congruent Arrays the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. This means that the same routehandle can be applied to a large class of similar Arrays that differ in the number of elements in the left most undistributed dimensions.
This method is overloaded for:
ESMF_TYPEKIND_I4, ESMF_TYPEKIND_I8,
ESMF_TYPEKIND_R4, ESMF_TYPEKIND_R8.
This call is collective across the current VM.
The second dimension of factorIndexList steps through the list of pairs, i.e. size(factorIndexList,2) == size(factorList). The first dimension of factorIndexList is either of size 2 or size 4.
In the size 2 format factorIndexList(1,:) specifies the sequence index of the source element in the source Array while factorIndexList(2,:) specifies the sequence index of the destination element in the destination Array. For this format to be a valid option source and destination Arrays must have matching number of tensor elements (the product of the sizes of all Array tensor dimensions). Under this condition an identity matrix can be applied within the space of tensor elements for each sparse matrix factor.
The size 4 format is more general and does not require a matching tensor element count. Here the factorIndexList(1,:) specifies the sequence index while factorIndexList(2,:) specifies the tensor sequence index of the source element in the source Array. Further factorIndexList(3,:) specifies the sequence index and factorIndexList(4,:) specifies the tensor sequence index of the destination element in the destination Array.
See section 28.2.17 for details on the definition of Array sequence indices and tensor sequence indices.
INTERFACE:
! Private name; call using ESMF_ArrayBundleSMMStore()
subroutine ESMF_ArrayBundleSMMStoreNF(srcArrayBundle, dstArrayBundle, &
routehandle, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: srcArrayBundle
type(ESMF_ArrayBundle), intent(inout) :: dstArrayBundle
type(ESMF_RouteHandle), intent(inout) :: routehandle
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Store an ArrayBundle sparse matrix multiplication operation from srcArrayBundle to dstArrayBundle. The sparse matrix multiplication between ArrayBundles is defined as the sequence of individual Array sparse matrix multiplications over all source and destination Array pairs in sequence. The method requires that srcArrayBundle and dstArrayBundle reference an identical number of ESMF_Array objects.
The effect of this method on ArrayBundles that contain aliased members is undefined.
PETs that specify non-zero matrix coefficients must use the <type><kind> overloaded interface and provide the factorList and factorIndexList arguments. Providing factorList and factorIndexList arguments with size(factorList) = (/0/) and size(factorIndexList) = (/2,0/) or (/4,0/) indicates that a PET does not provide matrix elements. Alternatively, PETs that do not provide matrix elements may also call into the overloaded interface without factorList and factorIndexList arguments.
See the description of method ESMF_ArraySMMStore() for the definition of the Array based operation.
The routine returns an ESMF_RouteHandle that can be used to call
ESMF_ArrayBundleSMM() on any pair of ArrayBundles that
are weakly congruent and typekind conform with the Arrays contained in
srcArrayBundle and dstArrayBundle.
Congruent Arrays possess matching DistGrids, and the shape of the local
array tiles, i.e. the memory allocation, matches between the Arrays for
every DE. For weakly congruent
Arrays the sizes of the undistributed dimensions, that vary faster with
memory than the first distributed dimension, are permitted to be different.
This means that the same routehandle can be applied to a large class
of similar Arrays that differ in the number of elements in the left most
undistributed dimensions.
This call is collective across the current VM.
INTERFACE:
subroutine ESMF_ArrayBundleWrite(arraybundle, fileName, &
singleFile, overwrite, status, timeslice, iofmt, rc)
ARGUMENTS:
type(ESMF_ArrayBundle), intent(in) :: arraybundle
character(*), intent(in) :: fileName
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: singleFile
logical, intent(in), optional :: overwrite
type(ESMF_FileStatus_Flag), intent(in), optional :: status
integer, intent(in), optional :: timeslice
type(ESMF_IOFmt_Flag), intent(in), optional :: iofmt
integer, intent(out), optional :: rc
DESCRIPTION:
Write the Arrays into a file. For this API to be functional, the environment variable ESMF_PIO should be set to "internal" when the ESMF library is built. Please see the section on Data I/O, 37.3.
Limitations:
The arguments are:
The Array class is an alternative to the Field class for representing distributed, structured data. Unlike Fields, which are built to carry grid coordinate information, Arrays can only carry information about the indices associated with grid cells. Since they do not have coordinate information, Arrays cannot be used to calculate interpolation weights. However, if the user can supply interpolation weights, the Array sparse matrix multiply operation can be used to apply the weights and transfer data to the new grid. Arrays can also perform redistribution, scatter, and gather communication operations.
Like Fields, Arrays can be added to a State and used in inter-Component data communications. Arrays can also be grouped together into ArrayBundles so that collective operations can be performed on the whole group. One motivation for this is convenience; another is the ability to schedule optimized, collective data transfers.
From a technical standpoint, the ESMF Array class is an index space based, distributed data storage class. It provides DE-local memory allocations within DE-centric index regions and defines the relationship to the index space described by the ESMF DistGrid. The Array class offers common communication patterns within the index space formalism. As part of the ESMF index space layer, Array has close relationship to the DistGrid and DELayout classes.
An ESMF_Array is a distributed object that must exist on all PETs of the current context. Each PET-local instance of an Array object contains memory allocations for all PET-local DEs. There may be 0, 1, or more DEs per PET and the number of DEs per PET can differ between PETs for the same Array object. Memory allocations may be provided for each PET by the user during Array creation or can be allocated as part of the Array create call. Many of the concepts of the proposed ESMF_Array class are illustrated by the following examples.
The create call of the ESMF_Array class has been overloaded extensively to facilitate the need for generality while keeping simple cases simple. The following program demonstrates one of the simpler cases, where existing local Fortran arrays are to be used to provide the PET-local memory allocations for the Array object.
program ESMF_ArrayFarrayEx #include "ESMF.h" use ESMF use ESMF_TestMod implicit none
The Fortran language provides a variety of ways to define and allocate an array. Actual Fortran array objects must either be explicit-shape or deferred-shape. In the first case the memory allocation and deallocation is automatic from the user's perspective and the details of the allocation (static or dynamic, heap or stack) are left to the compiler. (Compiler flags may be used to control some of the details). In the second case, i.e. for deferred-shape actual objects, the array definition must include the pointer or allocatable attribute and it is the user's responsibility to allocate memory. While it is also the user's responsibility to deallocate memory for arrays with the pointer attribute the compiler will automatically deallocate allocatable arrays under certain circumstances defined by the Fortran standard.
The ESMF_ArrayCreate() interface has been written to accept native Fortran arrays of any flavor as a means to allow user-controlled memory management. The Array create call will check on each PET if sufficient memory has been provided by the specified Fortran arrays and will indicate an error if a problem is detected. However, the Array create call cannot validate the lifetime of the provided memory allocations. If, for instance, an Array object was created in a subroutine from an automatic explicit-shape array or an allocatable array, the memory allocations referenced by the Array object will be automatically deallocated on return from the subroutine unless provisions are made by the application writer to prevent such behavior. The Array object cannot control when memory that has been provided by the user during Array creation becomes deallocated, however, the Array will indicate an error if its memory references have been invalidated.
The easiest, portable way to provide safe native Fortran memory allocations to Array create is to use arrays with the pointer attribute. Memory allocated for an array pointer will not be deallocated automatically. However, in this case the possibility of memory leaks becomes an issue of concern. The deallocation of memory provided to an Array in form of a native Fortran allocation will remain the users responsibility.
None of the concerns discussed above are an issue in this example where the
native Fortran array farray is defined in the main program. All
different types of array memory allocation are demonstrated in this example.
First farrayE is defined as a 2D explicit-shape array on each PET which
will automatically provide memory for 
elements.
! local variables real(ESMF_KIND_R8) :: farrayE(10,10) ! explicit shape Fortran array
Then an allocatable array farrayA is declared which will be used to show user-controlled dynamic memory allocation.
real(ESMF_KIND_R8), allocatable :: farrayA(:,:) ! allocatable Fortran array
Finally an array with pointer attribute farrayP is declared, also used for user-controlled dynamic memory allocation.
real(ESMF_KIND_R8), pointer :: farrayP(:,:) ! Fortran array pointer
A matching array pointer must also be available to gain access to the arrays held by an Array object.
real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching Fortran array ptr type(ESMF_DistGrid) :: distgrid ! DistGrid object type(ESMF_Array) :: array ! Array object integer :: rc
call ESMF_Initialize(defaultlogfilename="ArrayFarrayEx.Log", &
logkindflag=ESMF_LOGKIND_MULTI, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
On each PET farrayE can be accessed directly to initialize the entire PET-local array.
farrayE = 12.45d0 ! initialize to some value
In order to create an Array object a DistGrid must first be created that describes the total index space and how it is decomposed and distributed. In the simplest case only the minIndex and maxIndex of the total space must be provided.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
This example is assumed to run on 4 PETs. The default 2D decomposition will then be into 4 x 1 DEs as to ensure 1 DE per PET.
Now the Array object can be created using the farrayE and the DistGrid just created.
array = ESMF_ArrayCreate(farray=farrayE, distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
The 40 x 10 index space defined by the minIndex and maxIndex arguments paired with the default decomposition will result in the following distributed Array.
+---------------------------> 2nd dimension
| (1,1)-------+
| | |
| | DE 0 | <--- farray on PET 0
| | |
| +------(10,10)
| (11,1)-------+
| | |
| | DE 1 | <--- farray on PET 1
| | |
| +------(20,10)
| (21,1)-------+
| | |
| | DE 2 | <--- farray on PET 2
| | |
| +------(30,10)
| (31,1)-------+
| | |
| | DE 3 | <--- farray on PET 3
| | |
| +------(40,10)
v
1st dimension
Providing farrayE during Array creation does not change anything about the actual farrayE object. This means that each PET can use its local farrayE directly to access the memory referenced by the Array object.
print *, farrayE
Another way of accessing the memory associated with an Array object is to use ArrayGet() to obtain an Fortran pointer that references the PET-local array.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
print *, farrayPtr
Finally the Array object must be destroyed. The PET-local memory of the farrayEs will remain in user control and will not be altered by ArrayDestroy().
call ESMF_ArrayDestroy(array, rc=rc)
Since the memory allocation for each farrayE is automatic there is nothing more to do.
The interaction between farrayE and the Array class is representative also for the two other cases farrayA and farrayP. The only difference is in the handling of memory allocations.
allocate(farrayA(10,10)) ! user controlled allocation
farrayA = 23.67d0 ! initialize to some value
array = ESMF_ArrayCreate(farray=farrayA, distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
print *, farrayA ! print PET-local farrayA directly call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)! obtain array pointer print *, farrayPtr ! print PET-local piece of Array through pointer call ESMF_ArrayDestroy(array, rc=rc) ! destroy the Array deallocate(farrayA) ! user controlled de-allocation
The farrayP case is identical.
allocate(farrayP(10,10)) ! user controlled allocation
farrayP = 56.81d0 ! initialize to some value
array = ESMF_ArrayCreate(farray=farrayP, distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
print *, farrayP ! print PET-local farrayA directly call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)! obtain array pointer print *, farrayPtr ! print PET-local piece of Array through pointer call ESMF_ArrayDestroy(array, rc=rc) ! destroy the Array deallocate(farrayP) ! user controlled de-allocation
To wrap things up the DistGrid object is destroyed and ESMF can be finalized.
call ESMF_DistGridDestroy(distgrid, rc=rc) ! destroy the DistGrid
call ESMF_Finalize(rc=rc)
end program
The example of the previous section showed how easy it is to create an Array object from existing PET-local Fortran arrays. The example did, however, not define any halo elements around the DE-local regions. The following code demonstrates how an Array object with space for a halo can be set up.
program ESMF_ArrayFarrayHaloEx #include "ESMF.h" use ESMF use ESMF_TestMod implicit none
The allocatable array farrayA will be used to provide the PET-local Fortran array for this example.
! local variables real(ESMF_KIND_R8), allocatable :: farrayA(:,:) ! allocatable Fortran array real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching Fortran array ptr type(ESMF_DistGrid) :: distgrid ! DistGrid object type(ESMF_Array) :: array ! Array object integer :: rc, i, j real :: localSum
call ESMF_Initialize(defaultlogfilename="ArrayFarrayHaloEx.Log", &
logkindflag=ESMF_LOGKIND_MULTI, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
The Array is to cover the exact same index space as in the previous example. Furthermore decomposition and distribution are also kept the same. Hence the same DistGrid object will be created and it is expected to execute this example with 4 PETs.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
This DistGrid describes a 40 x 10 index space that will be decomposed into 4 DEs when executed on 4 PETs, associating 1 DE per PET. Each DE-local exclusive region contains 10 x 10 elements. The DistGrid also stores and provides information about the relationship between DEs in index space, however, DistGrid does not contain information about halos. Arrays contain halo information and it is possible to create multiple Arrays covering the same index space with identical decomposition and distribution using the same DistGrid object, while defining different, Array-specific halo regions.
The extra memory required to cover the halo in the Array object must be taken into account when allocating the PET-local farrayA arrays. For a halo of 2 elements in each direction the following allocation will suffice.
allocate(farrayA(14,14)) ! Fortran array with halo: 14 = 10 + 2 * 2
The farrayA can now be used to create an Array object with enough space for a two element halo in each direction. The Array creation method checks for each PET that the local Fortran array can accommodate the requested regions.
The default behavior of ArrayCreate() is to center the exclusive region within the total region. Consequently the following call will provide the 2 extra elements on each side of the exclusive 10 x 10 region without having to specify any additional arguments.
array = ESMF_ArrayCreate(farray=farrayA, distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
The exclusive Array region on each PET can be accessed through a suitable Fortran array pointer. See section 28.2.6 for more details on Array regions.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
Following Array bounds convention, which by default puts the beginning of the exclusive region at (1, 1, ...), the following loop will add up the values of the local exclusive region for each DE, regardless of how the bounds were chosen for the original PET-local farrayA arrays.
localSum = 0.
do j=1, 10
do i=1, 10
localSum = localSum + farrayPtr(i, j)
enddo
enddo
Elements with 
or 
in the [-1,0] or [11,12] ranges are located outside the
exclusive region and may be used to define extra computational points or
halo operations.
Cleanup and shut down ESMF.
call ESMF_ArrayDestroy(array, rc=rc)
deallocate(farrayA) call ESMF_DistGridDestroy(distgrid, rc=rc)
call ESMF_Finalize(rc=rc)
end program
Alternative to the direct usage of Fortran arrays during Array creation it is also possible to first create an ESMF_LocalArray and create the Array from it. While this may seem more burdensome for the 1 DE per PET cases discussed in the previous sections it allows a straight forward generalization to the multiple DE per PET case. The following example first recaptures the previous example using an ESMF_LocalArray and then expands to the multiple DE per PET case.
program ESMF_ArrayLarrayEx #include "ESMF.h" use ESMF use ESMF_TestMod implicit none
The current ESMF_LocalArray interface requires Fortran arrays to be defined with pointer attribute.
! local variables real(ESMF_KIND_R8), pointer :: farrayP(:,:) ! Fortran array pointer real(ESMF_KIND_R8), pointer :: farrayPtr(:,:) ! matching Fortran array ptr type(ESMF_LocalArray) :: larray ! ESMF_LocalArray object type(ESMF_LocalArray) :: larrayRef ! ESMF_LocalArray object type(ESMF_DistGrid) :: distgrid ! DistGrid object type(ESMF_Array) :: array ! Array object integer :: rc, i, j, de real :: localSum type(ESMF_LocalArray), allocatable :: larrayList(:) ! LocalArray object list type(ESMF_LocalArray), allocatable :: larrayRefList(:)!LocalArray obj. list type(ESMF_VM):: vm integer:: localPet, petCount
call ESMF_Initialize(vm=vm, defaultlogfilename="ArrayLarrayEx.Log", &
logkindflag=ESMF_LOGKIND_MULTI, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
call ESMF_VMGet(vm, localPet=localPet, petCount=petCount, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
if (petCount /= 4) then
finalrc = ESMF_FAILURE
goto 10
endif
DistGrid and array allocation remains unchanged.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
allocate(farrayP(14,14)) ! allocate Fortran array on each PET with halo
Now instead of directly creating an Array object using the PET-local farrayPs an ESMF_LocalArray object will be created on each PET.
larray = ESMF_LocalArrayCreate(farrayP, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
The Array object can now be created from larray. The Array creation method checks for each PET that the LocalArray can accommodate the requested regions.
array = ESMF_ArrayCreate(localarrayList=(/larray/), distgrid=distgrid, rc=rc)
Once created there is no difference in how the Array object can be used. The exclusive Array region on each PET can be accessed through a suitable Fortran array pointer as before.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
Alternatively it is also possible (independent of how the Array object was created) to obtain the reference to the array allocation held by Array in form of an ESMF_LocalArray object. The farrayPtr can then be extracted using LocalArray methods.
call ESMF_ArrayGet(array, localarray=larrayRef, rc=rc)
call ESMF_LocalArrayGet(larrayRef, farrayPtr, rc=rc)
Either way the farrayPtr reference can be used now to add up the values of the local exclusive region for each DE. The following loop works regardless of how the bounds were chosen for the original PET-local farrayP arrays and consequently the PET-local larray objects.
localSum = 0.
do j=1, 10
do i=1, 10
localSum = localSum + farrayPtr(i, j)
enddo
enddo
print *, "localSum=", localSum
Cleanup.
call ESMF_ArrayDestroy(array, rc=rc) call ESMF_LocalArrayDestroy(larray, rc=rc) deallocate(farrayP) ! use the pointer that was used in allocate statement call ESMF_DistGridDestroy(distgrid, rc=rc)
While the usage of LocalArrays is unnecessarily cumbersome for 1 DE per PET Arrays, it provides a straight forward path for extending the interfaces to multiple DEs per PET.
In the following example a 8 x 8 index space will be decomposed into 2 x 4 = 8 DEs. The situation is captured by the following DistGrid object.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/8,8/), &
regDecomp=(/2,4/), rc=rc)
The distgrid object created in this manner will contain 8 DEs no matter how many PETs are available during execution. Assuming an execution on 4 PETs will result in the following distribution of the decomposition.
+---------------------------------------> 2nd dimension
| (1,1)
| +-----------+-----------+-----------+-----------+
| | DE0, PET0 | DE2, PET1 | DE4, PET2 | DE6, PET3 |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| +-----------+-----------+-----------+-----------+
| | DE1, PET0 | DE3, PET1 | DE5, PET2 | DE7, PET3 |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| | | | | |
| | * * | * * | * * | * * |
| +-----------+-----------+-----------+-----------+
| (8,8)
v
1st dimension
Obviously each PET is associated with 2 DEs. Each PET must allocate enough space for all its DEs. This is done by allocating as many DE-local arrays as there are DEs on the PET. The reference to these array allocations is passed into ArrayCreate via a LocalArray list argument that holds as many elements as there are DEs on the PET. Here each PET must allocate for two DEs.
allocate(larrayList(2)) ! 2 DEs per PET
allocate(farrayP(4, 2)) ! without halo each DE is of size 4 x 2
farrayP = 123.456d0
larrayList(1) = ESMF_LocalArrayCreate(farrayP, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc) !1st DE
allocate(farrayP(4, 2)) ! without halo each DE is of size 4 x 2
farrayP = 456.789d0
larrayList(2) = ESMF_LocalArrayCreate(farrayP, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc) !2nd DE
Notice that it is perfectly fine to re-use farrayP for all allocations of DE-local Fortran arrays. The allocated memory can be deallocated at the end using the array pointer contained in the larrayList.
With this information an Array object can be created. The distgrid object indicates 2 DEs for each PET and ArrayCreate() expects to find two LocalArray elements in larrayList.
array = ESMF_ArrayCreate(localarrayList=larrayList, distgrid=distgrid, rc=rc)
Usage of a LocalArray list is the only way to provide a list of variable length of Fortran array allocations to ArrayCreate() for each PET. The array object created by the above call is an ESMF distributed object. As such it must follow the ESMF convention that requires that the call to ESMF_ArrayCreate() must be issued in unison by all PETs of the current context. Each PET only calls ArrayCreate() once, even if there are multiple DEs per PET.
The ArrayGet() method provides access to the list of LocalArrays on each PET.
allocate(larrayRefList(2)) call ESMF_ArrayGet(array, localarrayList=larrayRefList, rc=rc)
Finally, access to the actual Fortran pointers is done on a per DE basis. Generally each PET will loop over its DEs.
do de=1, 2
call ESMF_LocalArrayGet(larrayRefList(de), farrayPtr, rc=rc)
localSum = 0.
do j=1, 2
do i=1, 4
localSum = localSum + farrayPtr(i, j)
enddo
enddo
print *, "localSum=", localSum
enddo
Note: If the VM associates multiple PEs with a PET the application writer may decide to use OpenMP loop parallelization on the de loop.
Cleanup requires that the PET-local deallocations are done before the pointers to the actual Fortran arrays are lost. Notice that larrayList is used to obtain the pointers used in the deallocate statement. Pointers obtained from the larrayRefList, while pointing to the same data, cannot be used to deallocated the array allocations!
do de=1, 2
call ESMF_LocalArrayGet(larrayList(de), farrayPtr, rc=rc)
deallocate(farrayPtr)
call ESMF_LocalArrayDestroy(larrayList(de), rc=rc)
enddo deallocate(larrayList) deallocate(larrayRefList) call ESMF_ArrayDestroy(array, rc=rc) if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT) call ESMF_DistGridDestroy(distgrid, rc=rc) if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
With that ESMF can be shut down cleanly.
call ESMF_Finalize(rc=rc)
end program
In the examples of the previous sections the user provided memory allocations for each of the DE-local regions for an Array object. The user was able to use any of the Fortran methods to allocate memory, or go through the ESMF_LocalArray interfaces to obtain memory allocations before passing them into ArrayCreate(). Alternatively ESMF offers methods that handle Array memory allocations inside the library.
As before, to create an ESMF_Array object an ESMF_DistGrid must be created. The DistGrid object holds information about the entire index space and how it is decomposed into DE-local exclusive regions. The following line of code creates a DistGrid for a 5x5 global index space that is decomposed into 2 x 3 = 6 DEs.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
The following is a representation of the index space and its decomposition into DEs. Each asterisk (*) represents a single element.
+---------------------------------------> 2nd dimension
| (1,1)
| +-----------+-----------+------+
| | DE 0 | DE 2 | DE 4 |
| | | | |
| | * * | * * | * |
| | | | |
| | * * | * * | * |
| | | | |
| | * * | * * | * |
| +-----------+-----------+------+
| | | | |
| | DE 1 | DE 3 | DE 5 |
| | | | |
| | * * | * * | * |
| | | | |
| | * * | * * | * |
| +-----------+-----------+------+
| (5,5)
v
1st dimension
Besides the DistGrid it is the type, kind and rank information, "tkr" for short, that is required to create an Array object. It turns out that the rank of the Array object is fully determined by the DistGrid and other (optional) arguments passed into ArrayCreate(), so that explicit specification of the Array rank is redundant.
The simplest way to supply the type and kind information of the Array is directly through the typekind argument. Here a double precision Array is created on the previously created DistGrid. Since no other arguments are specified that could alter the rank of the Array it becomes equal to the dimCount of the DistGrid, i.e a 2D Array is created on top of the DistGrid.
array = ESMF_ArrayCreate(typekind=ESMF_TYPEKIND_R8, distgrid=distgrid, rc=rc)
The different methods on how an Array object is created have no effect on the use of ESMF_ArrayDestroy().
call ESMF_ArrayDestroy(array, rc=rc)
Alternatively the same Array can be created specifying the "tkr" information in form of an ArraySpec variable. The ArraySpec explicitly contains the Array rank and thus results in an over specification on the ArrayCreate() interface. ESMF checks all input information for consistency and returns appropriate error codes in case any inconsistencies are found.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
The Array object created by the above call is an ESMF distributed object. As such it must follow the ESMF convention that requires that the call to ESMF_ArrayCreate() must be issued in unison by all PETs of the current context.
There are two different methods by which the user can access the data held inside an ESMF Array object. The first method provides direct access to a native language array object. Specifically, the farrayPtr argument returned by ESMF_ArrayGet() is a Fortran array pointer that can be used to access the PET-local data inside the Array object.
Many applications work in the 1 DE per PET mode, with exactly one DE on every PET. Accessing the Array memory on each PET for this situation is especially simple as is shown in section 28.2.1. However, the Array class is not restricted to the special 1 DE per PET case, but supports multiple separate memory allocations on each PET. The number of such PET-local allocations is given by the localDeCount, i.e. there is one memory allocation for every DE that is associated with the local PET.
Access to a specific local memory allocation of an Array object is still
accomplished by returning the farrayPtr argument. However, for
the formally optional localDe argument to
ESMF_ArrayGet() turns into a practically required argument. While
in general the localDe in ESMF is simply a local index variable that
enumerates the DEs that are associated with the local PET (e.g. see section
48.3.7), the bounds of this index variable are
strictly defined as [0,...,localDeCount-1] when it is used as an
input argument. The following code demonstrates this.
First query the Array for localDeCount. This number may be different on each PET and indicates how many DEs are mapped against the local PET.
call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc)
Looping the localDe index variable from 0 to localDeCount-1 allows access to each of the local memory allocations of the Array object:
do localDe=0, localDeCount-1
call ESMF_ArrayGet(array, farrayPtr=myFarray, localDe=localDe, rc=rc)
! use myFarray to access local DE data
enddo
The second method to access the memory allocations in an Array object is to go through the ESMF LocalArray object. To this end the Array is queried for a list of PET-local LocalArray objects. The LocalArray objects in the list correspond to the DEs on the local PET. Here the localDe argument is solely a user level index variable, and in principle the lower bound can be chosen freely. However, for better alignment with the previous case (where localDe served as an input argument to an ESMF method) the following example again fixes the lower bound at zero.
allocate(larrayList(0:localDeCount-1)) call ESMF_ArrayGet(array, localarrayList=larrayList, rc=rc)
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
! use myFarray to access local DE data
enddo
See section 28.2.3 for more on LocalArray usage in Array. In most cases memory access through a LocalArray list is less convenient than the direct farrayPtr method because it adds an extra object level between the ESMF Array and the native language array.
Each ESMF_Array object is decomposed into DEs as specified by the associated ESMF_DistGrid object. Each piece of this decomposition, i.e. each DE, holds a chunk of the Array data in its own local piece of memory. The details of the Array decomposition are described in the following paragraphs.
At the center of the Array decomposition is the ESMF_DistGrid class. The DistGrid object specified during Array creation contains three essential pieces of information:
Each element of an Array is associated with a single DE. The union of elements associated with a DE, as defined by the DistGrid above, corresponds to a LR chunk of index space, called the exclusive region of the DE.
There is a hierarchy of four regions that can be identified for each DE in an Array object. Their definition and relationship to each other is as follows:
+-totalLBound(:)----------------------------------+
|\ |
| \ <--- totalLWidth(:) |
| \ |
| +-computationalLBound(:)------------------+ |
| |\ | |
| | \ <--- computationalLWidth(:) | |
| | \ | |
| | +-exclusiveLBound(:)-------------+ | |
| | | | | |
| | | +------+ +-----+ | | |
| | | | | | | | | |
| | | | +------+ | | | |
| | | | "Interior Region" | | | |
| | | +-----+ | | | |
| | | | | | | |
| | | +-------------+ | | |
| | | | | |
| | | "Exclusive Region" | | |
| | +-------------exclusiveUBound(:)-+ | |
| | \ | |
| | computationalUWidth(:) --> \ | |
| | \ | |
| | "Computational Region" \| |
| +------------------computationalUBound(:)-+ |
| \ |
| totalUWidth(:) -> \ |
| "Total Region" \|
+--------------------------------- totalUBound(:)-+
With the following definitions:
computationalLWidth(:) = exclusiveLBound(:) - computationalLBound(:) computationalUWidth(:) = computationalUBound(:) - exclusiveUBound(:)and
totalLWidth(:) = exclusiveLBound(:) - totalLBound(:) totalUWidth(:) = totalUBound(:) - exclusiveUBound(:)
The exclusive region is determined during Array creation by the DistGrid argument. Optional arguments may be used to specify the computational region when the Array is created, by default it will be set equal to the exclusive region. The total region, i.e. the actual memory allocation for each DE, is also determined during Array creation. When creating the Array object from existing Fortran arrays the total region is set equal to the memory provided by the Fortran arrays. Otherwise the default is to allocate as much memory as is needed to accommodate the union of the DE-local exclusive and computational region. Finally it is also possible to use optional arguments to the ArrayCreate() call to specify the total region of the object explicitly.
The ESMF_ArrayCreate() call checks that the input parameters are consistent and will result in an Array that fulfills all of the above mentioned requirements for its DE-local regions.
Once an Array object has been created the exclusive and total regions are fixed. The computational region, however, may be adjusted within the limits of the total region using the ArraySet() call.
The interior region is very different from the other regions in that it cannot be specified. The interior region for each DE is a consequence of the choices made for the other regions collectively across all DEs into which an Array object is decomposed. An Array object can be queried for its DE-local interior regions as to offer additional information to the user necessary to write more efficient code.
By default the bounds of each DE-local total region are defined as to put the start of the DE-local exclusive region at the "origin" of the local index space, i.e. at (1, 1, ..., 1). With that definition the following loop will access each element of the DE-local memory segment for each PET-local DE of the Array object used in the previous sections and print its content.
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
do i=1, size(myFarray, 1)
do j=1, size(myFarray, 2)
print *, "localPET=", localPet, " localDE=", &
localDe, ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
enddo
The loop over Array elements at the end of the last section only works correctly because of the default definition of the computational and total regions used in the example. In general, without such specific knowledge about an Array object, it is necessary to use a more formal approach to access its regions with DE-local indices.
The DE-local exclusive region takes a central role in the definition of Array bounds. Even as the computational region may adjust during the course of execution the exclusive region remains unchanged. The exclusive region provides a unique reference frame for the index space of all Arrays associated with the same DistGrid.
There is a choice between two indexing options that needs to be made during Array creation. By default each DE-local exclusive region starts at (1, 1, ..., 1). However, for some computational kernels it may be more convenient to choose the index bounds of the DE-local exclusive regions to match the index space coordinates as they are defined in the corresponding DistGrid object. The second option is only available if the DistGrid object does not contain any non-contiguous decompositions (such as cyclically decomposed dimensions).
The following example code demonstrates the safe way of dereferencing the DE-local exclusive regions of the previously created array object.
allocate(exclusiveUBound(2, 0:localDeCount-1)) ! dimCount=2
allocate(exclusiveLBound(2, 0:localDeCount-1)) ! dimCount=2
call ESMF_ArrayGet(array, indexflag=indexflag, &
exclusiveLBound=exclusiveLBound, exclusiveUBound=exclusiveUBound, rc=rc)
if (indexflag == ESMF_INDEX_DELOCAL) then
! this is the default
! print *, "DE-local exclusive regions start at (1,1)"
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
do i=1, exclusiveUBound(1, localDe)
do j=1, exclusiveUBound(2, localDe)
! print *, "DE-local exclusive region for localDE=", localDe, &
! ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
enddo
else if (indexflag == ESMF_INDEX_GLOBAL) then
! only if set during ESMF_ArrayCreate()
! print *, "DE-local exclusive regions of this Array have global bounds"
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
do i=exclusiveLBound(1, localDe), exclusiveUBound(1, localDe)
do j=exclusiveLBound(2, localDe), exclusiveUBound(2, localDe)
! print *, "DE-local exclusive region for localDE=", localDe, &
! ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
enddo
endif
call ESMF_ArrayDestroy(array, rc=rc) ! destroy the array object
Obviously the second branch of this simple code will work for either case, however, if a complex computational kernel was written assuming ESMF_INDEX_DELOCAL type bounds the second branch would simply be used to indicate the problem and bail out.
The advantage of the ESMF_INDEX_GLOBAL index option is that the Array bounds directly contain information on where the DE-local Array piece is located in a global index space sense. When the ESMF_INDEX_DELOCAL option is used the correspondence between local and global index space must be made by querying the associated DistGrid for the DE-local indexList arguments.
In the previous examples the computational region of array was chosen by default to be identical to the exclusive region defined by the DistGrid argument during Array creation. In the following the same arrayspec and distgrid objects as before will be used to create an Array but now a larger computational region shall be defined around each DE-local exclusive region. Furthermore, extra space will be defined around the computational region of each DE to accommodate a halo and/or serve as memory padding.
In this example the indexflag argument is set to ESMF_INDEX_GLOBAL indicating that the bounds of the exclusive region correspond to the index space coordinates as they are defined by the DistGrid object.
The same arrayspec and distgrid objects as before are used which also allows the reuse of the already allocated larrayList variable.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
computationalLWidth=(/0,3/), computationalUWidth=(/1,1/), &
totalLWidth=(/1,4/), totalUWidth=(/3,1/), &
indexflag=ESMF_INDEX_GLOBAL, rc=rc)
Obtain the larrayList on every PET.
allocate(localDeToDeMap(0:localDeCount-1))
call ESMF_ArrayGet(array, localarrayList=larrayList, &
localDeToDeMap=localDeToDeMap, rc=rc)
The bounds of DE 1 for array are shown in the following diagram to illustrate the situation. Notice that the totalLWidth and totalUWidth arguments in the ArrayCreate() call define the total region with respect to the exclusive region given for each DE by the distgrid argument.
+-(3,-3)---------------------------------+
|\ |
| +-(4,-2)-+-(4,1)--------------------+--+
| | | | |
| | | | |
| | | DE 1 | |
| | | | |
| | | | |
| | | Exclusive Region | |
| | +--------------------(5,2)-+ |
| | Computational Region |
| +-------------------------------(6,3)--+
| |
| Total Region |
+---------------------------------(8,3)--+
When working with this array it is possible for the computational kernel to overstep the exclusive region for both read/write access (computational region) and potentially read-only access into the total region outside of the computational region, if a halo operation provides valid entries for these elements.
The Array object can be queried for absolute bounds
allocate(computationalLBound(2, 0:localDeCount-1)) ! dimCount=2
allocate(computationalUBound(2, 0:localDeCount-1)) ! dimCount=2
allocate(totalLBound(2, 0:localDeCount-1)) ! dimCount=2
allocate(totalUBound(2, 0:localDeCount-1)) ! dimCount=2
call ESMF_ArrayGet(array, exclusiveLBound=exclusiveLBound, &
exclusiveUBound=exclusiveUBound, &
computationalLBound=computationalLBound, &
computationalUBound=computationalUBound, &
totalLBound=totalLBound, &
totalUBound=totalUBound, rc=rc)
or for the relative widths.
allocate(computationalLWidth(2, 0:localDeCount-1)) ! dimCount=2
allocate(computationalUWidth(2, 0:localDeCount-1)) ! dimCount=2
allocate(totalLWidth(2, 0:localDeCount-1)) ! dimCount=2
allocate(totalUWidth(2, 0:localDeCount-1)) ! dimCount=2
call ESMF_ArrayGet(array, computationalLWidth=computationalLWidth, &
computationalUWidth=computationalUWidth, totalLWidth=totalLWidth, &
totalUWidth=totalUWidth, rc=rc)
Either way the dereferencing of Array data is centered around the DE-local exclusive region:
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
! initialize the DE-local array
myFarray = 0.1d0 * localDeToDeMap(localDe)
! first time through the total region of array
! print *, "myFarray bounds for DE=", localDeToDeMap(localDe), &
! lbound(myFarray), ubound(myFarray)
do j=exclusiveLBound(2, localDe), exclusiveUBound(2, localDe)
do i=exclusiveLBound(1, localDe), exclusiveUBound(1, localDe)
! print *, "Excl region DE=", localDeToDeMap(localDe), &
! ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
do j=computationalLBound(2, localDe), computationalUBound(2, localDe)
do i=computationalLBound(1, localDe), computationalUBound(1, localDe)
! print *, "Excl region DE=", localDeToDeMap(localDe), &
! ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
do j=totalLBound(2, localDe), totalUBound(2, localDe)
do i=totalLBound(1, localDe), totalUBound(1, localDe)
! print *, "Total region DE=", localDeToDeMap(localDe), &
! ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
! second time through the total region of array
do j=exclusiveLBound(2, localDe)-totalLWidth(2, localDe), &
exclusiveUBound(2, localDe)+totalUWidth(2, localDe)
do i=exclusiveLBound(1, localDe)-totalLWidth(1, localDe), &
exclusiveUBound(1, localDe)+totalUWidth(1, localDe)
! print *, "Excl region DE=", localDeToDeMap(localDe), &
! ": array(",i,",",j,")=", myFarray(i,j)
enddo
enddo
enddo
All previous examples were written for the 2D case. There is, however, no restriction within the Array or DistGrid class that limits the dimensionality of Array objects beyond the language specific limitations (7D for Fortran).
In order to create an n-dimensional Array the rank indicated by both the arrayspec and the distgrid arguments specified during Array create must be equal to n. A 1D Array of double precision real data hence requires the following arrayspec.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=1, rc=rc)
The index space covered by the Array and the decomposition description is
provided to the Array create method by the distgrid argument. The index
space in this example has 16 elements and covers the interval ![$[-10, 5]$](img47.png){kind=small}
. It is
decomposed into as many DEs as there are PETs in the current context.
distgrid1D = ESMF_DistGridCreate(minIndex=(/-10/), maxIndex=(/5/), &
regDecomp=(/petCount/), rc=rc)
A 1D Array object with default regions can now be created.
array1D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid1D, rc=rc)
The creation of a 3D Array proceeds analogous to the 1D case. The rank of the arrayspec must be changed to 3
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
and an appropriate 3D DistGrid object must be created
distgrid3D = ESMF_DistGridCreate(minIndex=(/1,1,1/), &
maxIndex=(/16,16,16/), regDecomp=(/4,4,4/), rc=rc)
before an Array object can be created.
array3D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid3D, rc=rc)
The distgrid3D object decomposes the 3-dimensional index space into
DEs. These DEs are laid out across the computational
resources (PETs) of the current component according to a default DELayout that
is created during the DistGrid create call. Notice that in the index space
proposal a DELayout does not have a sense of dimensionality. The DELayout
function is simply to map DEs to PETs. The DistGrid maps chunks of index space
against DEs and thus its rank is equal to the number of index space
dimensions.
The previously defined DistGrid and the derived Array object decompose the index space along all three dimension. It is, however, not a requirement that the decomposition be along all dimensions. An Array with the same 3D index space could as well be decomposed along just one or along two of the dimensions. The following example shows how for the same index space only the last two dimensions are decomposed while the first Array dimension has full extent on all DEs.
call ESMF_ArrayDestroy(array3D, rc=rc)
call ESMF_DistGridDestroy(distgrid3D, rc=rc)
distgrid3D = ESMF_DistGridCreate(minIndex=(/1,1,1/), &
maxIndex=(/16,16,16/), regDecomp=(/1,4,4/), rc=rc)
array3D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid3D, rc=rc)
call ESMF_DistGridGet(distgrid3D, delayout=delayout, rc=rc) ! get DELayout
distgrid2D = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/16,16/), &
regDecomp=(/4,4/), delayout=delayout, rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
array2D = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid2D, rc=rc)
Now the following kernel is sure to work with array3D and array2D.
call ESMF_DELayoutGet(delayout, localDeCount=localDeCount, rc=rc)
allocate(larrayList1(0:localDeCount-1))
call ESMF_ArrayGet(array3D, localarrayList=larrayList1, rc=rc)
allocate(larrayList2(0:localDeCount-1))
call ESMF_ArrayGet(array2D, localarrayList=larrayList2, rc=rc)
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList1(localDe), myFarray3D, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
myFarray3D = 0.1d0 * localDe ! initialize
call ESMF_LocalArrayGet(larrayList2(localDe), myFarray2D, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
myFarray2D = 0.5d0 * localDe ! initialize
do k=1, 4
do j=1, 4
dummySum = 0.d0
do i=1, 16
dummySum = dummySum + myFarray3D(i,j,k) ! sum up the (j,k) column
enddo
dummySum = dummySum * myFarray2D(j,k) ! multiply with local 2D element
! print *, "dummySum(",j,k,")=",dummySum
enddo
enddo
enddo
Except for the special Array create interface that implements a copy from an existing Array object all other Array create interfaces require the specification of at least two arguments: farray and distgrid, larrayList and distgrid, or arrayspec and distgrid. In all these cases both required arguments contain a sense of dimensionality. The relationship between these two arguments deserves extra attention.
The first argument, farray, larrayList or arrayspec, determines the rank of the created Array object, i.e. the dimensionality of the actual data storage. The rank of a native language array, extracted from an Array object, is equal to the rank specified by either of these arguments. So is the rank that is returned by the ESMF_ArrayGet() call.
The rank specification contained in the distgrid argument, which is of type ESMF_DistGrid, on the other hand has no affect on the rank of the Array. The dimCount specified by the DistGrid object, which may be equal, greater or less than the Array rank, determines the dimensionality of the decomposition.
While there is no constraint between DistGrid dimCount and Array rank, there is an important relationship between the two, resulting in the concept of index space dimensionality. Array dimensions can be arbitrarily mapped against DistGrid dimension, rendering them decomposed dimensions. The index space dimensionality is equal to the number of decomposed Array dimensions.
Array dimensions that are not mapped to DistGrid dimensions are the undistributed dimensions of the Array. They are not part of the index space. The mapping is specified during ESMF_ArrayCreate() via the distgridToArrayMap argument. DistGrid dimensions that have not been associated with Array dimensions are replicating dimensions. The Array will be replicated across the DEs that lie along replication DistGrid dimensions.
Undistributed Array dimensions can be used to store multi-dimensional data for each Array index space element. One application of this is to store the components of a vector quantity in a single Array. The same 2D distgrid object as before will be used.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
The rank in the arrayspec argument, however, must change from 2 to 3 in order to provide for the extra Array dimension.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
During Array creation with extra dimension(s) it is necessary to specify the bounds of these undistributed dimension(s). This requires two additional arguments, undistLBound and undistUBound, which are vectors in order to accommodate multiple undistributed dimensions. The other arguments remain unchanged and apply across all undistributed components.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
totalLWidth=(/0,1/), totalUWidth=(/0,1/), &
undistLBound=(/1/), undistUBound=(/2/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
This will create array with 2+1 dimensions. The 2D DistGrid is used to describe decomposition into DEs with 2 Array dimensions mapped to the DistGrid dimensions resulting in a 2D index space. The extra Array dimension provides storage for multi component user data within the Array object.
By default the distgrid dimensions are associated with the first Array dimensions in sequence. For the example above this means that the first 2 Array dimensions are decomposed according to the provided 2D DistGrid. The 3rd Array dimension does not have an associated DistGrid dimension, rendering it an undistributed Array dimension.
Native language access to an Array with undistributed dimensions is in principle the same as without extra dimensions.
call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc) allocate(larrayList(0:localDeCount-1)) call ESMF_ArrayGet(array, localarrayList=larrayList, rc=rc)
The following loop shows how a Fortran pointer to the DE-local data chunks can be obtained and used to set data values in the exclusive regions. The myFarray3D variable must be of rank 3 to match the Array rank of array. However, variables such as exclusiveUBound that store the information about the decomposition, remain to be allocated for the 2D index space.
call ESMF_ArrayGet(array, exclusiveLBound=exclusiveLBound, &
exclusiveUBound=exclusiveUBound, rc=rc)
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray3D, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
myFarray3D = 0.0 ! initialize
myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
exclusiveLBound(2,localDe):exclusiveUBound(2,localDe), &
1) = 5.1 ! dummy assignment
myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
exclusiveLBound(2,localDe):exclusiveUBound(2,localDe), &
2) = 2.5 ! dummy assignment
enddo
deallocate(larrayList)
For some applications the default association rules between DistGrid and Array dimensions may not satisfy the user's needs. The optional distgridToArrayMap argument can be used during Array creation to explicitly specify the mapping between DistGrid and Array dimensions. To demonstrate this the following lines of code reproduce the above example but with rearranged dimensions. Here the distgridToArrayMap argument is a list with two elements corresponding to the DistGrid dimCount of 2. The first element indicates which Array dimension the first DistGrid dimension is mapped against. Here the 1st DistGrid dimension maps against the 3rd Array dimension and the 2nd DistGrid dimension maps against the 1st Array dimension. This leaves the 2nd Array dimension to be the extra and undistributed dimension in the resulting Array object.
call ESMF_ArrayDestroy(array, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/3, 1/), totalLWidth=(/0,1/), totalUWidth=(/0,1/), &
undistLBound=(/1/), undistUBound=(/2/), rc=rc)
Operations on the Array object as a whole are unchanged by the different mapping of dimensions.
When working with Arrays that contain explicitly mapped Array and DistGrid dimensions it is critical to know the order in which the entries of width and bound arguments that are associated with distributed Array dimensions are specified. The size of these arguments is equal to the DistGrid dimCount, because the maximum number of distributed Array dimensions is given by the dimensionality of the index space.
The order of dimensions in these arguments, however, is not that of the associated DistGrid. Instead each entry corresponds to the distributed Array dimensions in sequence. In the example above the entries in totalLWidth and totalUWidth correspond to Array dimensions 1 and 3 in this sequence.
The distgridToArrrayMap argument optionally provided during Array create indicates how the DistGrid dimensions map to Array dimensions. The inverse mapping, i.e. Array to DistGrid dimensions, is just as important. The ESMF_ArrayGet() call offers both mappings as distgridToArrrayMap and arrayToDistGridMap, respectively. The number of elements in arrayToDistGridMap is equal to the rank of the Array. Each element corresponds to an Array dimension and indicates the associated DistGrid dimension by an integer number. An entry of "0" in arrayToDistGridMap indicates that the corresponding Array dimension is undistributed.
Correct understanding about the association between Array and DistGrid dimensions becomes critical for correct data access into the Array.
allocate(arrayToDistGridMap(3)) ! arrayRank = 3
call ESMF_ArrayGet(array, arrayToDistGridMap=arrayToDistGridMap, &
exclusiveLBound=exclusiveLBound, exclusiveUBound=exclusiveUBound, &
localDeCount=localDeCount, rc=rc)
if (arrayToDistGridMap(2) /= 0) then ! check if extra dimension at
! expected index indicate problem and bail out
endif
! obtain larrayList for local DEs
allocate(larrayList(0:localDeCount-1))
call ESMF_ArrayGet(array, localarrayList=larrayList, rc=rc)
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray3D, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
1, exclusiveLBound(2,localDe):exclusiveUBound(2, &
localDe)) = 10.5 !dummy assignment
myFarray3D(exclusiveLBound(1,localDe):exclusiveUBound(1,localDe), &
2, exclusiveLBound(2,localDe):exclusiveUBound(2, &
localDe)) = 23.3 !dummy assignment
enddo
deallocate(exclusiveLBound, exclusiveUBound)
deallocate(arrayToDistGridMap)
deallocate(larrayList)
call ESMF_ArrayDestroy(array, rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
Thus far most examples demonstrated cases where the DistGrid dimCount was equal to the Array rank. The previous section introduced the concept of Array tensor dimensions when dimCount < rank. In this section dimCount and rank are assumed completely unconstrained and the relationship to distgridToArrayMap and arrayToDistGridMap will be discussed.
The Array class allows completely arbitrary mapping between Array and DistGrid dimensions. Most cases considered in the previous sections used the default mapping which assigns the DistGrid dimensions in sequence to the lower Array dimensions. Extra Array dimensions, if present, are considered non-distributed tensor dimensions for which the optional undistLBound and undistUBound arguments must be specified.
The optional distgridToArrayMap argument provides the option to override the default DistGrid to Array dimension mapping. The entries of the distgridToArrayMap array correspond to the DistGrid dimensions in sequence and assign a unique Array dimension to each DistGrid dimension. DistGrid and Array dimensions are indexed starting at 1 for the lowest dimension. A value of "0" in the distgridToArrayMap array indicates that the respective DistGrid dimension is not mapped against any Array dimension. What this means is that the Array will be replicated along this DistGrid dimension.
As a first example consider the case where a 1D Array
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=1, rc=rc)
is created on the 2D DistGrid used during the previous section.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
Here the default DistGrid to Array dimension mapping is used which assigns the Array dimensions in sequence to the DistGrid dimensions starting with dimension "1". Extra DistGrid dimensions are considered replicator dimensions because the Array will be replicated along those dimensions. In the above example the 2nd DistGrid dimension will cause 1D Array pieces to be replicated along the DEs of the 2nd DistGrid dimension. Replication in the context of ESMF_ArrayCreate() does not mean that data values are communicated and replicated between different DEs, but it means that different DEs provide memory allocations for identical exclusive elements.
Access to the data storage of an Array that has been replicated along DistGrid dimensions is the same as for Arrays without replication.
call ESMF_ArrayGet(array, localDeCount=localDeCount, rc=rc)
allocate(larrayList(0:localDeCount-1))
allocate(localDeToDeMap(0:localDeCount-1))
call ESMF_ArrayGet(array, localarrayList=larrayList, &
localDeToDeMap=localDeToDeMap, rc=rc)
The array object was created without additional padding which means that the bounds of the Fortran array pointer correspond to the bounds of the exclusive region. The following loop will cycle through all local DEs, print the DE number as well as the Fortran array pointer bounds. The bounds should be:
lbound ubound
DE 0: 1 3 --+
DE 2: 1 3 --| 1st replication set
DE 4: 1 3 --+
DE 1: 1 2 --+
DE 3: 1 2 --| 2nd replication set
DE 5: 1 2 --+
do localDe=0, localDeCount-1
call ESMF_LocalArrayGet(larrayList(localDe), myFarray1D, &
datacopyflag=ESMF_DATACOPY_REFERENCE, rc=rc)
print *, "localPet: ", localPet, "DE ",localDeToDeMap(localDe)," [", &
lbound(myFarray1D), ubound(myFarray1D),"]"
enddo
deallocate(larrayList)
deallocate(localDeToDeMap)
call ESMF_ArrayDestroy(array, rc=rc)
The Fortran array pointer in the above loop was of rank 1 because the Array object was of rank 1. However, the distgrid object associated with array is 2-dimensional! Consequently DistGrid based information queried from array will be 2D. The distgridToArrayMap and arrayToDistGridMap arrays provide the necessary mapping to correctly associate DistGrid based information with Array dimensions.
The next example creates a 2D Array
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
on the previously used 2D DistGrid. By default, i.e. without the distgridToArrayMap argument, both DistGrid dimensions would be associated with the two Array dimensions. However, the distgridToArrayMap specified in the following call will only associate the second DistGrid dimension with the first Array dimension. This will render the first DistGrid dimension a replicator dimension and the second Array dimension a tensor dimension for which 1D undistLBound and undistUBound arguments must be supplied.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/0,1/), undistLBound=(/11/), &
undistUBound=(/14/), rc=rc)
call ESMF_ArrayDestroy(array, rc=rc)
Finally, the same arrayspec and distgrid arguments are used to create a 2D Array that is fully replicated in both dimensions of the DistGrid. Both Array dimensions are now tensor dimensions and both DistGrid dimensions are replicator dimensions.
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/0,0/), undistLBound=(/11,21/), &
undistUBound=(/14,22/), rc=rc)
The result will be an Array with local lower bound (/11,21/) and upper bound (/14,22/) on all 6 DEs of the DistGrid.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
Replicated Arrays can also be created from existing local Fortran arrays. The following Fortran array allocation will provide a 3 x 10 array on each PET.
allocate(myFarray2D(3,10))
Assuming a petCount of 4 the following DistGrid defines a 2D index space that is distributed across the PETs along the first dimension.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
The following call creates an Array object on the above distgrid using the locally existing myFarray2D Fortran arrays. The difference compared to the case with automatic memory allocation is that instead of arrayspec the Fortran array is provided as argument. Furthermore, the undistLBound and undistUBound arguments can be omitted, defaulting into Array tensor dimension lower bound of 1 and an upper bound equal to the size of the respective Fortran array dimension.
array = ESMF_ArrayCreate(farray=myFarray2D, distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, distgridToArrayMap=(/0,2/), rc=rc)
The array object associates the 2nd DistGrid dimension with the 2nd Array dimension. The first DistGrid dimension is not associated with any Array dimension and will lead to replication of the Array along the DEs of this direction.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
It is a common situation, particularly in legacy code, that an ESMF Array object must be filled with data originating from a large Fortran array stored on a single PET.
if (localPet == 0) then
allocate(farray(10,20,30))
do k=1, 30
do j=1, 20
do i=1, 10
farray(i, j, k) = k*1000 + j*100 + i
enddo
enddo
enddo
else
allocate(farray(0,0,0))
endif
distgrid = ESMF_DistGridCreate(minIndex=(/1,1,1/), maxIndex=(/10,20,30/), &
rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=3, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
The ESMF_ArrayScatter() method provides a convenient way of scattering array data from a single root PET across the DEs of an ESMF Array object.
call ESMF_ArrayScatter(array, farray=farray, rootPet=0, rc=rc)
deallocate(farray)
The destination of the ArrayScatter() operation are all the DEs of a single tile. For multi-tile Arrays the destination tile can be specified. The shape of the scattered Fortran array must match the shape of the destination tile in the ESMF Array.
Gathering data decomposed and distributed across the DEs of an ESMF Array object into a single Fortran array on root PET is accomplished by calling ESMF_ArrayGather().
if (localPet == 3) then
allocate(farray(10,20,30))
else
allocate(farray(0,0,0))
endif
call ESMF_ArrayGather(array, farray=farray, rootPet=3, rc=rc)
deallocate(farray)
The source of the ArrayGather() operation are all the DEs of a single tile. For multi-tile Arrays the source tile can be specified. The shape of the gathered Fortran array must match the shape of the source tile in the ESMF Array.
The ESMF_ArrayScatter() operation allows to fill entire replicated Array objects with data coming from a single root PET.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/5,5/), &
regDecomp=(/2,3/), rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
distgridToArrayMap=(/0,0/), undistLBound=(/11,21/), &
undistUBound=(/14,22/), rc=rc)
The shape of the Fortran source array used in the Scatter() call must be that of the contracted Array, i.e. contracted DistGrid dimensions do not count. For the array just created this means that the source array on rootPet must be of shape 4 x 2.
if (localPet == 0) then
allocate(myFarray2D(4,2))
do j=1,2
do i=1,4
myFarray2D(i,j) = i * 100.d0 + j * 1.2345d0 ! initialize
enddo
enddo
else
allocate(myFarray2D(0,0))
endif
call ESMF_ArrayScatter(array, farray=myFarray2D, rootPet=0, rc=rc)
deallocate(myFarray2D)
This will have filled each local 4 x 2 Array piece with the replicated data of myFarray2D.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
As a second example for the use of Scatter() and Gather() consider the following replicated Array created from existing local Fortran arrays.
allocate(myFarray2D(3,10)) distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/40,10/), rc=rc)
array = ESMF_ArrayCreate(farray=myFarray2D, distgrid=distgrid, &
indexflag=ESMF_INDEX_DELOCAL, distgridToArrayMap=(/0,2/), rc=rc)
The array object associates the 2nd DistGrid dimension with the 2nd Array dimension. The first DistGrid dimension is not associated with any Array dimension and will lead to replication of the Array along the DEs of this direction. Still, the local arrays that comprise the array object refer to independent pieces of memory and can be initialized independently.
myFarray2D = localPet ! initialize
However, the notion of replication becomes visible when an array of shape 3 x 10 on root PET 0 is scattered across the Array object.
if (localPet == 0) then
allocate(myFarray2D2(5:7,11:20))
do j=11,20
do i=5,7
myFarray2D2(i,j) = i * 100.d0 + j * 1.2345d0 ! initialize
enddo
enddo
else
allocate(myFarray2D2(0,0))
endif
call ESMF_ArrayScatter(array, farray=myFarray2D2, rootPet=0, rc=rc)
deallocate(myFarray2D2)
The Array pieces on every DE will receive the same source data, resulting in a replication of data along DistGrid dimension 1.
When the inverse operation, i.e. ESMF_ArrayGather(), is applied to a replicated Array an intrinsic ambiguity needs to be considered. ESMF defines the gathering of data of a replicated Array as the collection of data originating from the numerically higher DEs. This means that data in replicated elements associated with numerically lower DEs will be ignored during ESMF_ArrayGather(). For the current example this means that changing the Array contents on PET 1, which here corresponds to DE 1,
if (localPet == 1) then
myFarray2D = real(1.2345, ESMF_KIND_R8)
endif
will not affect the result of
allocate(myFarray2D2(3,10)) myFarray2D2 = 0.d0 ! initialize to a known value call ESMF_ArrayGather(array, farray=myFarray2D2, rootPet=0, rc=rc)
The result remains completely defined by the unmodified values of Array in DE 3, the numerically highest DE. However, overriding the DE-local Array piece on DE 3
if (localPet==3) then
myFarray2D = real(5.4321, ESMF_KIND_R8)
endif
will change the outcome of
call ESMF_ArrayGather(array, farray=myFarray2D2, rootPet=0, rc=rc)
as expected.
deallocate(myFarray2D2) call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
One of the most fundamental communication pattern in domain decomposition codes is the halo operation. The ESMF Array class supports halos by allowing memory for extra elements to be allocated on each DE. See sections 28.2.2 and 28.2.8 for examples and details on how to create an Array with extra DE-local elements.
Here we consider an Array object that is created on a DistGrid that defines a 10 x 20 index space, decomposed into 4 DEs using a regular 2 x 2 decomposition.
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), rc=rc)
The Array holds 2D double precision float data.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
The totalLWidth and totalUWidth arguments are used during Array creation to allocate 2 extra elements along every direction outside the exclusive region defined by the DistGrid for every DE. (The indexflag set to ESMF_INDEX_GLOBAL in this example does not affect the halo behavior of Array. The setting is simply more convenient for the following code.)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
totalLWidth=(/2,2/), totalUWidth=(/2,2/), indexflag=ESMF_INDEX_GLOBAL, &
rc=rc)
Without the explicit definition of boundary conditions in the DistGrid the following inner connections are defined.
+-------------------+ +-------------------+
| \ 2 / | | \ 2 / |
| +-------------+ | | +-------------+ |
| | DE 0 | | | | DE 2 | |
| | | | | | | |
|2 | 5 x 10 | 2| <-> |2 | 5 x 10 | 2|
| | | | | | | |
| | | | | | | |
| +-------------+ | | +-------------+ |
| / 2 \ | | / 2 \ |
+-------------------+ +-------------------+
^ \/ ^
| /\ |
v v
+-------------------+ +-------------------+
| \ 2 / | | \ 2 / |
| +-------------+ | | +-------------+ |
| | DE 1 | | | | DE 3 | |
| | | | | | | |
|2 | 5 x 10 | 2| <-> |2 | 5 x 10 | 2|
| | | | | | | |
| | | | | | | |
| +-------------+ | | +-------------+ |
| / 2 \ | | / 2 \ |
+-------------------+ +-------------------+
The exclusive region on each DE is of shape 5 x 10, while the total region
on each DE is of shape (5+2+2) x (10+2+2) = 9 x 14. In a typical application
the elements in the exclusive region are updated exclusively by the PET that
owns the DE. In this example the exclusive elements on every DE are
initialized to the value 
of the geometric function
 |
(1) |
 |
(2) |
 |
(3) |
and 
.
a = 2. * 3.14159 / 10. b = 2. * 3.14159 / 20. call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
call ESMF_ArrayGet(array, exclusiveLBound=eLB, exclusiveUBound=eUB, rc=rc)
do j=eLB(2,1), eUB(2,1)
do i=eLB(1,1), eUB(1,1)
farrayPtr(i,j) = sin(a*i) * cos(b*j) ! test function
enddo
enddo
The above loop only initializes the exclusive elements on each DE. The extra elements, outside the exclusive region, are left untouched, holding undefined values. Elements outside the exclusive region that correspond to exclusive elements in neighboring DEs can be filled with the data values in those neighboring elements. This is the definition of the halo operation.
In ESMF the halo communication pattern is first precomputed and stored in a RouteHandle object. This RouteHandle can then be used repeatedly to perform the same halo operation in the most efficient way.
The default halo operation for an Array is precomputed by the following call.
call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle, rc=rc)
The haloHandle now holds the default halo operation for array, which matches as many elements as possible outside the exclusive region to their corresponding halo source elements in neighboring DEs. Elements that could not be matched, e.g. at the edge of the global domain with open boundary conditions, will not be updated by the halo operation.
The haloHandle is applied through the ESMF_ArrayHalo() method.
call ESMF_ArrayHalo(array=array, routehandle=haloHandle, rc=rc)
Finally the resources held by haloHandle need to be released.
call ESMF_ArrayHaloRelease(routehandle=haloHandle, rc=rc)
The array object created above defines a 2 element wide rim around the exclusive region on each DE. Consequently the default halo operation used above will have resulted in updating both elements along the inside edges. For simple numerical kernels often a single halo element is sufficient. One way to achieve this would be to reduce the size of the rim surrounding the exclusive region to 1 element along each direction. However, if the same Array object is also used for higher order kernels during a different phase of the calculation, a larger element rim is required. For this case ESMF_ArrayHaloStore() offers two optional arguments haloLDepth and haloUDepth. Using these arguments a reduced halo depth can be specified.
call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle, &
haloLDepth=(/1,1/), haloUDepth=(/1,1/), rc=rc)
This halo operation with a depth of 1 is sufficient to support a simple quadratic differentiation kernel.
allocate(farrayTemp(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)))
do step=1, 4
call ESMF_ArrayHalo(array=array, routehandle=haloHandle, rc=rc)
do j=eLB(2,1), eUB(2,1)
do i=eLB(1,1), eUB(1,1)
if (i==1) then
! global edge
farrayTemp(i,j) = 0.5 * (-farrayPtr(i+2,j) + 4.*farrayPtr(i+1,j) &
- 3.*farrayPtr(i,j)) / a
else if (i==10) then
! global edge
farrayTemp(i,j) = 0.5 * (farrayPtr(i-2,j) - 4.*farrayPtr(i-1,j) &
+ 3.*farrayPtr(i,j)) / a
else
farrayTemp(i,j) = 0.5 * (farrayPtr(i+1,j) - farrayPtr(i-1,j)) / a
endif
enddo
enddo
farrayPtr(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)) = farrayTemp
enddo
deallocate(farrayTemp)
call ESMF_ArrayHaloRelease(routehandle=haloHandle, rc=rc)
The special treatment of the global edges in the above kernel is due to the fact that the underlying DistGrid object does not define any special boundary conditions. By default open global boundaries are assumed which means that the rim elements on the global edges are untouched during the halo operation, and cannot be used in the symmetric numerical derivative formula. The kernel can be simplified (and the calculation is more precise) with periodic boundary conditions along the first Array dimension.
First destroy the current Array and DistGrid objects.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
Create a DistGrid with periodic boundary condition along the first dimension.
allocate(connectionList(1)) ! one connection
call ESMF_DistGridConnectionSet(connection=connectionList(1), &
tileIndexA=1, tileIndexB=1, positionVector=(/10, 0/), rc=rc)
distgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/2,2/), connectionList=connectionList, rc=rc)
deallocate(connectionList)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, &
totalLWidth=(/2,2/), totalUWidth=(/2,2/), indexflag=ESMF_INDEX_GLOBAL, &
rc=rc)
Initialize the exclusive elements to the same geometric function as before.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr, rc=rc)
call ESMF_ArrayGet(array, exclusiveLBound=eLB, exclusiveUBound=eUB, rc=rc)
do j=eLB(2,1), eUB(2,1)
do i=eLB(1,1), eUB(1,1)
farrayPtr(i,j) = sin(a*i) * cos(b*j) ! test function
enddo
enddo
The numerical kernel only operates along the first dimension. An asymmetric halo depth can be used to take this fact into account.
call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle, &
haloLDepth=(/1,0/), haloUDepth=(/1,0/), rc=rc)
Now the same numerical kernel can be used without special treatment of global edge elements. The symmetric derivative formula can be used for all exclusive elements.
allocate(farrayTemp(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)))
do step=1, 4
call ESMF_ArrayHalo(array=array, routehandle=haloHandle, rc=rc)
do j=eLB(2,1), eUB(2,1)
do i=eLB(1,1), eUB(1,1)
farrayTemp(i,j) = 0.5 * (farrayPtr(i+1,j) - farrayPtr(i-1,j)) / a
enddo
enddo
farrayPtr(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)) = farrayTemp
enddo
The precision of the above kernel can be improved by going to a higher order interpolation. Doing so requires that the halo depth must be increased. The following code resets the exclusive Array elements to the test function, precomputes a RouteHandle for a halo operation with depth 2 along the first dimension, and finally uses the deeper halo in the higher order kernel.
do j=eLB(2,1), eUB(2,1)
do i=eLB(1,1), eUB(1,1)
farrayPtr(i,j) = sin(a*i) * cos(b*j) ! test function
enddo
enddo
call ESMF_ArrayHaloStore(array=array, routehandle=haloHandle2, &
haloLDepth=(/2,0/), haloUDepth=(/2,0/), rc=rc)
do step=1, 4
call ESMF_ArrayHalo(array=array, routehandle=haloHandle2, rc=rc)
do j=eLB(2,1), eUB(2,1)
do i=eLB(1,1), eUB(1,1)
farrayTemp(i,j) = (-farrayPtr(i+2,j) + 8.*farrayPtr(i+1,j) &
- 8.*farrayPtr(i-1,j) + farrayPtr(i-2,j)) / (12.*a)
enddo
enddo
farrayPtr(eLB(1,1):eUB(1,1), eLB(2,1):eUB(2,1)) = farrayTemp
enddo
deallocate(farrayTemp)
ESMF supports having multiple halo operations defined on the same Array object at the same time. Each operation can be accessed through its unique RouteHandle. The above kernel could have made ESMF_ArrayHalo() calls with a depth of 1 along the first dimension using the previously precomputed haloHandle if it needed to. Both RouteHandles need to release their resources when no longer used.
call ESMF_ArrayHaloRelease(routehandle=haloHandle, rc=rc)
call ESMF_ArrayHaloRelease(routehandle=haloHandle2, rc=rc)
Finally the Array and DistGrid objects can be destroyed.
call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
In the previous section the Array halo operation was demonstrated for regularly decomposed ESMF Arrays. However, the ESMF halo operation is not restricted to regular decompositions. The same Array halo methods apply unchanged to Arrays that are created on arbitrarily distributed DistGrids. This includes the non-blocking features discussed in section 28.2.19.
All of the examples in this section are based on the same arbitrarily distributed DistGrid. Section 35.3.6 discusses DistGrids with user-supplied, arbitrary sequence indices in detail. Here a global index space range from 1 through 20 is decomposed across 4 DEs. There are 4 PETs in this example with 1 DE per PET. Each PET constructs its local seqIndexList variable.
do i=1, 5
seqIndexList(i) = localPet + (i - 1) * petCount + 1
enddo
This results in the following cyclic distribution scheme:
DE 0 on PET 0: seqIndexList = (/1, 5, 9, 13, 17/) DE 1 on PET 1: seqIndexList = (/2, 6, 10, 14, 18/) DE 2 on PET 2: seqIndexList = (/3, 7, 11, 15, 19/) DE 3 on PET 3: seqIndexList = (/4, 8, 12, 16, 20/)
The local seqIndexList variables are then used to create a DistGrid with the indicated arbitrary distribution pattern.
distgrid = ESMF_DistGridCreate(arbSeqIndexList=seqIndexList, rc=rc)
The resulting DistGrid is one-dimensional, although the user code may interpret the sequence indices as a 1D map into a problem of higher dimensionality.
In this example the local DE on each PET is associated with a 5 element exclusive region. Providing seqIndexList of different size on the different PETs is supported and would result in different number of exclusive elements on each PET.
Creating an ESMF Array on top of a DistGrid with arbitrary sequence indices is in principle no different from creating an Array on a regular DistGrid. However, while an Array that was created on a regular DistGrid automatically inherits the index space topology information that is contained within the DistGrid object, there is no such topology information available for DistGrid objects with arbitrary sequence indices. As a consequence of this, Arrays created on arbitrary DistGrids do not automatically have the information that is required to associated halo elements with the exclusive elements across DEs. Instead the user must supply this information explicitly during Array creation.
Multiple ArrayCreate() interfaces exist that allow the creation of an Array on a DistGrid with arbitrary sequence indices. The sequence indices for the halo region of the local DE are supplied through an additional argument with dummy name haloSeqIndexList. As in the regular case, the ArrayCreate() interfaces differ in the way that the memory allocations for the Array elements are passed into the call. The following code shows how an ESMF Array can be wrapped around existing PET-local memory allocations. The allocations are of different size on each PET as to accommodate the correct number of local Array elements (exclusive region + halo region).
allocate(farrayPtr1d(5+localPet+1)) !use explicit Fortran allocate statement
if (localPet==0) then
array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
haloSeqIndexList=(/6/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==1) then
array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
haloSeqIndexList=(/1,19/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==2) then
array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
haloSeqIndexList=(/16,6,9/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==3) then
array = ESMF_ArrayCreate(distgrid, farrayPtr1d, &
haloSeqIndexList=(/1,3,1,4/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
The haloSeqIndexList arguments are 1D arrays of sequence indices. It is through this argument that the user associates the halo elements with exclusive Array elements covered by the DistGrid. In this example there are different number of halo elements on each DE. They are associated with exclusive elements as follows:
halo on DE 0 on PET 0: <seqIndex=6> 2nd exclusive element on DE 1
halo on DE 1 on PET 1: <seqIndex=1> 1st exclusive element on DE 0
<seqIndex=19> 5th exclusive element on DE 2
halo on DE 2 on PET 2: <seqIndex=16> 4th exclusive element on DE 3
<seqIndex=6> 2nd exclusive element on DE 1
<seqIndex=9> 3rd exclusive element on DE 0
halo on DE 3 on PET 3: <seqIndex=1> 1st exclusive element on DE 0
<seqIndex=3> 1st exclusive element on DE 2
<seqIndex=1> 1st exclusive element on DE 0
<seqIndex=4> 1st exclusive element on DE 3
The above haloSeqIndexList arguments were constructed very artificially in order to show the following general features:
The ArrayCreate() call checks that the provided Fortran memory allocation is correctly sized to hold the exclusive elements, as indicated by the DistGrid object, plus the halo elements as indicated by the local haloSeqIndexList argument. The size of the Fortran allocation must match exactly or a runtime error will be returned.
Analogous to the case of Arrays on regular DistGrids, it is the exclusive region of the local DE that is typically modified by the code running on each PET. All of the ArrayCreate() calls that accept the haloSeqIndexList argument place the exclusive region at the beginning of the memory allocation on each DE and use the remaining space for the halo elements. The following loop demonstrates this by filling the exclusive elements on each DE with initial values. Remember that in this example each DE holds 5 exclusive elements associated with different arbitrary sequence indices.
do i=1, 5
farrayPtr1d(i) = seqIndexList(i) / 10.
enddo
Now the exclusive elements of array are initialized on each DE, however, the halo elements remain unchanged. A RouteHandle can be set up that encodes the required communication pattern for a halo exchange. The halo exchange is precomputed according to the arbitrary sequence indices specified for the exclusive elements by the DistGrid and the sequence indices provided by the user for each halo element on the local DE in form of the haloSeqIndexList argument during ArrayCreate().
call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)
Executing this halo operation will update the local halo elements according to the associated sequence indices.
call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)
As always it is good practice to release the RouteHandle when done with it.
call ESMF_ArrayHaloRelease(haloHandle, rc=rc)
Also the Array object should be destroyed when no longer needed.
call ESMF_ArrayDestroy(array, rc=rc)
Further, since the memory allocation was done explicitly using the Fortran allocate() statement, it is necessary to explicitly deallocate in order to prevent memory leaks in the user application.
deallocate(farrayPtr1d)
Alternatively the exact same Array can be created where ESMF does the memory allocation and deallocation. In this case the typekind of the Array must be specified explicitly.
if (localPet==0) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/6/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==1) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/1,19/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==2) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/16,6,9/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==3) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/1,3,1,4/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
Use ArrayGet() to gain access to the local memory allocation.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr1d, rc=rc)
The returned Fortran pointer can now be used to initialize the exclusive elements on each DE as in the previous case.
do i=1, 5
farrayPtr1d(i) = seqIndexList(i) / 10.
enddo
Identical halo operations are constructed and used.
call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)
call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)
call ESMF_ArrayHaloRelease(haloHandle, rc=rc)
call ESMF_ArrayDestroy(array, rc=rc)
A current limitation of the Array implementation restricts DistGrids that contain user-specified, arbitrary sequence indices to be exactly 1D when used to create Arrays. See section 28.3 for a list of current implementation restrictions. However, an Array created on such a 1D arbitrary DistGrid is allowed to have undistributed dimensions. The following example creates an Array on the same arbitrary DistGrid, with the same arbitrary sequence indices for the halo elements as before, but with one undistributed dimension with a size of 3.
if (localPet==0) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/6/), undistLBound=(/1/), undistUBound=(/3/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==1) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/1,19/), undistLBound=(/1/), undistUBound=(/3/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==2) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/16,6,9/), undistLBound=(/1/), undistUBound=(/3/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==3) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
haloSeqIndexList=(/1,3,1,4/), undistLBound=(/1/), undistUBound=(/3/), &
rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
By default the DistGrid dimension is mapped to the first Array dimension, associating the remaining Array dimensions with the undistributed dimensions in sequence. The dimension order is important when accessing the individual Array elements. Here the same initialization as before is extended to cover the undistributed dimension.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr2d, rc=rc)
do j=1, 3
do i=1, 5
farrayPtr2d(i,j) = seqIndexList(i) / 10. + 100.*j
enddo
enddo
In the context of the Array halo operation additional undistributed dimensions are treated in a simple factorized manner. The same halo association between elements that is encoded in the 1D arbitrary sequence index scheme is applied to each undistributed element separately. This is completely transparent on the user level and the same halo methods are used as before.
call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)
call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)
call ESMF_ArrayHaloRelease(haloHandle, rc=rc)
call ESMF_ArrayDestroy(array, rc=rc)
In some situations it is more convenient to associate some or all of the undistributed dimensions with the first Array dimensions. This can be done easily by explicitly mapping the DistGrid dimension to an Array dimension other than the first one. The following code creates essentially the same Array as before, but with swapped dimension order.
if (localPet==0) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1/), &
undistLBound=(/1/), undistUBound=(/3/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==1) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1,2/), &
undistLBound=(/1/), undistUBound=(/3/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==2) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1,2,3/), &
undistLBound=(/1/), undistUBound=(/3/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==3) then
array = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1,2,3,4/), &
undistLBound=(/1/), undistUBound=(/3/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
The swapped dimension order results in a swapping of i and j when accessing Array elements in the loop.
call ESMF_ArrayGet(array, farrayPtr=farrayPtr2d, rc=rc)
do j=1, 3
do i=1, 5
farrayPtr2d(j,i) = seqIndexList(i) / 10. + 100.*j
enddo
enddo
Again there is no difference in how the the halo operations are applied.
call ESMF_ArrayHaloStore(array, routehandle=haloHandle, rc=rc)
call ESMF_ArrayHalo(array, routehandle=haloHandle, rc=rc)
call ESMF_ArrayDestroy(array, rc=rc)
One of the benefits of mapping the undistributed dimension(s) to the "left side" of the Array dimensions is that Arrays that only differ in the size of the undistributed dimension(s) are weakly congruent in this arrangement. Weakly congruent Arrays can reuse the same RouteHandle, saving the overhead that is caused by the precompute step. In order to demonstrate this the RouteHandle of the previous halo call was not yet released and will be applied to a weakly congruent Array.
The following code creates an Array that is weakly congruent to the the previous Array by using the same input information as before, only that the size of the undistributed dimension is now 6 instead of 3.
if (localPet==0) then
array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1/), &
undistLBound=(/1/), undistUBound=(/6/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==1) then
array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1,2/), &
undistLBound=(/1/), undistUBound=(/6/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==2) then
array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1,2,3/), &
undistLBound=(/1/), undistUBound=(/6/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
if (localPet==3) then
array2 = ESMF_ArrayCreate(distgrid=distgrid, typekind=ESMF_TYPEKIND_R8, &
distgridToArrayMap=(/2/), haloSeqIndexList=(/1,2,3,4/), &
undistLBound=(/1/), undistUBound=(/6/), rc=rc)
if (rc /= ESMF_SUCCESS) call ESMF_Finalize(endflag=ESMF_END_ABORT)
endif
Again the exclusive Array elements must be initialized.
call ESMF_ArrayGet(array2, farrayPtr=farrayPtr2d, rc=rc)
do j=1, 6
do i=1, 5
farrayPtr2d(j,i) = seqIndexList(i) / 10. + 100.*j
enddo
enddo
Now the haloHandle that was previously pre-computed for array can be used directly for the weakly congruent array2.
call ESMF_ArrayHalo(array2, routehandle=haloHandle, rc=rc)
Release the RouteHandle after its last use and clean up the remaining Array and DistGrid objects.
call ESMF_ArrayHaloRelease(haloHandle, rc=rc)
call ESMF_ArrayDestroy(array2, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
Arrays used in different models often cover the same index space region, however, the distribution of the Arrays may be different, e.g. the models run on exclusive sets of PETs. Even if the Arrays are defined on the same list of PETs the decomposition may be different.
srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/4,1/), rc=rc)
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/1,4/), rc=rc)
The number of elements covered by srcDistgrid is identical to the number of elements covered by dstDistgrid - in fact the index space regions covered by both DistGrid objects are congruent.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)
By construction srcArray and dstArray are of identical type and kind. Further the number of exclusive elements matches between both Arrays. These are the prerequisites for the application of an Array redistribution in default mode. In order to increase performance of the actual redistribution the communication pattern is precomputed and stored in an ESMF_RouteHandle object.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The redistHandle can now be used repeatedly on the srcArray, dstArray pair to redistributed data from source to destination Array.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The use of the precomputed redistHandle is not restricted to the (srcArray, dstArray) pair. Instead the redistHandle can be used to redistribute data between any two Arrays that are weakly congruent to the Array pair used during precomputation. Arrays are congruent if they are defined on DistGrids that match with ESMF_DISTGRIDMATCH_EXACT or higher, and have matching DE-local array allocations (i.e. the same amount of padding around the exclusive region as well as the same shape in the undistributed dimensions). For Arrays to be weakly congruent the shape and size of the undistributed dimensions that have a smaller stride than the first distributed dimension need not be the same. This definition covers even the case where an Array does not have any undistributed dimensions.
For instance, an Array where the first two dimensions are undistributed, and are of size 5 and 6 (i.e. undistributed shape (5,6)), is weakly congruent to an Array that only has a single undistributed first dimension. Furthermore, the size of this single undistributed dimension is not restricted by the number of undistributed elements in the first Array (i.e. here it does not have to be 5x6=30). The Array is also weakly congruent to an Array that does not have a first undistributed dimension at all. In either case the only restriction is that the distributed dimensions must be congruent.
The transferability of RouteHandles between Array pairs that are weakly congruent can greatly reduce the number of communication store calls needed. In a typical application Arrays are often defined on the same decomposition, typically leading to congruent distributed dimensions. However, these Arrays do not always have the same shape or size in the undistributed dimensions.
For the current case, the redistHandle was precomputed for simple 2D Arrays without undistributed dimensions. The RouteHandle transferability rule allows us to use this same RouteHandle to redistribute between two 3D Array that are built on the same 2D DistGrid, but have an undistributed dimension that has a smaller stride than the distributed dimensions.
call ESMF_ArraySpecSet(arrayspec3d, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
srcArray1 = ESMF_ArrayCreate(arrayspec=arrayspec3d, distgrid=srcDistgrid, &
distgridToArrayMap=(/2,3/), undistLBound=(/1/), undistUBound=(/10/), rc=rc)
dstArray1 = ESMF_ArrayCreate(arrayspec=arrayspec3d, distgrid=dstDistgrid, &
distgridToArrayMap=(/2,3/), undistLBound=(/1/), undistUBound=(/10/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray1, dstArray=dstArray1, &
routehandle=redistHandle, rc=rc)
The following variation of the code shows that the same RouteHandle can be applied to an Array pair where the number of undistributed dimensions does not match between source and destination Array. The two requirements are simply that each side is weakly congruent to the Array used during store, and that the total number of undistributed elements on source and destination side is the same. Here we prepare a source Array with two leading undistributed dimensions that multiply out to 2x5=10 undistributed elements. The destination array is the same as before with only a single leading undistributed dimension of size 10.
call ESMF_ArraySpecSet(arrayspec4d, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)
srcArray2 = ESMF_ArrayCreate(arrayspec=arrayspec4d, distgrid=srcDistgrid, &
distgridToArrayMap=(/3,4/), undistLBound=(/1,1/), undistUBound=(/2,5/), &
rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray2, dstArray=dstArray1, &
routehandle=redistHandle, rc=rc)
When done, the resources held by redistHandle need to be deallocated by the user code before the RouteHandle becomes inaccessible.
call ESMF_ArrayRedistRelease(routehandle=redistHandle, rc=rc)
In default mode, i.e. without providing the optional srcToDstTransposeMap argument, ESMF_ArrayRedistStore() does not require equal number of dimensions in source and destination Array. Only the total number of elements must match. Specifying srcToDstTransposeMap switches ESMF_ArrayRedistStore() into transpose mode. In this mode each dimension of srcArray is uniquely associated with a dimension in dstArray, and the sizes of associated dimensions must match for each pair.
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/20,10/), &
rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)
This dstArray object covers a 20 x 10 index space while the srcArray, defined further up, covers a 10 x 20 index space. Setting srcToDstTransposeMap = (/2,1/) will associate the first and second dimension of srcArray with the second and first dimension of dstArray, respectively. This corresponds to a transpose of dimensions. Since the decomposition and distribution of dimensions may be different for source and destination redistribution may occur at the same time.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/2,1/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The transpose mode of ESMF_ArrayRedist() is not limited to distributed dimensions of Arrays. The srcToDstTransposeMap argument can be used to transpose undistributed dimensions in the same manner. Furthermore transposing distributed and undistributed dimensions between Arrays is also supported.
The srcArray used in the following examples is of rank 4 with 2 distributed and 2 undistributed dimensions. The distributed dimensions are the two first dimensions of the Array and are distributed according to the srcDistgrid which describes a total index space region of 100 x 200 elements. The last two Array dimensions are undistributed dimensions of size 2 and 3, respectively.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)
srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/100,200/), &
rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
undistLBound=(/1,1/), undistUBound=(/2,3/), rc=rc)
The first dstArray to consider is defined on a DistGrid that also describes a 100 x 200 index space region. The distribution indicated by dstDistgrid may be different from the source distribution. Again the first two Array dimensions are associated with the DistGrid dimensions in sequence. Furthermore, the last two Array dimensions are undistributed dimensions, however, the sizes are 3 and 2, respectively.
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/100,200/), &
rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
undistLBound=(/1,1/), undistUBound=(/3,2/), rc=rc)
The desired mapping between srcArray and dstArray dimensions is expressed by srcToDstTransposeMap = (/1,2,4,3/), transposing only the two undistributed dimensions.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/1,2,4,3/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Next consider a dstArray that is defined on the same dstDistgrid, but with a different order of Array dimensions. The desired order is specified during Array creation using the argument distgridToArrayMap = (/2,3/). This map associates the first and second DistGrid dimensions with the second and third Array dimensions, respectively, leaving Array dimensions one and four undistributed.
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2,3/), undistLBound=(/1,1/), undistUBound=(/3,2/), &
rc=rc)
Again the sizes of the undistributed dimensions are chosen in reverse order compared to srcArray. The desired transpose mapping in this case will be srcToDstTransposeMap = (/2,3,4,1/).
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/2,3,4,1/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Finally consider the case where dstArray is constructed on a 200 x 3 index space and where the undistributed dimensions are of size 100 and 2.
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/200,3/), &
rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
undistLBound=(/1,1/), undistUBound=(/100,2/), rc=rc)
By construction srcArray and dstArray hold the same number of elements, albeit in a very different layout. Nevertheless, with a srcToDstTransposeMap that maps matching dimensions from source to destination an Array redistribution becomes a well defined operation between srcArray and dstArray.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, srcToDstTransposeMap=(/3,1,4,2/), rc=rc)
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The default mode of Array redistribution, i.e. without providing a srcToDstTransposeMap to ESMF_ArrayRedistStore(), also supports undistributed Array dimensions. The requirement in this case is that the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, be the same for source and destination Array. In this mode the number of undistributed dimensions need not match between source and destination.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=4, rc=rc)
srcDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/4,1/), rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
undistLBound=(/1,1/), undistUBound=(/2,4/), rc=rc)
dstDistgrid = ESMF_DistGridCreate(minIndex=(/1,1/), maxIndex=(/10,20/), &
regDecomp=(/1,4/), rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2,3/), undistLBound=(/1,1/), undistUBound=(/2,4/), &
rc=rc)
Both srcArray and dstArray have two undistributed dimensions and
a total count of undistributed elements of
.
The Array redistribution operation is defined in terms of sequentialized undistributed dimensions. In the above case this means that a unique sequence index will be assigned to each of the 8 undistributed elements. The sequence indices will be 1, 2, ..., 8, where sequence index 1 is assigned to the first element in the first (i.e. fastest varying in memory) undistributed dimension. The following undistributed elements are labeled in consecutive order as they are stored in memory.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
The redistribution operation by default applies the identity operation between the elements of undistributed dimensions. This means that source element with sequence index 1 will be mapped against destination element with sequence index 1 and so forth. Because of the way source and destination Arrays in the current example were constructed this corresponds to a mapping of dimensions 3 and 4 on srcArray to dimensions 1 and 4 on dstArray, respectively.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Array redistribution does not require the same number of undistributed dimensions in source and destination Array, merely the total number of undistributed elements must match.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/1,3/), undistLBound=(/11/), undistUBound=(/18/), &
rc=rc)
This dstArray object only has a single undistributed dimension, while the srcArray, defined further back, has two undistributed dimensions. However, the total undistributed element count for both Arrays is 8.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
In this case the default identity operation between the elements of undistributed dimensions corresponds to a merging of dimensions 3 and 4 on srcArray into dimension 2 on dstArray.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=redistHandle, rc=rc)
Sparse matrix multiplication is a fundamental Array communication method. One frequently used application of this method is the interpolation between pairs of Arrays. The principle is this: the value of each element in the exclusive region of the destination Array is expressed as a linear combination of potentially all the exclusive elements of the source Array. Naturally most of the coefficients of these linear combinations will be zero and it is more efficient to store explicit information about the non-zero elements than to keep track of all the coefficients.
There is a choice to be made with respect to the format in which to store the information about the non-zero elements. One option is to store the value of each coefficient together with the corresponding destination element index and source element index. Destination and source indices could be expressed in terms of the corresponding DistGrid tile index together with the coordinate tuple within the tile. While this format may be the most natural way to express elements in the source and destination Array, it has two major drawbacks. First the coordinate tuple is dimCount specific and second the format is extremely bulky. For 2D source and destination Arrays it would require 6 integers to store the source and destination element information for each non-zero coefficient and matters get worse for higher dimensions.
Both problems can be circumvented by interpreting source and destination Arrays as sequentialized strings or vectors of elements. This is done by assigning a unique sequence index to each exclusive element in both Arrays. With that the operation of updating the elements in the destination Array as linear combinations of source Array elements takes the form of a sparse matrix multiplication.
The default sequence index rule assigns index 
to the minIndex corner
element of the first tile of the DistGrid on which the Array is defined. It then
increments the sequence index by for each element running through the
DistGrid dimensions by order. The index space position of the DistGrid tiles
does not affect the sequence labeling of elements. The default sequence indices
for
srcDistgrid = ESMF_DistGridCreate(minIndex=(/-1,0/), maxIndex=(/1,3/), rc=rc)
for each element are:
-------------------------------------> 2nd dim
|
| +------+------+------+------+
| |(-1,0)| | |(-1,3)|
| | | | | |
| | 1 | 4 | 7 | 10 |
| +------+------+------+------+
| | | | | |
| | | | | |
| | 2 | 5 | 8 | 11 |
| +------+------+------+------+
| | (1,0)| | | (1,3)|
| | | | | |
| | 3 | 6 | 9 | 12 |
| +------+------+------+------+
|
v
1st dim
The assigned sequence indices are decomposition and distribution invariant by construction. Furthermore, when an Array is created with extra elements per DE on a DistGrid the sequence indices (which only cover the exclusive elements) remain unchanged.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
rc=rc)
The extra padding of 1 element in each direction around the exclusive elements on each DE are "invisible" to the Array spare matrix multiplication method. These extra elements are either updated by the computational kernel or by Array halo operations.
An alternative way to assign sequence indices to all the elements in the tiles covered by a DistGrid object is to use a special ESMF_DistGridCreate() call. This call has been specifically designed for 1D cases with arbitrary, user-supplied sequence indices.
seqIndexList(1) = localPet*10 seqIndexList(2) = localPet*10 + 1 dstDistgrid = ESMF_DistGridCreate(arbSeqIndexList=seqIndexList, rc=rc)
This call to ESMF_DistGridCreate() is collective across the current VM. The arbSeqIndexList argument specifies the PET-local arbitrary sequence indices that need to be covered by the local DE. The resulting DistGrid has one local DE per PET which covers the entire PET-local index range. The user supplied sequence indices must be unique, but the sequence may be interrupted. The four DEs of dstDistgrid have the following local 1D index space coordinates (given between "()") and sequence indices:
covered by DE 0 covered by DE 1 covered by DE 2 covered by DE 3
on PET 0 on PET 1 on PET 2 on PET 3
----------------------------------------------------------------------
(1) : 0 (1) : 10 (1) : 20 (1) : 30
(2) : 1 (2) : 11 (2) : 21 (2) : 31
Again the DistGrid object provides the sequence index labeling for the exclusive elements of an Array created on the DistGrid regardless of extra, non-exclusive elements.
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, rc=rc)
With the definition of sequence indices, either by the default rule or as user provided arbitrary sequence indices, it is now possible to uniquely identify each exclusive element in the source and destination Array by a single integer number. Specifying a pair of source and destination elements takes two integer number regardless of the number of dimensions.
The information required to carry out a sparse matrix multiplication are the pair of source and destination sequence indices and the associated multiplication factor for each pair. ESMF requires this information in form of two Fortran arrays. The factors are stored in a 1D array of the appropriate type and kind, e.g. real(ESMF_KIND_R8)::factorList(:). Array sparse matrix multiplications are supported between Arrays of different type and kind. The type and kind of the factors can also be chosen freely. The sequence index pairs associated with the factors provided by factorList are stored in a 2D Fortran array of default integer kind of the shape integer::factorIndexList(2,:). The sequence indices of the source Array elements are stored in the first row of factorIndexList while the sequence indices of the destination Array elements are stored in the second row.
Each PET in the current VM must call into ESMF_ArraySMMStore() to precompute and store the communication pattern for the sparse matrix multiplication. The multiplication factors may be provided in parallel, i.e. multiple PETs may specify factorList and factorIndexList arguments when calling into ESMF_ArraySMMStore(). PETs that do not provide factors either call with factorList and factorIndexList arguments containing zero elements or issue the call omitting both arguments.
if (localPet == 0) then
allocate(factorList(1)) ! PET 0 specifies 1 factor
allocate(factorIndexList(2,1))
factorList = (/0.2/) ! factors
factorIndexList(1,:) = (/5/) ! seq indices into srcArray
factorIndexList(2,:) = (/30/) ! seq indices into dstArray
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else if (localPet == 1) then
allocate(factorList(3)) ! PET 1 specifies 3 factor
allocate(factorIndexList(2,3))
factorList = (/0.5, 0.5, 0.8/) ! factors
factorIndexList(1,:) = (/8, 2, 12/) ! seq indices into srcArray
factorIndexList(2,:) = (/11, 11, 30/) ! seq indices into dstArray
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else
! PETs 2 and 3 do not provide factors
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
endif
The RouteHandle object sparseMatMulHandle produced by ESMF_ArraySMMStore() can now be used to call ESMF_ArraySMM() collectively across all PETs of the current VM to perform
dstArray = 0.0
do n=1, size(combinedFactorList)
dstArray(combinedFactorIndexList(2, n)) +=
combinedFactorList(n) * srcArray(combinedFactorIndexList(1, n))
enddo
in parallel. Here combinedFactorList and combinedFactorIndexList
are the combined lists defined by the respective local lists provided by
PETs 0 and 1 in parallel. For this example
call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
will initialize the entire dstArray to 0.0 and then update two elements:
on DE 1: dstArray(2) = 0.5 * srcArray(0,0) + 0.5 * srcArray(0,2)
and
on DE 3: dstArray(1) = 0.2 * srcArray(0,1) + 0.8 * srcArray(1,3).
The call to ESMF_ArraySMM() does provide the option to turn the default dstArray initialization off. If argument zeroregion is set to ESMF_REGION_EMPTY
call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, zeroregion=ESMF_REGION_EMPTY, rc=rc)
skips the initialization and elements in dstArray are updated according to:
do n=1, size(combinedFactorList)
dstArray(combinedFactorIndexList(2, n)) +=
combinedFactorList(n) * srcArray(combinedFactorIndexList(1, n)).
enddo
The ESMF_RouteHandle object returned by ESMF_ArraySMMStore() can be applied to any src/dst Array pairs that are weakly congruent to the Array pair used during precomputation. Arrays are congruent if they are defined on matching DistGrids and the shape of local array allocations match for all DEs. For weakly congruent Arrays the sizes of the undistributed dimensions, that vary faster with memory than the first distributed dimension, are permitted to be different. See section 28.2.16 for an example of this feature demonstrated for the Redist case. The exact same principle applies to the SMM case.
The resources held by sparseMatMulHandle need to be deallocated by the user code before the handle becomes inaccessible.
call ESMF_ArraySMMRelease(routehandle=sparseMatMulHandle, rc=rc)
The Array sparse matrix multiplication also applies to Arrays with undistributed dimensions. The undistributed dimensions are interpreted in a sequentialized manner, much like the distributed dimensions, introducing a second sequence index for source and destination elements. Sequence index 1 is assigned to the first element in the first (i.e. fastest varying in memory) undistributed dimension. The following undistributed elements are labeled in consecutive order as they are stored in memory.
In the simplest case the Array sparse matrix multiplication will apply an identity matrix to the vector of sequentialized undistributed Array elements for every non-zero element in the sparse matrix. The requirement in this case is that the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, be the same for source and destination Array.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
distgridToArrayMap=(/1,2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)
Setting up factorList and factorIndexList is identical to the case for Arrays without undistributed dimensions. Also the call to ESMF_ArraySMMStore() remains unchanged. Internally, however, the source and destination Arrays are checked to make sure the total undistributed element count matches.
if (localPet == 0) then
allocate(factorList(1)) ! PET 0 specifies 1 factor
allocate(factorIndexList(2,1))
factorList = (/0.2/) ! factors
factorIndexList(1,:) = (/5/) ! seq indices into srcArray
factorIndexList(2,:) = (/30/) ! seq indices into dstArray
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else if (localPet == 1) then
allocate(factorList(3)) ! PET 1 specifies 3 factor
allocate(factorIndexList(2,3))
factorList = (/0.5, 0.5, 0.8/) ! factors
factorIndexList(1,:) = (/8, 2, 12/) ! seq indices into srcArray
factorIndexList(2,:) = (/11, 11, 30/) ! seq indices into dstArray
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else
! PETs 2 and 3 do not provide factors
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
endif
The call into the ESMF_ArraySMM() operation is completely transparent with respect to whether source and/or destination Arrays contain undistributed dimensions.
call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
This operation will initialize the entire dstArray to 0.0 and then update four elements:
on DE 1: dstArray[1](2) = 0.5 * srcArray(0,0)[1] + 0.5 * srcArray(0,2)[1], dstArray[2](2) = 0.5 * srcArray(0,0)[2] + 0.5 * srcArray(0,2)[2]
and
on DE 3: dstArray[1](1) = 0.2 * srcArray(0,1)[1] + 0.8 * srcArray(1,3)[1], dstArray[2](1) = 0.2 * srcArray(0,1)[2] + 0.8 * srcArray(1,3)[2].
Here indices between "()" refer to distributed dimensions while indices between "[]" correspond to undistributed dimensions.
In a more general version of the Array sparse matrix multiplication the total undistributed element count, i.e. the product of the sizes of all undistributed dimensions, need not be the same for source and destination Array. In this formulation each non-zero element of the sparse matrix is identified with a unique element in the source and destination Array. This requires a generalization of the factorIndexList argument which now must contain four integer numbers for each element. These numbers in sequence are the sequence index of the distributed dimensions and the sequence index of the undistributed dimensions of the element in the source Array, followed by the sequence index of the distributed dimensions and the sequence index of the undistributed dimensions of the element in the destination Array.
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=3, rc=rc)
srcArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=srcDistgrid, &
totalLWidth=(/1,1/), totalUWidth=(/1,1/), indexflag=ESMF_INDEX_GLOBAL, &
distgridToArrayMap=(/1,2/), undistLBound=(/1/), undistUBound=(/2/), rc=rc)
call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_R8, rank=2, rc=rc)
dstArray = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=dstDistgrid, &
distgridToArrayMap=(/2/), undistLBound=(/1/), undistUBound=(/4/), rc=rc)
Setting up factorList is identical to the previous cases since there is still only one value associated with each non-zero matrix element. However, each entry in factorIndexList now has 4 instead of just 2 components.
if (localPet == 0) then
allocate(factorList(1)) ! PET 0 specifies 1 factor
allocate(factorIndexList(4,1))
factorList = (/0.2/) ! factors
factorIndexList(1,:) = (/5/) ! seq indices into srcArray
factorIndexList(2,:) = (/1/) ! undistr. seq indices into srcArray
factorIndexList(3,:) = (/30/) ! seq indices into dstArray
factorIndexList(4,:) = (/2/) ! undistr. seq indices into dstArray
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else if (localPet == 1) then
allocate(factorList(3)) ! PET 1 specifies 3 factor
allocate(factorIndexList(4,3))
factorList = (/0.5, 0.5, 0.8/) ! factors
factorIndexList(1,:) = (/8, 2, 12/) ! seq indices into srcArray
factorIndexList(2,:) = (/2, 1, 1/) ! undistr. seq indices into srcArray
factorIndexList(3,:) = (/11, 11, 30/) ! seq indices into dstArray
factorIndexList(4,:) = (/4, 4, 2/) ! undistr. seq indices into dstArray
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, factorList=factorList, &
factorIndexList=factorIndexList, rc=rc)
deallocate(factorList)
deallocate(factorIndexList)
else
! PETs 2 and 3 do not provide factors
call ESMF_ArraySMMStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
endif
The call into the ESMF_ArraySMM() operation remains unchanged.
call ESMF_ArraySMM(srcArray=srcArray, dstArray=dstArray, &
routehandle=sparseMatMulHandle, rc=rc)
This operation will initialize the entire dstArray to 0.0 and then update two elements:
on DE 1: dstArray[4](2) = 0.5 * srcArray(0,0)[1] + 0.5 * srcArray(0,2)[2],
and
on DE 3: dstArray[2](1) = 0.2 * srcArray(0,1)[1] + 0.8 * srcArray(1,3)[1],
Here indices in 
refer to distributed dimensions while indices in ![$[]$](img58.png){kind=small}
correspond to undistributed dimensions.
The ESMF_ArrayScatter() and ESMF_ArrayGather() calls, introduced in section 28.2.13, provide a convenient way of communicating data between a Fortran array and all of the DEs of a single Array tile. A key requirement of ESMF_ArrayScatter() and ESMF_ArrayGather() is that the shape of the Fortran array and the Array tile must match. This means that the dimCount must be equal, and that the size of each dimension must match. Element reordering during scatter and gather is only supported on a per dimension level, based on the decompflag option available during DistGrid creation.
While the ESMF_ArrayScatter() and ESMF_ArrayGather() methods cover a broad, and important spectrum of cases, there are situations that require a different set of rules to scatter and gather data between a Fortran array and an ESMF Array object. For instance, it is often convenient to create an Array on a DistGrid that was created with arbitrary, user-supplied sequence indices. See section 35.3.6 for more background on DistGrids with arbitrary sequence indices.
allocate(arbSeqIndexList(10)) ! each PET will have 10 elements
do i=1, 10
arbSeqIndexList(i) = (i-1)*petCount + localPet+1 ! initialize unique
! seq. indices
enddo
distgrid = ESMF_DistGridCreate(arbSeqIndexList=arbSeqIndexList, rc=rc)
deallocate(arbSeqIndexList) call ESMF_ArraySpecSet(arrayspec, typekind=ESMF_TYPEKIND_I4, rank=1, rc=rc)
array = ESMF_ArrayCreate(arrayspec=arrayspec, distgrid=distgrid, rc=rc)
This array object holds 10 elements on each DE, and there is one DE per PET, for a total element count of 10 x petCount. The arbSeqIndexList, used during DistGrid creation, was constructed cyclic across all DEs. DE 0, for example, on a 4 PET run, would hold sequence indices 1, 5, 9, ... . DE 1 would hold 2, 6, 10, ..., and so on.
The usefulness of the user-specified arbitrary sequence indices becomes clear when they are interpreted as global element ids. The ArrayRedist() and ArraySMM() communication methods are based on sequence index mapping between source and destination Arrays. Other than providing a canonical sequence index order via the default sequence scheme, outlined in 28.2.17, ESMF does not place any restrictions on the sequence indices. Objects that were not created with user supplied sequence indices default to the ESMF sequence index order.
A common, and useful interpretation of the arbitrary sequence indices, specified during DistGrid creation, is that of relating them to the canonical ESMF sequence index order of another data object. Within this interpretation the array object created above could be viewed as an arbitrary distribution of a (petCount x 10) 2D array.
if (localPet == 0) then
allocate(farray(petCount,10)) ! allocate 2D Fortran array petCount x 10
do j=1, 10
do i=1, petCount
farray(i,j) = 100 + (j-1)*petCount + i ! initialize to something
enddo
enddo
else
allocate(farray(0,0)) ! must allocate an array of size 0 on all other PETs
endif
For a 4 PET run, farray on PET 0 now holds the following data.
-----1----2----3------------10-----> j
|
1 101, 105, 109, .... , 137
|
2 102, 106, 110, .... , 138
|
3 103, 107, 111, .... , 139
|
4 104, 108, 112, .... , 140
|
|
v
i
On all other PETs farray has a zero size allocation.
Following the sequence index interpretation from above, scattering the data contained in farray on PET 0 across the array object created further up, seems like a well defined operation. Looking at it a bit closer, it becomes clear that it is in fact more of a redistribution than a simple scatter operation. The general rule for such a "redist-scatter" operation, of a Fortran array, located on a single PET, into an ESMF Array, is to use the canonical ESMF sequence index scheme to label the elements of the Fortran array, and to send the data to the Array element with the same sequence index.
The just described "redist-scatter" operation is much more general than the standard ESMF_ArrayScatter() method. It does not require shape matching, and supports full element reordering based on the sequence indices. Before farray can be scattered across array in the described way, it must be wrapped into an ESMF Array object itself, essentially labeling the array elements according to the canonical sequence index scheme.
distgridAux = ESMF_DistGridCreate(minIndex=(/1,1/), &
maxIndex=(/petCount,10/), &
regDecomp=(/1,1/), rc=rc) ! DistGrid with only 1 DE
The first step is to create a DistGrid object with only a single DE. This DE must be located on the PET on which the Fortran data array resides. In this example farray holds data on PET 0, which is where the default DELayout will place the single DE defined in the DistGrid. If the farray was setup on a different PET, an explicit DELayout would need to be created first, mapping the only DE to the PET on which the data is defined.
Next the Array wrapper object can be created from the farray and the just created DistGrid object.
arrayAux = ESMF_ArrayCreate(farray=farray, distgrid=distgridAux, &
indexflag=ESMF_INDEX_DELOCAL, rc=rc)
At this point all of the pieces are in place to use ESMF_ArrayRedist() to do the "redist-scatter" operation. The typical store/execute/release pattern must be followed.
call ESMF_ArrayRedistStore(srcArray=arrayAux, dstArray=array, &
routehandle=scatterHandle, rc=rc)
call ESMF_ArrayRedist(srcArray=arrayAux, dstArray=array, &
routehandle=scatterHandle, rc=rc)
In this example, after ESMF_ArrayRedist() was called, the content of array on a 4 PET run would look like this:
PET 0: 101, 105, 109, .... , 137
PET 1: 102, 106, 110, .... , 138
PET 2: 103, 107, 111, .... , 139
PET 3: 104, 108, 112, .... , 140
Once set up, scatterHandle can be used repeatedly to scatter data from farray on PET 0 to all the DEs of array. All of the resources should be released once scatterHandle is no longer needed.
call ESMF_ArrayRedistRelease(routehandle=scatterHandle, rc=rc)
The opposite operation, i.e. gathering of the array data into farray on PET 0, follows a very similar setup. In fact, the arrayAux object already constructed for the scatter direction, can directly be re-used. The only thing that is different for the "redist-gather", are the srcArray and dstArray argument assignments, reflecting the opposite direction of data movement.
call ESMF_ArrayRedistStore(srcArray=array, dstArray=arrayAux, &
routehandle=gatherHandle, rc=rc)
call ESMF_ArrayRedist(srcArray=array, dstArray=arrayAux, &
routehandle=gatherHandle, rc=rc)
Just as for the scatter case, the gatherHandle can be used repeatedly to gather data from array into farray on PET 0. All of the resources should be released once gatherHandle is no longer needed.
call ESMF_ArrayRedistRelease(routehandle=gatherHandle, rc=rc)
Finally the wrapper Array arrayAux and the associated DistGrid object can also be destroyed.
call ESMF_ArrayDestroy(arrayAux, rc=rc)
call ESMF_DistGridDestroy(distgridAux, rc=rc)
Further, the primary data objects of this example must be deallocated and destroyed.
deallocate(farray) call ESMF_ArrayDestroy(array, rc=rc)
call ESMF_DistGridDestroy(distgrid, rc=rc)
All ESMF_RouteHandle based communication methods, like ESMF_ArrayRedist(), ESMF_ArrayHalo() and ESMF_ArraySMM(), can be executed in blocking or non-blocking mode. The non-blocking feature is useful, for example, to overlap computation with communication, or to implement a more loosely synchronized inter-Component interaction scheme than is possible with the blocking communication mode.
Access to the non-blocking execution mode is provided uniformly across all RouteHandle based communication calls. Every such call contains the optional routesyncflag argument of type ESMF_RouteSync_Flag. Section 51.48 lists all of the valid settings for this flag.
It is an execution time decision to select whether to invoke a precomputed communication pattern, stored in a RouteHandle, in the blocking or non-blocking mode. Neither requires specifically precomputed RouteHandles - i.e. a RouteHandle is neither specifically blocking nor specifically non-blocking.
call ESMF_ArrayRedistStore(srcArray=srcArray, dstArray=dstArray, &
routehandle=routehandle, rc=rc)
The returned RouteHandle routehandle can be used in blocking or non-blocking execution calls. The application is free to switch between both modes for the same RouteHandle.
By default routesyncflag is set to ESMF_ROUTESYNC_BLOCKING in all of the RouteHandle execution methods, and the behavior is that of the VM-wide collective communication calls described in the previous sections. In the blocking mode the user must assume that the communication call will not return until all PETs have exchanged the precomputed information. On the other hand, the user has no guarantee about the exact synchronization behavior, and it is unsafe to make specific assumptions. What is guaranteed in the blocking communication mode is that when the call returns on the local PET, all data exchanges associated with all local DEs have finished. This means that all in-bound data elements are valid and that all out-bound data elements can safely be overwritten by the user.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_BLOCKING, rc=rc)
The same exchange pattern, that is encoded in routehandle, can be executed in non-blocking mode, simply by setting the appropriate routesyncflag when calling into ESMF_ArrayRedist().
At first sight there are obvious similarities between the non-blocking RouteHandle based execution paradigm and the non-blocking message passing calls provided by MPI. However, there are significant differences in the behavior of the non-blocking point-to-point calls that MPI defines and the non-blocking mode of the collective exchange patterns described by ESMF RouteHandles.
Setting routesyncflag to ESMF_ROUTESYNC_NBSTART in any RouteHandle execution call returns immediately after all out-bound data has been moved into ESMF internal transfer buffers and the exchange has been initiated.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_NBSTART, rc=rc)
Once a call with routesyncflag = ESMF_ROUTESYNC_NBSTART returns, it is safe to modify the out-bound data elements in the srcArray object. However, no guarantees are made for the in-bound data elements in dstArray at this phase of the non-blocking execution. It is unsafe to access these elements until the exchange has finished locally.
One way to ensure that the exchange has finished locally is to call with routesyncflag set to ESMF_ROUTESYNC_NBWAITFINISH.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_NBWAITFINISH, rc=rc)
Calling with routesyncflag = ESMF_ROUTESYNC_NBWAITFINISH instructs the communication method to wait and block until the previously started exchange has finished, and has been processed locally according to the RouteHandle. Once the call returns, it is safe to access both in-bound and out-bound data elements in dstArray and srcArray, respectively.
Some situations require more flexibility than is provided by the ESMF_ROUTESYNC_NBSTART - ESMF_ROUTESYNC_NBWAITFINISH pair. For instance, a Component that needs to interact with several other Components, virtually simultaneously, would initiated several different exchanges with ESMF_ROUTESYNC_NBSTART. Calling with ESMF_ROUTESYNC_NBWAITFINISH for any of the outstanding exchanges may potentially block for a long time, lowering the throughput. In the worst case a dead lock situation may arise. Calling with routesyncflag = ESMF_ROUTESYNC_NBTESTFINISH addresses this problem.
call ESMF_ArrayRedist(srcArray=srcArray, dstArray=dstArray, &
routehandle=routehandle, routesyncflag=ESMF_ROUTESYNC_NBTESTFINISH, &
finishedflag=finishflag, rc=rc)
This call tests the locally outstanding data transfer operation in routehandle, and finishes the exchange as much as currently possible. It does not block until the entire exchange has finished locally, instead it returns immediately after one round of testing has been completed. The optional return argument finishedflag is set to .true. if the exchange is completely finished locally, and set to .false. otherwise.
The user code must decide, depending on the value of the returned finishedflag, whether additional calls are required to finish an outstanding non-blocking exchange. If so, it can be done by calling ESMF_ArrayRedist() repeatedly with ESMF_ROUTESYNC_NBTESTFINISH until finishedflag comes back with a value of .true.. Such a loop allows other pieces of user code to be executed between the calls. A call with ESMF_ROUTESYNC_NBWAITFINISH can alternatively be used to block until the exchange has locally finished.
Noteworthy property. It is allowable to invoke a RouteHandle based communication call with routesyncflag set to ESMF_ROUTESYNC_NBTESTFINISH or ESMF_ROUTESYNC_NBWAITFINISH on a specific RouteHandle without there being an outstanding non-blocking exchange. As a matter of fact, it is not required that there was ever a call made with ESMF_ROUTESYNC_NBSTART for the RouteHandle. In these cases the calls made with ESMF_ROUTESYNC_NBTESTFINISH or ESMF_ROUTESYNC_NBWAITFINISH will simply return immediately (with finishedflag set to .true.).
Noteworthy property. It is fine to mix blocking and non-blocking invocations of the same RouteHandle based communication call across the PETs. This means that it is fine for some PETs to issue the call with ESMF_ROUTESYNC_BLOCKING (or using the default), while other PETs call the same communication call with ESMF_ROUTESYNC_NBSTART.
Noteworthy restriction. A RouteHandle that is currently involved in an outstanding non-blocking exchange may not be used to start any further exchanges, neither blocking nor non-blocking. This restriction is independent of whether the newly started RouteHandle based exchange is made for the same or for different data objects.
The Array class is part of the ESMF index space layer and is built on top of the DistGrid and DELayout classes. The DELayout class introduces the notion of decomposition elements (DEs) and their layout across the available PETs. The DistGrid describes how index space is decomposed by assigning logically rectangular index space pieces or DE-local tiles to the DEs. The Array finally associates a local memory allocation with each local DE.
The following is a list of implementation specific details about the current ESMF Array.
INTERFACE:
interface assignment(=) array1 = array2ARGUMENTS:
type(ESMF_Array) :: array1 type(ESMF_Array) :: array2STATUS:
DESCRIPTION:
Assign array1 as an alias to the same ESMF Array object in memory as array2. If array2 is invalid, then array1 will be equally invalid after the assignment.
The arguments are:
INTERFACE:
interface operator(==) if (array1 == array2) then ... endif OR result = (array1 == array2)RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_Array), intent(in) :: array1 type(ESMF_Array), intent(in) :: array2STATUS:
DESCRIPTION:
Test whether array1 and array2 are valid aliases to the same ESMF Array object in memory. For a more general comparison of two ESMF Arrays, going beyond the simple alias test, the ESMF_ArrayMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
interface operator(/=) if (array1 /= array2) then ... endif OR result = (array1 /= array2)RETURN VALUE:
logical :: resultARGUMENTS:
type(ESMF_Array), intent(in) :: array1 type(ESMF_Array), intent(in) :: array2STATUS:
DESCRIPTION:
Test whether array1 and array2 are not valid aliases to the same ESMF Array object in memory. For a more general comparison of two ESMF Arrays, going beyond the simple alias test, the ESMF_ArrayMatch() function (not yet implemented) must be used.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayCopy(arrayOut, arrayIn, rc)ARGUMENTS:
type(ESMF_Array), intent(inout) :: arrayOut
type(ESMF_Array), intent(in) :: arrayIn
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
DESCRIPTION:
Copy data from one ESMF_Array object to another.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateFromPtr<rank><type><kind>(distgrid, farrayPtr, & datacopyflag, distgridToArrayMap, computationalEdgeLWidth, & computationalEdgeUWidth, computationalLWidth, & computationalUWidth, totalLWidth, & totalUWidth, name, rc)RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateDataPtr<rank><type><kind>ARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: distgridToArrayMap(:) integer, intent(in), optional :: computationalEdgeLWidth(:) integer, intent(in), optional :: computationalEdgeUWidth(:) integer, intent(in), optional :: computationalLWidth(:) integer, intent(in), optional :: computationalUWidth(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Create an ESMF_Array object from existing local native Fortran arrays with pointer attribute. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the farrayPtr argument which will be different on each PET. The bounds of the local arrays are preserved by this call and determine the bounds of the total region of the resulting Array object. Bounds of the DE-local exclusive regions are set to be consistent with the total regions and the specified distgrid argument. Bounds for Array dimensions that are not distributed are automatically set to the bounds provided by farrayPtr.
This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateFromPtrArb<rank><type><kind>(distgrid, farrayPtr, & haloSeqIndexList, keywordEnforcer datacopyflag, distgridToArrayMap, & name, rc)RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateDataPtrArb<rank><type><kind>ARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid <type> (ESMF_KIND_<kind>), pointer :: farrayPtr(<rank>) integer, intent(in) :: haloSeqIndexList(:) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: distgridToArrayMap(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Create an ESMF_Array object from existing local native Fortran arrays with pointer attribute, according to distgrid. Besides farrayPtr each PET must issue this call with identical arguments in order to create a consistent Array object. The bounds of the local arrays are preserved by this call and determine the bounds of the total region of the resulting Array object. Bounds of the DE-local exclusive regions are set to be consistent with the total regions and the specified distgrid argument. Bounds for Array dimensions that are not distributed are automatically set to the bounds provided by farrayPtr.
This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateAsmdShap<rank><type><kind>(distgrid, farray, & indexflag, datacopyflag, distgridToArrayMap, & computationalEdgeLWidth, computationalEdgeUWidth, computationalLWidth, & computationalUWidth, totalLWidth, & totalUWidth, undistLBound, undistUBound, name, rc)RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateDataAssmdShape<rank><type><kind>ARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: distgridToArrayMap(:) integer, intent(in), optional :: computationalEdgeLWidth(:) integer, intent(in), optional :: computationalEdgeUWidth(:) integer, intent(in), optional :: computationalLWidth(:) integer, intent(in), optional :: computationalUWidth(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(in), optional :: undistLBound(:) integer, intent(in), optional :: undistUBound(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Create an ESMF_Array object from an existing local native Fortran array. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the farray argument which will be different on each PET. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for Array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.
This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateAsmdShapArb<rank><type><kind>(distgrid, farray, & indexflag, haloSeqIndexList, datacopyflag, distgridToArrayMap, & computationalEdgeLWidth, computationalEdgeUWidth, computationalLWidth, & computationalUWidth, totalLWidth, & totalUWidth, undistLBound, undistUBound, name, rc)RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateDataAssmdShapeArb<rank><type><kind>ARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid <type> (ESMF_KIND_<kind>), intent(in), target :: farray(<rank>) type(ESMF_Index_Flag), intent(in) :: indexflag integer, intent(in) :: haloSeqIndexList(:) -- The following arguments require argument keyword syntax (e.g. rc=rc). -- type(ESMF_DataCopy_Flag), intent(in), optional :: datacopyflag integer, intent(in), optional :: distgridToArrayMap(:) integer, intent(in), optional :: computationalEdgeLWidth(:) integer, intent(in), optional :: computationalEdgeUWidth(:) integer, intent(in), optional :: computationalLWidth(:) integer, intent(in), optional :: computationalUWidth(:) integer, intent(in), optional :: totalLWidth(:) integer, intent(in), optional :: totalUWidth(:) integer, intent(in), optional :: undistLBound(:) integer, intent(in), optional :: undistUBound(:) character (len=*), intent(in), optional :: name integer, intent(out), optional :: rcSTATUS:
DESCRIPTION:
Create an ESMF_Array object from an existing local native Fortran array. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the farray argument which will be different on each PET. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for Array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.
This interface requires a 1 DE per PET decomposition. The Array object will not be created and an error will be returned if this condition is not met.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateLocalArray(distgrid, localarrayList, &
indexflag, datacopyflag, distgridToArrayMap, computationalEdgeLWidth, &
computationalEdgeUWidth, computationalLWidth, computationalUWidth, &
totalLWidth, totalUWidth, undistLBound, undistUBound, name, rc)
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateLocalArrayARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid
type(ESMF_LocalArray), intent(in) :: localarrayList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Index_Flag), intent(in), optional :: indexflag
type(ESMF_DataCopy_Flag),intent(in), optional :: datacopyflag
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: computationalEdgeLWidth(:)
integer, intent(in), optional :: computationalEdgeUWidth(:)
integer, intent(in), optional :: computationalLWidth(:)
integer, intent(in), optional :: computationalUWidth(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object from existing ESMF_LocalArray objects. The decomposition and distribution is specified by the distgrid argument. Each PET must issue this call with identical arguments in order to create a consistent Array object. The only exception is the localarrayList argument which will be
different on each PET. The local arrays provided must be dimensioned
according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for Array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.
This interface is able to handle multiple DEs per PET.
The not distributed Array dimensions form a tensor of rank = array.rank - distgrid.dimCount. The widths of the computational region are set to the provided value, or zero by default, for all tensor elements. Use ESMF_ArraySet() to change these default settings after the Array object has been created.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateLocalArrayArb(distgrid, localarrayList, &
haloSeqIndexList, indexflag, datacopyflag, &
distgridToArrayMap, computationalEdgeLWidth, computationalEdgeUWidth, &
computationalLWidth, computationalUWidth, &
totalLWidth, totalUWidth, undistLBound, undistUBound, name, rc)
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateLocalArrayArbARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid
type(ESMF_LocalArray), intent(in) :: localarrayList(:)
integer, intent(in) :: haloSeqIndexList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Index_Flag), intent(in), optional :: indexflag
type(ESMF_DataCopy_Flag),intent(in), optional :: datacopyflag
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: computationalEdgeLWidth(:)
integer, intent(in), optional :: computationalEdgeUWidth(:)
integer, intent(in), optional :: computationalLWidth(:)
integer, intent(in), optional :: computationalUWidth(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object from existing ESMF_LocalArray objects according to distgrid. Each PET must issue this call in unison in order to create a consistent Array object. The local arrays provided must be dimensioned according to the DE-local total region. Bounds of the exclusive regions are set as specified in the distgrid argument. Bounds for array dimensions that are not distributed can be chosen freely using the undistLBound and undistUBound arguments.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateAllocate(distgrid, typekind, indexflag, &
distgridToArrayMap, computationalEdgeLWidth, computationalEdgeUWidth, &
computationalLWidth, computationalUWidth, totalLWidth, totalUWidth, &
undistLBound, undistUBound, name, vm, rc)
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateAllocateARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid
type(ESMF_TypeKind_Flag), intent(in) :: typekind
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Index_Flag), intent(in), optional :: indexflag
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: computationalEdgeLWidth(:)
integer, intent(in), optional :: computationalEdgeUWidth(:)
integer, intent(in), optional :: computationalLWidth(:)
integer, intent(in), optional :: computationalUWidth(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
type(ESMF_VM), intent(in), optional :: vm
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object and allocate uninitialized data space according to typekind and distgrid. The Array rank is indirectly determined by the incoming information. Each PET must issue this call in unison in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the distgrid and the optional Width arguments.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateAllocateArb(distgrid, typekind, &
haloSeqIndexList, distgridToArrayMap, &
undistLBound, undistUBound, name, rc)
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateAllocateArbARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid
type(ESMF_TypeKind_Flag), intent(in) :: typekind
integer, intent(in) :: haloSeqIndexList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object and allocate uninitialized data space according to typekind and distgrid. The Array rank is indirectly determined by the incoming information. Each PET must issue this call in unison in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the distgrid and haloSeqIndexList arguments.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateAllocateAS(distgrid, arrayspec, indexflag, &
distgridToArrayMap, computationalEdgeLWidth, computationalEdgeUWidth, &
computationalLWidth, computationalUWidth, totalLWidth, totalUWidth, &
undistLBound, undistUBound, name, vm, rc)
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateAllocateASARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid
type(ESMF_ArraySpec), intent(in) :: arrayspec
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
type(ESMF_Index_Flag), intent(in), optional :: indexflag
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: computationalEdgeLWidth(:)
integer, intent(in), optional :: computationalEdgeUWidth(:)
integer, intent(in), optional :: computationalLWidth(:)
integer, intent(in), optional :: computationalUWidth(:)
integer, intent(in), optional :: totalLWidth(:)
integer, intent(in), optional :: totalUWidth(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
type(ESMF_VM), intent(in), optional :: vm
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object and allocate uninitialized data space according to arrayspec and distgrid. Each PET must issue this call with identical arguments in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the arguments to this call: distgrid and the optional Width arguments.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate()
function ESMF_ArrayCreateAllocateASArb(distgrid, arrayspec, &
haloSeqIndexList, distgridToArrayMap, &
undistLBound, undistUBound, name, rc)
RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateAllocateASArbARGUMENTS:
type(ESMF_DistGrid), intent(in) :: distgrid
type(ESMF_ArraySpec), intent(in) :: arrayspec
integer, intent(in) :: haloSeqIndexList(:)
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(in), optional :: distgridToArrayMap(:)
integer, intent(in), optional :: undistLBound(:)
integer, intent(in), optional :: undistUBound(:)
character (len=*), intent(in), optional :: name
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object and allocate uninitialized data space according to arrayspec and distgrid. Each PET must issue this call in unison in order to create a consistent Array object. DE-local allocations are made according to the total region defined by the arguments to this call: distgrid and haloSeqIndexList arguments.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
! Private name; call using ESMF_ArrayCreate() function ESMF_ArrayCreateCopy(array, rc)RETURN VALUE:
type(ESMF_Array) :: ESMF_ArrayCreateCopyARGUMENTS:
type(ESMF_Array), intent(in) :: array
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Create an ESMF_Array object as the copy of an existing Array.
The return value is the newly created ESMF_Array object.
The arguments are:
INTERFACE:
subroutine ESMF_ArrayDestroy(array, noGarbage, rc)ARGUMENTS:
type(ESMF_Array), intent(inout) :: array
-- The following arguments require argument keyword syntax (e.g. rc=rc). --
logical, intent(in), optional :: noGarbage
integer, intent(out), optional :: rc
STATUS:
DESCRIPTION:
Destroy an ESMF_Array, releasing the resources associated with the object.
By default a small remnant of the object is kept in memory in order to prevent problems with dangling aliases. The default garbage collection mechanism can be overridden with the noGarbage argum