next up previous contents
Next: 4 Infrastructure: Utilities Up: ESMC_crefdoc Previous: 2 Superstructure   Contents


3 Infrastructure: Fields and Grids

8 Overview of Infrastructure Data Handling

The ESMF infrastructure data classes are part of the framework's hierarchy of structures for handling Earth system model data and metadata on parallel platforms. The hierarchy is in complexity; the simplest data class in the infrastructure represents a distributed array and the most complex data class represents a bundle of physical fields that are discretized on the same grid. Data class methods are called both from user-written code and from other classes internal to the framework. Data classes are distributed over DEs, or Decomposition Elements. A DE represents a piece of a decomposition. A DELayout is a collection of DEs with some associated connectivity that describes a specific distribution. For example, the distribution of a grid divided into four segments in the x-dimension would be expressed in ESMF as a DELayout with four DEs lying along an x-axis. This abstract concept enables a data decomposition to be defined in terms of threads, MPI processes, virtual decomposition elements, or combinations of these without changes to user code. This is a primary strategy for ensuring optimal performance and portability for codes using the ESMF for communications. ESMF data classes are useful because they provide a standard, convenient way for developers to collect together information related to model or observational data. The information assembled in a data class includes a data pointer, a set of attributes (e.g. units, although attributes can also be user-defined), and a description of an associated grid. The same set of information within an ESMF data object can be used by the framework to arrange intercomponent data transfers, to perform I/O, for communications such as gathers and scatters, for simplification of interfaces within user code, for debugging, and for other functions. This unifies and organizes codes overall so that the user need not define different representations of metadata for the same field for I/O and for component coupling. Since it is critical that users be able to introduce ESMF into their codes easily and incrementally, ESMF data classes can be created based on native Fortran pointers. Likewise, there are methods for retrieving native Fortran pointers from within ESMF data objects. This allows the user to perform allocations using ESMF, and to retrieve Fortran arrays later for optimized model calculations. The ESMF data classes do not have associated differential operators or other mathematical methods. For flexibility, it is not necessary to build an ESMF data object all at once. For example, it's possible to create a field but to defer allocation of the associated field data until a later time.

Key Features
Hierarchy of data structures designed specifically for the Earth system domain and high performance, parallel computing.
Multi-use ESMF structures simplify user code overall.
Data objects support incremental construction and deferred allocation.
Native Fortran arrays can be associated with or retrieved from ESMF data objects, for ease of adoption, convenience, and performance.

8.1 Infrastructure Data Classes

The main classes that are used for model and observational data manipulation are as follows:

8.2 Design and Implementation Notes

  1. In communication methods such as Regrid, Redist, Scatter, etc. the FieldBundle and Field code cascades down through the Array code, so that the actual computations exist in only one place in the source.

9 Field Class

9.1 Description

An ESMF Field represents a physical field, such as temperature. The motivation for including Fields in ESMF is that bundles of Fields are the entities that are normally exchanged when coupling Components.

The ESMF Field class contains distributed, discretized field data, a reference to its associated grid, and metadata. The Field class maintains the relationship of how a data array maps onto a grid (e.g. one item per cell located at the cell center, one item per cell located at the NW corner, one item per cell vertex, ...). This means that different Fields which are on the same underlying ESMF Grid but have different staggerings can share the same Grid object without needing to replicate it multiple times.

Fields can be added to States for use in inter-Component data communications. Fields can also be added to FieldBundles, which are currently defined as groups of Fields on the same underlying grid. One motivation for FieldBundles is convenience; another is the ability to perform optimized collective data transfers.

Field communications, including data redistribution, regriding, scatter, and gather, are enabled in this release. Field halo update operation is not enabled in this release and will be enabled in subsequent releases.

ESMF does not currently support vector fields, so the components of a vector field must be stored as separate Field objects.

A Field serves as an annotator of data, since it carries a description of the grid it is associated with and metadata such as name and units. Fields can be used in this capacity alone, as convenient, descriptive containers into which arrays can be placed and retrieved. However, for most codes the primary use of Fields is in the context of import and export States, which are the objects that carry coupling information between Components. Fields enable data to be self-describing, and a State holding ESMF Fields contains data in a standard format that can be queried and manipulated.

The sections below go into more detail about Field usage.

9.1.1 Field Creation and Destruction

Fields can be created and destroyed at any time during application execution. However, these Field methods require some time to complete. We do not recommend that the user create or destroy Fields inside performance-critical computational loops.

All versions of the ESMF_FieldCreate() routines require a Grid object as input, or require a Grid be added before most operations involving Fields can be performed. The Grid contains the information needed to know which Decomposition Elements (DEs) are participating in the processing of this Field, and which subsets of the data are local to a particular DE.

The details of how the create process happens depends on which of the variants of the ESMF_FieldCreate() call is used. Some of the variants are discussed below.

There are versions of the ESMF_FieldCreate() interface which create the Field based on the input Grid. The ESMF can allocate the proper amount of space but not assign initial values. The user code can then get the pointer to the uninitialized buffer and set the initial data values.

Other versions of the ESMF_FieldCreate() interface allow user code to attach arrays that have already been allocated by the user. Empty Fields can also be created in which case the data can be added at some later time.

For versions of Create which do not specify data values, user code can create an ArraySpec object, which contains information about the typekind and rank of the data values in the array. Then at Field create time, the appropriate amount of memory is allocated to contain the data which is local to each DE.

When finished with a ESMF_Field, the ESMF_FieldDestroy method removes it. However, the objects inside the ESMF_Field created externally should be destroyed separately, since objects can be added to more than one ESMF_Field. For example, the same ESMF_Grid can be referenced by multiple ESMF_Fields. In this case the internal Grid is not deleted by the ESMF_FieldDestroy call.

9.2 Class API

9.3 C++: Class Interface ESMC_Field - Public C interface to the ESMF Field class (Source File: ESMC_Field.h)

The code in this file defines the public C Field class and declares method signatures (prototypes). The companion file ESMC_Field.C contains the definitions (full code bodies) for the Field methods.

10 Array Class

10.1 Description

The Array class is an alternative to the Field class for representing distributed, structured data. Unlike Fields, which are built to carry grid coordinate information, Arrays can only carry information about the indices associated with grid cells. Since they do not have coordinate information, Arrays cannot be used to calculate interpolation weights. However, if the user can supply interpolation weights (using a package such as SCRIP), the Array sparse matrix multiply operation can be used to apply the weights and transfer data to the new grid. Arrays can also perform redistribution, scatter, and gather operations.

Like Fields, Arrays can be added to a State and used in inter-component data communications. Arrays can also be grouped together into ArrayBundles so that collective operations can be performed on the whole group. One motivation for this is convenience; another is the ability to schedule optimized, collective data transfers.

From a technical standpoint, the ESMF_Array class is an index space based, distributed data storage class. It provides DE-local memory allocations within DE-centric index regions and defines the relationship to the index space described by DistGrid. The Array class offers common communication patterns within the index space formalism. As part of the ESMF index space layer Array has close relationship to the DistGrid and DELayout classes.

10.2 Class API

10.3 C++: Class Interface ESMC_Array - Public C interface to the ESMF Array class (Source File: ESMC_Array.h)

The code in this file defines the public C Array class and declares method signatures (prototypes). The companion file ESMC_Array.C contains the definitions (full code bodies) for the Array methods.

11 ArraySpec Class

11.1 Description

An ArraySpec is a very simple class that contains type, kind, and rank information about an array. This information is stored in two parameters. TypeKind describes the data type of the elements in the array and their precision. Rank is the number of dimensions in the array.

The only methods that are associated with the ArraySpec class are those that allow you to set and retrieve this information.

11.2 Class API

11.3 C++: Class Interface ESMC_ArraySpec - uniform access to arrays from F90 and C++ (Source File: ESMC_ArraySpec.h)

The code in this file defines the public C ArraySpec interfaces and declares method signatures (prototypes). The companion file ESMC_ArraySpec.C contains the definitions (full code bodies) for the ArraySpec methods.

12 Grid Class

12.1 Description

The ESMF Grid class is used to describe the geometry and discretization of logically rectangular physical grids. It also contains the description of the grid's underlying topology and the decomposition of the physical grid across the available computational resources. The most frequent use of the Grid class is to describe physical grids in user code so that sufficient information is available to perform ESMF methods such as regridding.

In the current release (v3.1.0) the functionality in this class is partially implemented. Multi-tile grids are not supported, and edge connectivities are not implemented and default to aperiodic. Other constraints of the current implementation are noted in the usage section and in the API descriptions.

Key Features
Representation of grids formed by logically rectangular regions, including uniform and rectilinear grids (e.g. lat-lon grids), curvilinear grids (e.g. displaced pole grids), and grids formed by connected logically rectangular regions (e.g. cubed sphere grids) [CONNECTED REGIONS ARE NOT YET SUPPORTED].
Support for 1D, 2D, 3D, and higher dimension grids.
Distribution of grids across computational resources for parallel operations - users set which grid dimensions are distributed.
Grids can be created already distributed, so that no single resource needs global information during the creation process.
Options to define periodicity and other edge connectivities either explicitly or implicitly via shape shortcuts [EDGE CONNECTIVITIES CURRENTLY DEFAULT TO APERIODIC BOUNDS].
Options for users to define grid coordinates themselves or call prefabricated coordinate generation routines for standard grids [NO GENERATION ROUTINES YET].
Options for incremental construction of grids.
Options for using a set of pre-defined stagger locations or for setting custom stagger locations.

12.1.1 Grid Representation in ESMF

ESMF Grids are based on the concepts described in A Standard Description of Grids Used in Earth System Models [Balaji 2006]. In this document Balaji introduces the mosaic concept as a means of describing a wide variety of Earth system model grids. A mosaic is composed of grid tiles connected at their edges. Mosaic grids includes simple, single tile grids as a special case.

The ESMF Grid class is a representation of a mosaic grid. Each ESMF Grid is constructed of one or more logically rectangular Tiles. A Tile will usually have some physical significance (e.g. the region of the world covered by one face of a cubed sphere grid).

The piece of a Tile that resides on one DE (for simple cases, a DE can be thought of as a processor - see section on the DELayout) is called a LocalTile. For example, the six faces of a cubed sphere grid are each Tiles, and each Tile can be divided into many LocalTiles.

Every ESMF Grid contains a DistGrid object, which defines the Grid's index space, topology, distribution, and connectivities. It enables the user to define the complex edge relationships of tripole and other grids. The DistGrid can be created explicitly and passed into a Grid creation routine, or it can be created implicitly if the user takes a Grid creation shortcut. Options for grid creation are described in more detail in section 12.1.8. The DistGrid used in Grid creation describes the properties of the Grid cells. In addition to this one, the Grid internally creates DistGrids for each stagger location. These stagger DistGrids are related to the original DistGrid, but may contain extra padding to represent the extent of the index space of the stagger. These DistGrids are what are used when a Field is created on a Grid.

12.1.2 Supported Grids

The range of supported grids in ESMF can be defined by:

12.1.3 Grid Topologies and Periodicity

ESMF has shortcuts for the creation of standard Grid topologies or shapes up to 3D. In many cases, these enable the user to bypass the step of creating a DistGrid before creating the Grid. The basic call is ESMF_GridCreateShapeTile(). With this call, the user can specify for each dimension whether there is no connection, it is periodic, it is a pole, or it is a bipole. The assumed connectivities for poles and bipoles are described in section [*]. Connectivities are specified using the ESMF_GridConn parameter, which has values such as ESMF_GRIDCONN_PERIODIC.

The table below shows the ESMF_GridConn settings used to create standard shapes in 2D using the ESMF_GridCreateShapeTile() call. Two values are specified for each dimension, one for the low end and one for the high end of the dimension's index values. Note that connectivities have not been implemented as of v4.0.0 and default to aperiodic bounds.

2D Shape connDim1(1) connDim1(2) connDim2(1) connDim2(2)

If the user's grid shape is too complex for an ESMF shortcut routine, or involves more than three dimensions, a DistGrid can be created to specify the shape in detail. This DistGrid is then passed into a Grid create call.

12.1.4 Grid Distribution

ESMF Grids have several options for data distribution (also referred to as decomposition). As ESMF Grids are cell based, these options are all specified in terms of how the cells in the Grid are broken up between DEs.

The main distribution options are regular, irregular, and arbitrary. A regular distribution is one in which the same number of contiguous grid cells are assigned to each DE in the distributed dimension. A irregular distribution is one in which unequal numbers of contiguous grid cells are assigned to each DE in the distributed dimension. An arbitrary distribution is one in which any grid cell can be assigned to any DE. Any of these distribution options can be applied to any of the grid shapes (i.e., rectangle) or types (i.e., rectilinear). Support for arbitrary distribution is limited in v4.0.0, See section [*] for more detail descriptions.

Figure 7 illustrates options for distribution.

Figure 7: Examples of regular and irregular decomposition of a grid a that is 6x6, and an arbitrary decomposition of a grid b that is 6x3.

A distribution can also be specified using the DistGrid, by passing object into a Grid create call.

12.1.5 Grid Coordinates

Grid Tiles can have uniform, rectilinear, or curvilinear coordinates. The coordinates of uniform grids are equally spaced along their axes, and can be fully specified by the coordinates of the two opposing points that define the grid's physical span. The coordinates of rectilinear grids are unequally spaced along their axes, and can be fully specified by giving the spacing of grid points along each axis. The coordinates of curvilinear grids must be specified by giving the explicit set of coordinates for each grid point. Curvilinear grids are often uniform or rectilinear grids that have been warped; for example, to place a pole over a land mass so that it does not affect the computations performed on an ocean model grid. Figure 8 shows examples of each type of grid.

Any of these logically rectangular grid types can be combined through edge connections to form a mosaic. Cubed sphere and yin-yang grids are examples of mosaic grids. Note that as of v4.0.0 multi-tile grids have not yet been implemented.

Figure 8: Types of logically rectangular grid tiles. Red circles show the values needed to specify grid coordinates for each type.

Each of these coordinate types can be set for each of the standard grid shapes described in section 12.1.3.

The table below shows how examples of common single Tile grids fall into this shape and coordinate taxonomy. Note that any of the grids in the table can have a regular or arbitrary distribution.

  Uniform Rectilinear Curvilinear
Sphere Global uniform lat-lon grid Gaussian grid Displaced pole grid
Rectangle Regional uniform lat-lon grid Gaussian grid section Polar stereographic grid section

12.1.6 Coordinate Specification and Generation

There are two ways of specifying coordinates in ESMF. The first way is for the user to set the coordinates. The second way is to take a shortcut and have the framework generate the coordinates.

No ESMF generation routines are currently available.

See Section [*] for more description and examples of setting coordinates.

12.1.7 Staggering

Staggering is a finite difference technique in which the values of different physical quantities are placed at different locations within a grid cell.

The ESMF Grid class supports a variety of stagger locations, including cell centers, corners, and edge centers. The default stagger location in ESMF is the cell center, and cell counts in Grid are based on this assumption. Combinations of the 2D ESMF stagger locations are sufficient to specify any of the Arakawa staggers. ESMF also supports staggering in 3D and higher dimensions. There are shortcuts for standard staggers, and interfaces through which users can create custom staggers.

As a default the ESMF Grid class provides symmetric staggering, so that cell centers are enclosed by cell perimeter (e.g. corner) stagger locations. This means the coordinate arrays for stagger locations other than the center will have an additional element of padding in order to enclose the cell center locations. However, to achieve other types of staggering, the user may alter or eliminate this padding by using the appropriate options when adding coordinates to a Grid.

In v4.0.0, only the cell center stagger location is supported for an arbitrarily distributed grid. For examples and a full description of the stagger interface see Section [*].

12.1.8 Options for Building Grids

ESMF Grid objects must represent a wide range of grid types and use cases, some of them quite complex. As a result, multiple ways to build Grid objects are required. This section describes the stages to building Grids, the options for each stage, and typical calling sequences.

In ESMF there are two main stages to building Grids. The ESMF_GridStatus value stored within the Grid object reflects the stage the Grid has attained (see Section [*]). These stages are:

  1. Create the Grid topology or shape. At the completion of this stage, the Grid has a specific topology and distribution, but empty coordinate arrays. The Grid can be used as the basis for allocating a Field. Its ESMF_GridStatus parameter has a value of ESMF_GRIDSTATUS_SHAPE_READY.

    The options for specifying the Grid shape are:

  2. Specify the Grid coordinates and any other information required for regridding (this can vary depending on the particular regridding method). At the completion of this stage, the Grid can be used in a regridding operation (once Grid is connected to regrid; as of v3.1.0, it is not). Its ESMF_GridStatus has a value of ESMF_GRIDSTATUS_REGRID_READY.

When creating the Grid shape and specifying the Grid coordinates, the user can either specify all required information at once, or can provide information incrementally. The call ESMF_GridCreateEmpty() builds a Grid object container that can be filled in with a subsequent call to the ESMF_GridSetCommitShapeTile() method. The ESMF_GridSetCommitShapeTile() creates the grid and sets the appropriate flag to indicate that its usable (the status equals ESMF_GRIDSTATUS_SHAPE_READY after the commit). The Grid is implicitly in a valid state after being committed.

For consistency's sake the ESMF_GridSetCommitShapeTile() call must occur on the same or a subset of the PETs as the ESMF_GridCreateEmpty() call. The ESMF_GridSetCommitShapeTile() call uses the VM for the context in which it's executed and the "empty" Grid contains no information about the VM in which it was run. If the ESMF_GridSetCommitShapeTile() call occurs in a subset of the PETs in which the ESMF_GridCreateEmpty() was executed, the Grid is created only in that subset. The grid objects outside the subset will still be "empty" and not usable.

The following table summarizes possible call sequences for building Grids.

Create Shape
From shape shortcut
grid = ESMF_GridCreateShapeTile(...)
Using DistGrid with general create interface
distgrid = ESMF_DistGridCreate(...)
grid = ESMF_GridCreate(distgrid, ...)
grid = ESMF_GridCreateEmpty(...)
call ESMF_GridSetCommitShapeTile(grid, ...)
Set Coordinates
Set coordinates by copy or reference
call ESMF_GridSetCoord(grid, ...)
Retrieve ESMF Array of coordinates from Grid and set values
call ESMF_GridGetCoord(grid, esmfArray, ...), set values
Retrieve local bounds and native array from Grid and set values
call ESMF_GridGetCoord(grid, lbound, ubound, array), set values

12.2 Class API: General Grid Methods

12.3 C++: Class Interface ESMC_Grid - Public C interface to the Grid object (Source File: ESMC_Grid.h)

The code in this file defines the C public Grid class and declares method signatures (prototypes). The companion file ESMC_Grid.C contains the definitions (full code bodies) for the Grid methods.



 #include "ESMCI_Grid.h"
 extern "C" {
   class declaration type
 typedef struct {
       ESMCI::Grid *grid;

13 LocStream Class

13.1 Description

A location stream (LocStream) is used to represent the locations of a set of data points. The values of the data points are stored within a Field or FieldBundle created using the LocStream.

In the data assimilation world, LocStreams can be thought of as a set of observations. Their locations are generally described using Cartesian (x, y, z), or (lat, lon, height) coordinates. There is no assumption of any regularity in the positions of the points. To make the concept more general, the locations for each data point are represented using a construct called Keys, which can include other descriptors besides location.

Although Keys are similar in concept to ESMF Attributes they have important differences. First, Keys always occur as vectors, never as scalars. Second, Keys are local to the DE: each DE can have a different Key list with a different number of of elements. Third, the local Key list always has the same number of elements as there are local observations on that DE. Finally, Keys may be used for the distribution of LocStreams. As such, they must be defined before the LocStream is distributed.

LocStreams can be very large. Data assimilation systems might use LocStreams with up to $10^{8}$ observations, so efficiency is critical.

Common operations involving LocStreams are similar to those involving Grids. In data assimilation, for example, there is an immediate need to:

  1. Create a Field or FieldBundle on a LocStream.
  2. Redistribute data between Fields defined on LocStreams.
  3. Gather a bundle of data defined on a LocStream to a root DE (for output). Similarly, scatter from a root DE.
  4. Halo region exchange for a Field defined by a haloed LocStream.
  5. Extract Fortran array from Field which was defined by a LocStream.

The operations on the Fortran arrays underlyinng LocStreams are usually simple numerical ones. However, it is necessary to sort them in place, and access only portions of the them. It would not be efficient to continually create new LocStreams to reflect this sorting. Instead, the sorting is managed by the application through permutation arrays while keeping the data in place. Locations can become inactive, e.g., if the quality control asserts that observation is invalid. This can be managed again by the application through masks.

13.1.1 How is a LocStream different than a Grid?

A LocStream differs from a Grid in that no topological structure is maintained between the points (e.g. the class contains no information about which point is the neighbor of which other point).

13.1.2 How is a LocStream different than a Mesh?

A Mesh consists of irregularly positioned points, but it has connectivity also: each data point has a set of neighboring data points. There is no requirement that the points in a LocStream have connectivity, indeed any particular spatial relationship to one another. Due to their heritage from data assimilation, many of the operations on LocStreams do not resemble typical operations on Meshes, for example in a finite-volume or finite-element code.

14 Mesh Class

14.1 Description

Unstructured grids are commonly used in the computational solution of Partial Differential equations. These are especially useful for problems that involve complex geometry, where using the less flexible structured grids can result in grid representation of regions where no computation is needed. Finite element and finite volume methods map naturally to unstructured grids and are used commonly in hydrology, ocean modeling, and many other applications.

In order to provide support for application codes using unstructured grids, the ESMF library provides a class for representing unstructured grids called the Mesh. Fields can be created on a Mesh to hold data. Fields created on a Mesh can also be used as either the source or destination or both of an interpolaton (i.e. an ESMF_FieldRegridStore() call) in ESMF allowing data to be moved to or from or between unstructured grids. This section describes the Mesh and how to create and use them in ESMF.

14.1.1 Mesh Representation in ESMF

A Mesh in ESMF is described in terms of nodes and elements. A node is a point in space which represents where the coordinate information in a Mesh is located. This is also where Field data may be located in a Mesh (i.e. Fields may be created on a Mesh's nodes). An element is a higher dimensional shape constructed of nodes. Elements give a Mesh its shape and define the relationship of the nodes to one another.

14.1.2 Supported Meshes

The range of Meshes supported by ESMF are defined by several factors: dimension, element types, and distribution.

ESMF currently only supports Meshes whose number of coordinate dimensions (spatial dimension) is 2 or 3. The dimension of the elements in a Mesh (parametric dimension) must be less than or equal to the spatial dimension, but also must be either 2 or 3. This means that an ESMF mesh may be either 2D elements in 2D space, 3D elements in 3D space, or a manifold constructed of 2D elements embedded in 3D space.

ESMF currently supports two types of elements for each Mesh parametric dimension. For a parametric dimension of 2 the supported element types are triangles or quadralaterals. For a parametric dimension of 3 the supported element types are tetrahedrons and hexahedrons. See Section [*] for diagrams of these. The Mesh supports any combination of element types within a particular dimension, but types from different dimensions may not be mixed, for example, a Mesh cannot be constructed of both quadralaterals and tetrahedra.

ESMF currently only supports distributions where every node on a PET must be a part of an element on that PET. In other words, there must not be nodes without an element on a PET.

14.2 Class API

15 DistGrid Class

15.1 Description

The ESMF_DistGrid class sits on top of the DELayout class and holds domain information in index space. A DistGrid object captures the index space topology and describes its decomposition in terms of DEs. Combined with DELayout and VM the DistGrid defines the data distribution of a domain decomposition across the computational resources of an ESMF component.

The global domain is defined as the union or ``patchwork'' of logically rectangular (LR) sub-domains or patches. The DistGrid create methods allow the specification of such a patchwork global domain and its decomposition into exclusive, DE-local LR regions according to various degrees of user specified constraints. Complex index space topologies can be constructed by specifying connection relationships between patches during creation.

The DistGrid class holds domain information for all DEs. Each DE is associated with a local LR region. No overlap of the regions is allowed. The DistGrid offers query methods that allow DE-local topology information to be extracted, e.g. for the construction of halos by higher classes.

A DistGrid object only contains decomposable dimensions. The minimum rank for a DistGrid object is 1. A maximum rank does not exist for DistGrid objects, however, ranks greater than 7 may lead to difficulties with respect to the Fortran API of higher classes based on DistGrid. The rank of a DELayout object contained within a DistGrid object must be equal to the DistGrid rank. Higher class objects that use the DistGrid, such as an Array object, may be of different rank than the associated DistGrid object. The higher class object will hold the mapping information between its dimensions and the DistGrid dimensions.

15.2 Class API

15.3 C++: Class Interface ESMC_DistGrid - Public C interface to the ESMF DistGrid class (Source File: ESMC_DistGrid.h)

The code in this file defines the public C DistGrid class and declares method signatures (prototypes). The companion file ESMC_DistGrid.C contains the definitions (full code bodies) for the DistGrid methods.

next up previous contents
Next: 4 Infrastructure: Utilities Up: ESMC_crefdoc Previous: 2 Superstructure   Contents