11 Architectural Overview

The ESMF architecture is characterized by the layering strategy shown in Figure 1. User code components that implement the science portions of an application, for example a sea ice or land model, are sandwiched between two layers. The upper layer is denoted the superstructure layer and the lower layer the infrastructure layer. The role of the superstructure layer is to provide a shell which encompasses user code and provides a context for interconnecting input and output data streams between components. The key elements of the superstructure are described in Section 11.2. These elements include classes that wrap user code, ensuring that all components present consistent interfaces. The infrastructure layer provides a foundation that developers of science components can use to speed construction and to ensure consistent, guaranteed behavior. The elements of the infrastructure include constructs to support parallel processing with data types tailored to Earth science applications, specialized libraries to support consistent time and calendar management and performance, error handling and scalable I/O tools. The infrastructure layer is described in Section 11.3. A hierarchical combination of superstructure, user code components, and infrastructure are joined together to form an ESMF application.

11.1 Key Concepts

The ESMF architecture and programming paradigm are based upon five key concepts: modularity, flexibility, hierarchical organization, communication within components, and a uniform communication API.

11.1.1 Modularity

The ESMF design is based upon modular Components. There are two types of Components, one of which represents models (Gridded Components) and one which represents couplers (Coupler Components). Data are always passed between Components using a data structure called a State, which can store Fields, FieldBundles of Fields, Arrays, and other States. A Gridded Component stores no information about the internals of the Gridded Components that it interacts with; this information is passed in through the argument lists of the initialize, run, and finalize methods. The information that is passed in through the argument list can be a State from another Gridded Component, or it can be a function pointer that performs a computation or communication on a State. These function pointers are called Transforms, and they are available as AttachableMethods created by Coupler Components. They are called inside the Gridded Component they are passed into. Although Transforms add some complexity to the framework (and their use is not required), they are what will enable ESMF to accommodate virtually any model of communication between Components.

Modularity means that an ESMF component stores nothing about the internals of other components. This allows components to be used more easily in multiple contexts.

11.1.2 Flexibility

The ESMF does not dictate how models should be coupled; it simply provides tools for creating couplers. For example, both a hub-and-spokes type coupling strategy and pairwise strategies are supported. The ESMF also allows model communications to occur mid-timestep, if desired. Sequential, concurrent, and mixed modes of execution are supported.

The ESMF does not impose restrictions on how data flows through an application. This accommodates scientific innovation - if you want your atmospheric model to communicate with your sea ice model mid-timestep, ESMF will not stop you.

11.1.3 Hierarchical organization

ESMF allows applications to be composed hierarchically. For example, physics and dynamics modules can be defined as separate Gridded Components, coupled together with a Coupler Component, and all of these nested within a single atmospheric Gridded Component. The atmospheric Gridded Component can be run standalone, or can be included in a larger climate or data assimilation application. See Figure 2 for an illustrative example.

The data structure that enables scalability in ESMF is the derived type Gridded Component. Fortran alone does not allow you to create generic components - you'd have to create derived types for PhysComp, and DynComp, and PhysDynCouplerComp, and AtmComp. In ESMF, these are always of type GridComp or CplComp, so they can be called by the same drivers (whether that driver is a standard ESMF driver or another model), and use the same methods without having to overload them with many specific derived types. It is the same idea when you want to support different implementations of the same component, like multiple dynamics.

The ESMF defines a hierarchical, scalable architecture that is natural for organizing very complex applications, and for allowing exchangeable Components.

Figure 2: A typical building block for an ESMF application consists of a parent Gridded Component, two or more child Gridded Components, and a Coupler Component. The parent Gridded Component is called by an application driver. All ESMF Components have initialize, run, and finalize methods. The diagram shows that when the application driver calls initialize on a parent Gridded Component, the call cascades down to all of its children, so that the result is that the entire ``tree'' of Components is initialized. The run and finalize methods work the same way. In this example a hurricane simulation is built from ocean and atmosphere Gridded Components. The data exchange between the ocean and atmosphere is handled by an ocean-atmosphere Coupler Component. Since the whole hurricane simulation is a Gridded Component, it could be easily be treated as a child and coupled to another Gridded Component, rather than being driven directly by the application driver. A similar diagram could be drawn for an atmospheric model containing physics and dynamics components, as described in Section 11.1.3.

11.1.4 Communication within Components

Communication in ESMF always occurs within a Component. It can occur internal to a Gridded Component, and have nothing to do with interactions with other Components (setting aside synchronization issues), or it can occur within a Coupler Component or a transform generated by a Coupler Component. A result of the rule that all communication happens within a Component is that Coupler Components must always be defined on the union of all the Components that they couple together. Models can choose to use whatever mechanism they want for intra-model communications.

The point is that although the ESMF defines some simple rules for communication, the communication mechanism that the framework uses is not hardwired into its architecture - the sends and receives or puts and gets are enclosed within Gridded Components, Coupler Components and Transforms. The intent is to accommodate multiple models of communication and technical innovations.

11.1.5 Uniform communication API

ESMF has a single API for shared and distributed memory that, unlike MPI, accounts for NUMA achitectures and does not treat all processes as being identical. It is possible for users to set ESMF communications to a strictly message passing mode and put in their own OpenMP commands.

The goal is to create a programming paradigm that is peformance sensitive to the architecture beneath it without being discouragingly complicated.

11.2 Superstructure

The ESMF superstructure layer in a unifying context within which user components are interconnected. Classes called Gridded Components, Coupler Components, and States are used within the superstructure to achieve this flexibility.

11.2.1 Import and export State classes

User code components under ESMF use special interface objects for Component to Component data exchanges. These objects are of type import State and export State. These special types support a variety of methods that allow user code components to do things like fill an export State object with data to be shared with other components or query an import State object to determine its contents. In keeping with the overall requirements for high-performance it is permitted for import State and export State contents to use references or pointers to Component data, so that costly data copies of potentially large data structures can be avoided where possible. The content of an import State and an export State can be made self-describing.

11.2.2 Interface standards

The import State and export State abstractions are designed to be flexible enough so that ESMF does not need to mandate a single format for fields. For example, ESMF does not prescribe the units of quantities exported or imported. However, ESMF does provide mechanisms to describe units, memory layout, and grid coordinates. This allows the ESMF software to support a range of different policies for physical fields. The interoperability experiments that we are using to demonstrate ESMF make use of the emerging CF conventions [1] for describing physical fields. This is a policy choice for that set of experiments. The ESMF software itself can support arbitrary conventions for labeling and characterizing the contents of States.

11.2.3 Gridded Component class

The Gridded Component class describes a user component that takes in one import State and produces one export State. Examples of Gridded Components are major Earth system model components such as land surface models, ocean models, atmospheric models and sea ice models. Components used for linear algebra manipulations in a state estimation or data assimilation optimization procedure are also created as Gridded Components. In general the fields within an import State and export State of a Gridded Component will use the same discrete grid.

11.2.4 Coupler Component class

The other top-level Component class supported in the ESMF architecture is a Coupler Component. This class is used for Components that take one or more import States as input and map them through spatial and temporal interpolation or extrapolation onto one or more output export States. In a Coupler Component it is often the case that the export State(s) is on a different discrete grid to that of the import State(s). For example, in a coupled ocean-atmosphere simulation a Coupler Component might be used to map a set of sea-surface fields in an ocean model to appropriate planetary boundary layer fields in an atmospheric model.

11.2.5 Flexible data and control flow

Import States, export States, Gridded Components and Coupler Components can be arrayed flexibly within a superstructure layer. Using these constructs, it is possible to configure a set of Components with multiple pairwise Coupler Components, Figure 4. It is also possible to configure a set of concurrently executing Gridded Components joined through a single Coupler Component of the style shown in Figure 3.

Figure 3: ESMF supports configurations with a single central Coupler Component. In this case inputs from all Gridded Components are transferred and regridded through the central coupler.

Figure 4: ESMF also supports configurations with multiple point to point Coupler Components. These take inputs from one Gridded Component and transfer and regrid the data before passing it to another Gridded Component. This schematic shows a flow of data between two Coupler Components that connect three Gridded Components: an atmosphere model with a land model, and the same atmosphere model with a data assimilation system.

The set of superstructure abstractions allows flexible data flow and control between components. However, components will often use different discrete grids, and time-stepping components may march forward with different time intervals. In a parallel compute environment different components may be distributed in a different manner on the underlying compute resources. The ESMF infrastructure layer provides elements to manage this complexity.

11.3 Infrastructure

Figure 5 illustrates three Gridded Components, each with a different Grids, being coupled together. In order to achieve this coupling several steps beyond defining import State and export State objects to act as data conduits are required. Coupler Components are needed that can interpolate between the different Grids. The necessary transformations may also involve mapping between different units and/or memory layout conventions for the Fields that pass between Components. In a parallel compute environment the Coupler Components may also be required to map between different domain decompositions. In order to advance in time correctly the separate Gridded Components must have compatible notions of time. Approaches to parallelism within the Gridded Components must also be compatible. The Infrastructure layer contains a set of classes that address these issues and assist in managing overall system complexity.

Figure 5: Schematic showing the coupling of components that use different discrete Grids and different time-stepping. In this example, Component NCAR Atmosphere might use a spectral Grid based on spherical harmonics, Component GFDL Ocean might use a latitude-longitude Grid but with a patched decomposition that does not include land masses, and Component NSIPP Land might use a m osaic-based Grid for representing vegetation patchiness and a catchment area based Grid for river routings. The ESMF infrastructure layer contains tools to help develop software for coupling between Components on different Grids, mapping between Components with different distributions in a multi-processor compute environment and synchronizing events between Components with different time-stepping intervals and algorithms.

11.3.1 FieldBundle, Field and Array classes

FieldBundle, Field and Array classes contain data together with descriptive physical and computational attribute information. The physical attributes include information that describes the units of the data. The computational attributes include information on the layout in memory of the field data. The Field class is primarily geared toward structured data. A comparable class, called Location Stream, provides a self-describing container for unstructured observational data streams.

11.3.2 Grid class

The Grid class is an extensible class that holds discrete grid information. It has subtypes that allow it to serve as a container for the full range of different physical grids that might arise in a coupled system. In the example in figure 5 objects of type Grid would hold grid information for each of the spectral grid, the latitude-longitude grid, the mosaic grid and the catchment grid.

The Grid class is also used to represent the decomposition of a data structure into subdomains, typically for parallel processing purposes. The class is designed to support a generalized ``ghosting'' for tiled decompositions of finite difference, finite volume and finite element codes.

11.3.3 Time and Calendar management

To support synchronization between Components, several time and calendar management classes are provided. These capabilities are provided in the Time, Time Interval, Calendar, Clock, and Alarm classes. These classes allow Gridded and Coupler Component processing to be latched to a common controlling Clock, and to schedule notification of regular events, such as a coupling intervals, and unique events.

11.3.4 Config resource file manager

The Config class is a utility for accessing configuration files that are in ASCII format. This utility enables configuration files to be prepared using more flexible formatting than Fortran namelists - for example, it permits the input of tables of data.

11.3.5 DELayout and virtual machine

To provide a mechanism for ensuring performance portability, ESMF defines DELayout and virtual machine (VM) classes. These classes provide a set of high-level and platform independent interfaces to performance critical parallel processing communication routines. These routines can be tuned to specific platforms to ensure optimal parallel performance on many platforms.

11.3.6 Logging and error handling

The LogErr class is designed to aid in managing the complexity of multi-Component applications. It provides ESMF with a unified mechanism for managing logs and error reporting.

11.3.7 File input and output

The infrastructure layer will define a set of IO classes for storing and retrieving Array, Field, and Grid information to and from persistent storage.