ESMF Releases

• Thank you! The ESMF team is grateful for the many years of support, ideation, feedback, testing, and code contributions that the ESMF user community has provided since its inception. We look forward to continuing this partnership with the community to further improve and add features to ESMF.
• The public Fortran API in this release is backward compatible with the previous release of ESMF, 8.6.0, and patch release 8.6.1. There were some API changes, none of which require user code changes. The complete list of API changes is summarized in a table showing interface changes since ESMF 8.6.0. The table includes the rationale and impact for each change.
• No bit-for-bit changes are expected for this release compared to ESMF 8.6.0 or 8.6.1.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged compared to ESMF 8.6.1.
• All of the fixes and improvements released with patch 8.6.1 are included in release 8.7.0.
• Starting with this release all pull requests into develop will be automatically tested using a combination of settings including: BOPT=O/g, COMPILER=gfortran/clang, COMM=mpiuni/mpich/openmpi, and TESTEXHAUSTIVE=ON/OFF. These tests can be manually executed on any fork/branch by going to Development Tests in the Actions tab on GitHub.
• Added ESMF_StateLog(), ESMF_FieldLog(), ESMF_ArrayBundleLog(), ESMF_ArrayLog(), and ESMF_DistGridLog() methods. These methods allow inspection of ESMF objects from the user level, which can be useful during code development and debugging. ESMF object logging is also leveraged by the standard NUOPC Verbosity attributes of the generic NUOPC components.
• Added ESMF_HConfigLog() and ESMF_HConfigMatch() methods to facilitate ESMF_HConfig object usage in application code. New sections with code examples were added to the reference manual discussing comparison of ESMF_HConfig objects, and usage of quoted strings and the Norway problem.
• Added the capability to set an ESMF_TimeInterval object from a string in ISO 8601 time interval format. This allows a user familiar with that format to more intuitively set a time interval in ESMF.
• Added a new repeat capability to the ESMF Clock. This capability can be enabled by setting the repeatDuration argument when creating a Clock. When enabled the clock stays within the range from the clock’s startTime to its startTime+repeatDuration. For example, when advancing if the current time goes past startTime+repeatDuration, then it loops back to startTime and continues from there. ESMF Alarms also work with this capability. One example of when this capability can be useful is when spinning up a model.
• An enhancement was made to ESMPy to allow associating a node mask with an ESMPy Mesh. The Mesh.add_nodes method now accepts a keyword, node_mask, through which a user can pass a mask array of length n_nodes. Previously it was only possible to associate a mask with a mesh's elements.
• Allow setting the endFlag when finalizing ESMPy. In particular, this allows the user to control whether the MPI library is finalized along with ESMPy or is still available afterwards.
• Builds using mpiuni (the stub, single-processor MPI library bundled with ESMF) now support ESMF_PIO=internal, allowing full I/O with mpiuni. This is particularly useful for ESMPy.
• The NUOPC Connection Options now support remapMethod “conserve_2nd” and extrapMethod “nearest_d”.
• New NUOPC application prototypes were added to showcase basic C API support through ESMX and NUOPC. This approach leverages components in dynamically loaded objects.
• A new NUOPC application prototype was added to demonstrate coupling a component written in Julia, with a cap in C.
• A more scalable method of calculating cubed sphere coordinates was added. This method can be selected using the new coordCalcFlag argument added to the ESMF_GridCreateCubedSphere interfaces. Setting the argument to ESMF_CUBEDSPHERECALC_LOCAL selects the new method.
• The routeHandle calculation was improved to be faster when mapping data locations to their eventual PET destinations. This improvement is expected to increase scalability, especially when redistributing data from a small number of PETs to a large number.
• An internal data member in the ESMF_Array class, with a size that scales with the total number of PETs (i.e. MPI tasks) across which the Array was created, has been eliminated. This improvement dramatically reduces the memory requirement for applications that have a large number of Arrays or Fields and run on large numbers of PETs.
• During ‘make install’ the FindESMF.cmake file is copied into the ESMF_INSTALL_PREFIX/cmake directory. The default location can be overwritten by setting an environment variable, ESMF_INSTALL_CMAKEDIR. This variable can be specified as an absolute path (starting with "/") or relative to ESMF_INSTALL_PREFIX.
• The ESMF capability to access components through dynamically loaded objects was made more portable by introducing wildcard support for the suffix of specified shared objects.

  Bug Fixes:

• A fix was made in ESMPy to permit the creation of Grids where the product of the dimensions exceeds approximately 268,000,000.
• The build of libesmftrace_preload.dylib on some Darwin (Mac) systems with OpenMPI has been fixed. (Incorrect link flags were being used in this build.)
• A fix was made to ESMPy to allow the creation of 3D Meshes. Previously creating a 3D Mesh in ESMPy would result in an error.
• The regridding system was fixed so that the patch regrid method (ESMF_REGRIDMETHOD_PATCH) can be used on a source Field built on a Mesh where the PET list of the component that contains the destination Field has some PETs that aren’t in the source Field’s component’s PET list. Previously this condition would have resulted in an error.
• A fix was made to ESMX to add link_module_paths to the CMAKE_MODULE_PATH list.
• A fix was made to ESMX to support using variables that are defined by setting link_packages in esmxBuild.yaml files.

• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

  Platform-specific issues:

• Compiling ESMF with GCC version 8.1.0 triggers an internal compiler error in ESMF_HConfig.F90 due to the use of allocatable character variables. Earlier and later versions of GCC do not have this issue.
• On Darwin, with version 15 of the clang C compiler, when building under Rosetta, it is sometimes necessary to add "-Wl,-ld_classic" to environment variables ESMF_CXXLINKOPTS, ESMF_CLINKOPTS, and ESMF_F90LINKOPTS to work around link errors. (For more details, see the related GitHub issue.)
• The cray (cce) compiler currently has problems running PIO with mpiuni, at least for some versions of this compiler (tested with version 15.0.1).

• This patch release is backward compatible with ESMF 8.6.0. There were no API changes in this release.
• No bit-for-bit changes are expected for this release compared to ESMF 8.6.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged compared to ESMF 8.6.0.
• The index space information (minIndex, maxIndex, elementCount) returned by method ESMF_FieldGet() is now correct for a Field on any ESMF_STAGGERLOC (Grid) or ESMF_MESHLOC (Mesh) location.
• The ESMF_Config class has been enhanced by removing the 1024 character limit. It is now possible to set strings to values longer than 1024 characters. The maximum size is determined by the ESMF_Config buffer size, which is 256k.
• Multiple bugs in the implementation of ESMF_Config have been fixed.
  • A bug has been fixed that prevented changing values to a different string length.
  • It is now possible to enclose number sign (#) characters within single or double quotes in order to include this character in string values.
  • It is now possible to enclose apostrophes / single quote (‘) characters within double quotes in order to include this character in string values.
  • It is now possible to enclose double quote (“) characters within single quotes in order to include this character in string values.
  • It is now possible to set values to blank and change blank values to non-blank values.
  • Note: ESMF_Config will be deprecated in a future major release for the preferred ESMF_HConfig class, which supports YAML coded configuration files. Deprecation has not been scheduled at this time due to the high impact it will have on existing configuration files.
• The ESMF library target within the FindESMF module for CMake has been changed to ‘ESMF::ESMF’. This conforms to the imported library standard as seen in modules provided by Kitware. For backwards compatibility ‘ESMF’ has been added as an alias target.
• The issue of inadvertently exporting ISO_C_BINDING symbols via the ESMF Fortran module has been addressed.
• ESMF_RUNTIME_ environment variable keys that are set on the top level of the configuration file ingested by the ESMF_Initialize() method are now correctly interpreted.
• The ESMF_NVML=OFF build option now works correctly as documented.
• The build of libesmftrace_preload.dylib on some Darwin (Mac) systems with OpenMPI has been fixed. (Incorrect link flags were being used in this build.)
• The problem of building ESMF with macro ESMF_NO_GETHOSTID defined has been fixed.
• The ESMFMESH unstructured mesh file format now correctly begins the connection index calculation at the start_index value when the start_index attribute is attached to the elementConn variable. Previously, beginning in ESMF 8.3.0, this attribute had been incorrectly ignored.
• The ESMF_HConfig and ESMF_HConfigIter equality operators have been fixed. Previously, on some platforms, identical objects could be detected as not equal which could lead to problems with iterations not terminating correctly, etc.

• Same as ESMF 8.6.0.

• The public Fortran API in this release is backward compatible with the previous release, ESMF 8.5.0. There were a few API changes, none of which require user code changes. The complete list of API changes is summarized in a table showing interface changes since ESMF 8.5.0. The table includes the rationale and impact for each change.
• No bit-for-bit changes were observed for this release compared to ESMF 8.5.0. This is based on test runs with the Intel compilers using options "-O2 -fp-model precise". However, changes were made to the implementation of the conservative weight calculation to remove a problem caused by the same calculation being optimized differently in two places in the code. Bit-for-bit changes compared to ESMF 8.5.0 from this change are possible when using regridding methods ESMF_REGRIDMETHOD_CONSERVE or ESMF_REGRIDMETHOD_CONSERVE_2ND and not using strict floating point compiler options.
• No changes that affect the status of existing regridding methods were made to the ESMF regrid weight generation methods and applications. However, the tables summarizing the ESMF regridding status were extended to cover the spherical vector regridding option added in this release.
• Keys to match all of the ESMF_RUNTIME_* environment variables were added to the App Options of the ESMX run configuration. This feature provides a convenient way to control the run-time behavior of the ESMX executable from its standard configuration file.
• Basic accelerator device management for multi-component applications was implemented. This feature allows the user to assign accelerator devices via the "devList" option to individual components. "devList" is available as a new argument to ESMF_GridCompCreate() and ESMF_CplCompCreate(), and is also accessible on the ESMX level as Component Label Option. Once specified, ESMF handles the device bookkeeping, and provides the user with context specific device information through the ESMF_VMGet() API. This feature can be leveraged in connection with popular accelerator programming paradigms like OpenACC, OpenMP, or standard language approaches to offload component code to the desired devices.
• Basic support for vector regridding was added to the ESMF_FieldRegridStore() method. The initial implementation of this feature is limited to 2D tangential vectors (expressed in terms of east and north components) on a spherical geometry (e.g. an ESMF_Grid with ESMF_COORDSYS_SPH_DEG) and requires that the vector components are stored in an ungridded Field dimension. The advantage of using this capability, which maps vectors through 3D Cartesian space, instead of regridding both components separately, is that it provides more accurate results, particularly in the polar regions.
• The "normalization" attribute in mapping files written by the ESMF regrid weight generation methods and applications was changed to now correctly set "N/A" for anything other than the conservative regridding methods. User code that accesses the "normalization" attribute in ESMF mapping files might need to be adjusted accordingly.
• A new entry point was added to the ESMF_FieldCreate() interface allowing creation from an existing Field object. The interface supports slicing with respect to trailing ungridded dimensions.
• The ESMF_ArrayCreate() entry point that allows creation from an existing Array object was extended to support slicing with respect to trailing undistributed dimensions.
• The Read and Write operations for multi-tile Arrays and Fields (e.g., for representing a cubed sphere grid as a six-tile grid) have been extended to permit I/O for Arrays / Fields with layouts other than 1 DE per PET. This change applies to the ESMF_ArrayRead(), ESMF_ArrayWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_FieldBundleRead(), and ESMF_FieldBundleWrite() methods. The remaining limitations of the implementation are discussed in the Restrictions and Future Work section of the I/O Capability in the reference manual.
• A basic ESMF C API was added to provide access to the ESMF tracing and profiling capability from code written in C.
• Crashes that were observed with ESMF applications during MPI_Finalize() with the Darshan or Cray Performance Analysis Tool active have been resolved. (The issue was traced back to an interaction with the MPI Tools interface that used to be initialized during ESMF_Initialize(). ESMF no longer initializes the MPI Tools interface by default to avoid this problem.)
• The issues noted in previous releases with regard to using the ESMF tracing and profiling feature for MPI calls have been resolved. This capability is now fully functional on all the supported platforms.
• NVIDIA's NVHPC compiler suite is now fully supported by ESMF for both ESMF_OS=Linux and ESMF_OS=Unicos, and setting ESMF_COMPILER=nvhpc. This is particularly important for users looking at exploring ESMF accelerator device support on systems with NVIDIA GPUs.

• Using vector regridding (vectorRegrid=.true.) through the ESMF_FieldRegridStore() interface in combination with either conservative regridding method (regridmethod=ESMF_REGRIDMETHOD_CONSERVE or regridmethod=ESMF_REGRIDMETHOD_CONSERVE_2ND) currently results in an error.
• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

  Platform-specific issues:

• Compiling ESMF with GCC version 8.1.0 triggers an internal compiler error in ESMF_HConfig.F90 due to the use of allocatable character variables. Earlier and later versions of GCC do not have this issue.
• On Darwin, with version 15 of the clang C compiler, when building under Rosetta, it is sometimes necessary to add "-Wl,-ld_classic" to environment variables ESMF_CXXLINKOPTS, ESMF_CLINKOPTS, and ESMF_F90LINKOPTS to work around link errors. (For more details, see the related GitHub issue.)

• The public Fortran API in this release is backward compatible with the last release of ESMF, 8.4.0, and patch releases 8.4.1 and 8.4.2. There were a few minor API changes, most of which do not require user code changes. The exception is a change to the Read() and Write() methods for multi-tile Arrays or Fields, where the tile number placeholder character in the fileName argument was changed from "#" to "*". A number of new interfaces were added. The complete list of API changes is summarized in a table showing interface changes since ESMF 8.4.0, including the rationale and impact for each change.
• Some bit-for-bit changes are expected for this release compared to ESMF 8.4.0, and patch releases 8.4.1 and 8.4.2. This is based on test runs with the Intel compiler suite using options "-O2 -fp-model precise", and due to the following changes:
  • Fixed a problem in the first-order conservative weight calculation that might occur in some situations when a destination cell is a concave quadrilateral. The fix was made to both the 2D Cartesian and 2D spherical weight conservative calculation. Expected bit-for-bit changes:
    • Small changes to first-order conservative weights for destination cells that are concave. These changes can occur for either 2D Cartesian or 2D spherical Grids or Meshes.
  • Updated the ESMF cubed sphere coordinate generation algorithm to use R8 reals. This makes the algorithm more accurate and also aligns it better with the same algorithm used by NASA. Expected bit-for-bit changes:
    • Small changes to the coordinate values produced by ESMF_GridCreateCubedSphere() or ESMF_MeshCreateCubedSphere(), and also changes to anything which depends on those values (e.g. regridding weights to/from an ESMF generated cube sphere Grid).
• No changes that affect regridding status were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• All of the fixes and improvements released with patches 8.4.1 and 8.4.2 are included in release 8.5.0.
• Significant improvements have been made to the Earth System Model eXecutable (ESMX) layer:
  • Component building has been integrated into the ESMX build system.
  • The build configuration has been enhanced through new build options.
  • The ESMX_Builder script has been added to simplify building ESMX applications.
  • The ESMX run configuration file has been updated to YAML formatting.
  • The names of the ESMX build and run configuration files can be customized.
  • The ESMX Data Component, which allows users to define import/export fields and prescribe data, has been added to the ESMX Layer.
• A hierarchical configuration class (ESMF_HConfig) with Fortran API was implemented. ESMF_HConfig covers the YAML 1.2 standard and integrates with the existing configuration class ESMF_Config for backward compatibility.
• The ESMF_Initialize() method now supports reading startup configurations from a YAML file.
• The ESMF_InfoSet() interface was overloaded to allow setting ESMF_Info key value pairs from an ESMF_HConfig object.
• The NUOPC methods that implement ingestion of PetList, RunSequence and Attribute information have been overloaded to support the YAML format through the ESMF_HConfig class.
• A new named constant, ESMF_STATEINTENT_INTERNAL, for type(ESMF_StateIntent_Flag) was added. This new option allows a state to be marked exclusively for component internal access.
• A generic geometry class (ESMF_Geom) was added to ESMF. A Geom object can be created from any object of the existing geometry classes (ESMF_Grid, ESMF_Mesh, ESMF_LocStream, and ESMF_XGrid). Having this class allows a user to pass a generic geometry object through a coupled system without knowing its exact type. To support greater flexibility in coupled systems, some ESMF operations are now supported on Geom (e.g. Field creation). More will be added as the need arises.
• Creating a Mesh, or adding nodes to an existing Mesh, without specifying the nodeOwners argument now works, even if there are PETs with zero nodes. Previously this condition led to a hang inside the respective ESMF method.
• Creating a Mesh without specifying node information now works, even if there are PETs with zero elements. Previously this condition led to a hang inside the respective ESMF method.
• Grids that contain DEs of zero width are now supported in regridding. Previously using a Grid of this type with some regridding methods (e.g. ESMF_REGRIDMETHOD_CONSERVE) would result in an error.
• The srcTermProcessing argument was added to the version of ESMF_FieldRegridStore() that operates on an XGrid. This fixes the issue where in some cases bit-for-bit reproducibility wasn't available in that version of ESMF_FieldRegridStore().
• The RouteHandle reuse optimization in ESMF_FieldBundleRegridStore() was extended to include the Grid-to/from-Mesh and Mesh-to/from-Mesh combinations. This optimization is leveraged by the NUOPC_Connector and results in a significant reduction of cost (both time and memory) during model initialization when Fields on Meshes are present in the import- and/or exportStates.
• All of the RedistStore() methods that take the optional srcToDstTransposeMap argument now support negative entries. The order of elements along each negative dimension entry is reversed.
• Two new dataFillScheme options, "nan" and "snan", have been added to the ESMF_FieldFill() method. These options fill a Field with IEEE quiet and signaling NaN ("Not a Number") values, respectively.
• An optional logical flag (isESMFAllocated) was added to the ESMF_FieldGet(), ESMF_ArrayGet(), and ESMF_LocalArrayGet() methods to allow the user to query whether the respective data allocation is held by ESMF.
• A bug in the auto-detection of file type in the ESMF_RegridWeightGen application was fixed. Previously SCRIP grid files could sometimes be incorrectly identified as GRIDSPEC files.
• The -t, --src_type, and --dst_type arguments of the ESMF_RegridWeightGen application that allow grid file types to be specified explicitly, have been re-enabled.
• The Read and Write operations for multi-tile Arrays and Fields (e.g., for representing a cubed sphere grid as a six-tile grid) have been extended to permit I/O for Arrays / Fields with undistributed / ungridded dimensions. This change applies to the ESMF_ArrayRead(), ESMF_ArrayWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_FieldBundleRead(), and ESMF_FieldBundleWrite() methods.
• For Read() and Write() operations for multi-tile Arrays and Fields (e.g., for representing a cubed sphere grid as a six-tile grid), the file name placeholder character in the fileName string that stands for the tile number has been changed from "#" to "*". This change was needed to support obtaining the fileName string from ESMF Config file, where the "#" character is interpreted as the start of a comment.
• Creating a cubed sphere grid under ESMPy with corner stagger by calling ESMF.Grid() with add_corner_stagger=True now works correctly. Previously this condition would lead to a run-time error.
• ESMF is available through Spack, a popular package manager for High Performance Computing. A user can build ESMF, and its dependencies, in a convenient and standard way using the Spack tools. ESMF core team members are now maintainers of the official ESMF Spack package. To ensure a robust integration, the ESMF build procedure through Spack is automatically tested using GitHub Actions.
• Intel oneAPI compiler support was improved and additional test combinations were added. A new ESMF User's Guide section discussing Intel compiler support was added to provide guidance with the classic to oneAPI transition.
• Fujitsu compiler support was added under ESMF_OS=Linux. Use ESMF_COMPILER=fujitsu to access the option.

• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

  Platform-specific issues:

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default, ESMF uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• Compiling ESMF with GCC version 8.1.0 triggers an internal compiler error in ESMF_HConfig.F90 due to the use of allocatable character variables. Earlier and later versions of GCC do not have this issue.
• On Darwin, with the GNU gfortran+gcc combination, when building MPICH3 from source, it is important to specify the "--enable-two-level-namespace" configure option. By default, i.e. without this option, on Darwin, the produced MPICH compiler wrappers include a linker flag (-flat_namespace) that causes issues with C++ exception handling. Building and linking ESMF applications with MPICH compiler wrappers that specify this linker option leads to "mysterious" application aborts during execution.
• On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting the environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
• There is an issue with intercepting the MPI calls for profiling on some of the supported platforms. This results in a single FAIL reported for ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Catania: Darwin+GNU+mpich
  • Green: Darwin+gfortranclang+mpich/openmpi
  • Gaea: Unicos+GNU+cray-mpich
• There is an issue with loading the libesmftrace_preload.so library on some of the supported platforms. This results in a reported CRASH for ESMF_TraceIOUTest.F90 and ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Discover: Linux+GNU+intelmpi
  • Gaea: Unicos+Intel+cray-mpich
  • Gaea: Unicos+Intel+mpiuni

• This patch release is backward compatible with ESMF 8.4.0 and 8.4.1
• No bit-for-bit changes are expected for this release compared to ESMF 8.4.0 and 8.4.1.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The ESMF profiler mode (activated by setting environment variable ESMF_RUNTIME_PROFILE=ON) has been made compatible with user code that calls the standard ESMF component methods from within OpenMP threaded regions. This is a rare scenario. However, it can be encountered e.g. under NOAA's Unified Forecast System (UFS) when enabling the threading option for NASA's GOCART chemistry component.
• A linking issue reported for the Darwin.gfortranclang.default build configuration, when building ESMF under the Spack package manager, has been fixed.

• Same as ESMF 8.4.1.

• This patch release is backward compatible with ESMF 8.4.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.4.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• A bug in the implementation of method ESMF_FieldGet() was fixed. The problematic code was accessing the optional, intent(out) "name" argument without the proper present() check. As a consequence, code not specifying an actual "name" argument when making this call was at risk of suffering from memory corruption issues. Due to the fact that the ESMF library internally is making calls to ESMF_FieldGet() without passing the "name" argument, it must be assumed that all user code is at risk of memory corruption issues when using ESMF 8.4.0.
• A problem of incorrect ESMPy project metadata was fixed. This issue was causing failures during installation when using python's setuptools module v67.1.0 or later: ("configuration error: `project.maintainers[{data__maintainers_x}]` must not contain {'author'} properties"). The fix allows ESMPy installation to work with the latest setuptools. We recommend this release for anyone doing a new installation of ESMPy.
• The NUOPC_Driver code responsible for stepping components through the initialization protocol has been optimized to reduce the amount of synchronization needed between components. This allows for a greater level of concurrency between components during initialization, resulting in a reduction of execution time.
• A number of code adjustments were made to support Intel's LLVM based OneAPI compiler suite under the common ESMF_COMPILER=intel setting. Classic and OneAPI Intel compiler flavors are now supported under the same setting.

• Same as ESMF 8.4.0.

• The public Fortran API in this release is backward compatible with the last release of ESMF 8.3.0 and patch release ESMF 8.3.1. There were a few minor API changes, none of which require user code changes. The list of API changes is summarized in a table showing interface changes since ESMF 8.3.0, including the rationale and impact for each change.
• The Python module name for ESMPy was changed in this release from "ESMF" to "esmpy" for better alignment with Python Enhancement Proposal (PEP) guidelines. This change will require user code changes! ESMPy users will have to do a global search and replace on their scripts to adjust to the module name change. Also notice that the internal ESMF source tree directory structure has changed, moving ./src/addon/ESMPy to ./src/addon/esmpy. This might affect anybody keeping an ESMF repository clone, and maintaining scripts that depend on the internal directory naming.
• No bit-for-bit changes were observed for this release compared to release ESMF 8.3.0. This is based on test runs with the Intel compilers using options "-O2 -fp-model precise".
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• All of the fixes and improvements released with patch 8.3.1 are also included in release 8.4.0. Of particular note is support for messages above the 2GiB limit by the ESMF VM communication layer.
• The Earth System Model eXecutable (ESMX) layer was added to ESMF. ESMX removes technical hurdles that impede hierarchical NUOPC model testing. It includes a unified executable capable of driving multiple configurations of model components: single component, one-way forced, and fully dynamic two-way coupled. A simple YAML file is used to list the component build dependencies (CMake based). A standard NUOPC run configuration file is used to specify the processor layout, run sequence, the field dictionary, and model component options. ESMX supports hierarchical testing by allowing decentralized testing at the component level before integration into larger coupling systems, as well as testing on the integration level.
• The concept of "Named Aliases" has been introduced. Regular aliases of deep ESMF objects continue being created using a simple assignment operator. Changing the name of a regular alias affects the object and thus all other aliases. Named aliases are created using the new ESMF_NamedAlias() function. Named aliases manage their own private name. Changing the name of a named alias does not affect the object or other aliases.
• Using a Mesh that contains one or more elements with greater than 4 sides when creating an XGrid is now supported. Previously doing so would result in an error when data was transferred into or out of the XGrid.
• LocStreams can now be created from a Cartesian Mesh file. Previously the attempt to do so would result in an error.
• Attempting to create a Grid from a GRIDSPEC mosaic file that doesn't contain variables with standard_name set to "grid_tile_spec", "geographic_latitude", or "geographic_longitude" now triggers clear ERROR log messages before returning with an error. Previously no ERROR messages were logged, making this situation hard to debug.
• Read and Write operations are now permitted for multi-tile Arrays and Fields. A common use case for this is representing a cubed sphere grid as a six-tile grid. This change applies to ESMF_ArrayRead(), ESMF_ArrayWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_FieldBundleRead(), and ESMF_FieldBundleWrite(). Each of these methods reads or writes to multiple files in the multi-tile case. See the respective API documentation for details of how the tile-specific file names are constructed.
  Current limitations are:
  • For I/O of ArrayBundles and FieldBundles, all Arrays / Fields in the bundle must contain the same number of tiles;
  • I/O is not yet permitted for multi-tile Arrays / Fields with ungridded / undistributed dimensions;
  • I/O is currently only permitted for multi-tile Arrays / Fields with 1 DE per PET.
• ESMF profiles now include a summary of the full application run time for convenience. Previously user instrumentation was required to introduce an end-to-end profiling level.
• The configuration for ESMF_COMPILER=gfortranclang now works on Darwin systems (Mac OS X). Note that on Darwin, the system-level g++ invokes clang++, and unless you ensure that you have a true g++ early in your path (or build the MPI compiler wrappers to ensure that they wrap the true g++), you will end up using clang++ even if you think you are using the GNU C++ compiler. Setting ESMF_COMPILER=gfortranclang is correct for this typical situation. Attempting to use ESMF_COMPILER=gfortran when the C++ compiler is actually clang++ now issues an error message.

• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

  Platform-specific issues:

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default, ESMF uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• On Darwin, with the GNU gfortran+gcc combination, when building MPICH3 from source, it is important to specify the "--enable-two-level-namespace" configure option. By default, i.e. without this option, on Darwin, the produced MPICH compiler wrappers include a linker flag (-flat_namespace) that causes issues with C++ exception handling. Building and linking ESMF applications with MPICH compiler wrappers that specify this linker option leads to "mysterious" application aborts during execution.
• On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting the environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
• There is an issue with intercepting the MPI calls for profiling on some of the supported platforms. This results in a single FAIL reported for ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Catania: Darwin+GNU+mpich
  • Green: Darwin+gfortranclang+mpich/openmpi
  • Gaea: Unicos+GNU+cray-mpich
• There is an issue with loading the libesmftrace_preload.so library on some of the supported platforms. This results in a reported CRASH for ESMF_TraceIOUTest.F90 and ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Discover: Linux+GNU+intelmpi
  • Gaea: Unicos+Intel+cray-mpich
  • Gaea: Unicos+Intel+mpiuni

• This patch release is backward compatible with ESMF 8.3.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.3.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The ESMF communication layer now supports single messages that are above the previous 2GiB limit. This applies to the direct usage of ESMF_VM communication calls, but also extends to ESMF_RouteHandle based communication methods: Regrid, Redist, Halo, and SMM. User applications have been observed to push over the previous 32-bit message limit when each PET addresses a substantial amount of memory and calls into the ESMF communication methods. Before this fix, a user could experience application crashes in the MPI layer due to ESMF attempting to send messages that exceed the 32-bit size limit.
• An issue inside ESMF_Info was fixed in which the data type and precision of attributes was not properly set when querying for an attribute through ESMF_InfoGet(). This led to a downstream issue of writing NetCDF attributes with a precision inconsistent with the attribute's precision inside the ESMF_Info object.
• Fixed an issue in which the IO layer was incorrectly querying for the number of compute cores on a node. In some cases, the bug led to application hangs during IO operations (e.g. ArrayWrite) that span multiple nodes.
• An issue was fixed in the IO layer that occurred when reading through PNetCDF into a destination with repeating elements (e.g. halo points). When used to read a Mesh from file, this problem caused bad coordinate values to be set in the created mesh. This in turn led to very poor performance when subsequently using the Mesh in a regrid weight generation operation.
• The IO performance and memory requirement for the Read() operation was significantly improved by eliminating a costly check only relevant for Write() operations.
• An issue observed under Darwin M1 systems with ESMP_Initailize() called by ESMPy was fixed.
• The ESMPy syntax was corrected to allow for calls with property accessor Manager.local_pet.
• The internal MOAB library included with ESMF now builds under old GCC 5.4.x.

• Same as ESMF 8.3.0.

• This release is backward compatible with the last release ESMF 8.2.0, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF 8.2.0, including the rationale and impact for each change.
• No bit-for-bit changes were observed for this release compared to release ESMF v8.2.0 with Intel compilers using "-O2 -fp-model precise".
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding.
• A new section was added to the NUOPC Reference Manual describing the use of NUOPC_AddNestedState() for the coupling of multiple nests or multiple data sets between components.
• The option to profile the execution time of each individual iteration through a NUOPC run sequence has been implemented in the Driver Component Metadata Profiling attribute. Setting the appropriate profiling bit results in a profile where the timing for each individual run sequence iteration is reported in the timing profile under a unique label. This information can be helpful for cases where the cost per iteration changes throughout the execution.
• An issue in the ESMF_StateReconcile() method used by ESMF and NUOPC to generate a consistent object view across multiple components was fixed. The optimization implemented in v8.1.0 introduced the unintended behavior of switching out geom objects (Grid, Mesh, etc.) for Fields contained in States that are used in multiple ESMF_StateReconcile() operations. The incorrect association of geom objects with Fields resulted in unexpected results during subsequent operations using those Fields, such as creating a RouteHandle for regridding.
• Progress was made in full adoption of MOAB as the internal mesh representation in ESMF. This includes updating the internal MOAB library included with ESMF to version 5.3.1 and the addition of combinatorial testing designed to ensure consistency and backward compatibility between the native mesh implementation and the MOAB-based implementation. Several consistency issues were resolved as a result of the new testing.
• The previous version of ESMF_MeshCreate() from a file used to read all the node coordinate information on every processor. For large mesh files this global read can lead to high memory consumption and prevent reading in certain large meshes entirely. To reduce the memory footprint, a fully distributed read of mesh coordinate information was implemented. This change allows the creation of much larger meshes from file.
• The nodeOwners argument for method ESMF_MeshCreate() and ESMF_MeshAddNodes() was made optional. This allows the user to defer specification of node ownership to ESMF in cases where a specific ownership assignment is not needed to match the application data distribution. When this argument is absent, ESMF generates a consistent assignment of node owners.
• The ESMF_GridCreate() method that creates a new Grid from an existing Grid with new DistGrid was extended to optionally return a RouteHandle object. The RouteHandle allows subsequent calls into the new ESMF_GridRedist() method to redistribute the coordinate values from the original source grid to the new destination grid. This feature supports efficient handling of dynamically changing grids between components.
• The implementation of the exchange grid (ESMF_XGrid) class that supports efficient conservative regridding between multiple grids on source and destination sides has been improved:
  • Element areas can now be set in a Field built on an XGrid by using the ESMF_FieldRegridGetArea() method. Previously, this capability was only supported for a Field built on a Grid or Mesh.
  • The algorithm used to generate interpolation weights was improved to guarantee that exchange grid cells are exactly overlapping exactly one cell on each side. Prior to this change, small numerical errors prevented this property from holding.
• Added the optional --checkFlag argument to ESMF_RegridWeightGen application. This flag allows the user to turn on more expensive error checking that may not be appropriate for an operational run. Initially this flag turns on a check for grid self-intersection during conservative regridding.
• The VM Epoch implementation now provides an option to reduce the memory pressure on the sending side PETs. By default, internal send buffers, once allocated, are kept until the VM is destroyed. This can lead to high memory pressure for cases where the same sending PETs participate in communication with multiple sets of receiving PETs. Setting keepAlloc=.false. when calling ESMF_VMEpochEnter(), instructs ESMF to immediately deallocate internal send buffers once the data has been transferred. This is in analogy to the handling of internal receive buffers with keepAlloc=.false. when calling ESMF_VMEpochExit(). The default remains .false. for both sides for efficiency.
• Two internal fixed size buffers that caused issues when precomputing RouteHandles (e.g. via RegridStore()) for high-resolution, high PET count cases (~10,000 and above) were modified. The size of one of the buffers was doubled, while the other fixed size limitation was removed. The symptom of the first buffer size issue (now increased in size) was an error trace in the ESMF Log starting with "ESMCI_DELayout.C:9616 ESMCI::XXE::storeBufferInfo() Internal error: Bad condition - bufferInfoList overflow!!!". The second buffer size issue (now eliminated) was an error trace starting with "ESMCI_DELayout.C:8416 ESMCI::XXE::execReady() Internal error: Bad condition - sendnbCount out of range".
• ESMF uses a library called ParallelIO (PIO) for its internal I/O operations, such as reading in mesh files and writing out fields. During this release, the version of PIO used internally was upgraded from a very outdated 1.x version to version 2.5. As a result, the binary output option ESMF_IOFMT_BIN is no longer supported and has been removed. A new option was also added to the ESMF build system to allow linking to an external build of the PIO library, as long as the external build is at least version 2.5.8. The upgrade eliminates the need to pass the compiler flags "-fallow-argument-mismatch -fallow-invalid-boz" when building with GNU 10.x or newer compilers. If the internal build of PIO is used, CMake version 2.8.12 or newer must be available in the system path. See the User's Guide for information about the environment variables used to configure PIO build and linking options.
• An extra column was added to the ESMF profiler summary output, reporting the number of PEs (CPU cores) associated with the executing PETs. This information is helpful for example when profiling components that run with ESMF-managed threading. In the single-threaded case, each PET is associated with a single PE, and the number of PEs equals that of PETs. However, for the multi-threaded case, where N threads (e.g. OpenMP) are spawned under each PET, the number of PEs will be N times the number of PETs.
• The ESMF_COMM build setting for MPICH has been reworked to better align with the current state of the MPICH project, and other ESMF_COMM settings. ESMF_COMM=mpich now covers the current MPICH versions 3 and 4. ESMF_COMM=mpich3 is still supported for backward compatibility. The old MPICH2 continues to be supported via ESMF_COMM=mpich2.
• A problem with ESMF library installation linking for dylibs under Darwin was fixed. Previously the installed ESMF library remained dependent on files under the src directory of the ESMF build tree.
• The FindESMF.cmake file included with ESMF, which is provided as a convenience to users that use CMake in their projects, has been updated. The module now searches ESMF_ROOT if ESMFMKFILE is not provided by the environment. Option USE_ESMF_STATIC_LIBS has been added to use the static ESMF library when building executables. This module requires CMake v3.12 and above.

• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

Platform-specific issues:

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default, ESMF uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• On Darwin, with the GNU gfortran+gcc combination, when building MPICH3 from source, it is important to specify the "--enable-two-level-namespace" configure option. By default, i.e. without this option, on Darwin, the produced MPICH compiler wrappers include a linker flag (-flat_namespace) that causes issues with C++ exception handling. Building and linking ESMF applications with MPICH compiler wrappers that specify this linker option leads to "mysterious" application aborts during execution.
• On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting the environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
• There is an issue with intercepting the MPI calls for profiling on some of the supported platforms. This results in a single FAIL reported for ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Catania: Darwin+GNU+MPICH3
  • Gaea: Unicos+GNU+cray-mpich
• There is an issue with loading the libesmftrace_preload.so library on some of the supported platforms. This results in a reported CRASH for ESMF_TraceIOUTest.F90 and ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Cori: Unicos+Intel+cray-mpich
  • Cori: Unicos+Intel+mpiuni
  • Discover: Linux+GNU+intelmpi
  • Gaea: Unicos+Intel+cray-mpich
  • Gaea: Unicos+Intel+mpiuni
  • Hera: Linux+GNU+intelmpi
  • Orion: Linux+GNU+mpiuni

• This release is backward compatible with the last major release update, ESMF 8.1.0 and patch release ESMF 8.1.1, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF_8_1_0, including the rationale and impact for each change.
• No bit-for-bit changes were observed for this release compared to release ESMF 8.1.0 and patch release ESMF 8.1.1, with Intel compilers using "-O2 -fp-model precise". However, the release contains code changes to the regridding implementation that have the potential to lead to bit-for-bit changes in regridding weights. Any release item with the potential to introduce a bit-for-bit change is indicated in the respective release note.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding.
• The NUOPC RunSequence syntax was extended to support Alarm Blocks. An alarm block specifies the time interval at which the elements within the block are executed. This adds additional flexibility to the RunSequence approach, e.g. to write restart files at certain intervals that are multiples of the parent timestep.
• Fields created on XGrids can now be used as either source, destination, or both when calling the general ESMF regrid methods (ESMF_FieldRegridStore(), ESMF_FieldRegrid(), ESMF_FieldBundleRegridStore(), ESMF_FieldBundleRegrid()). This enables the use of all ESMF regridding methods (bilinear, patch, etc.) and options (extrapolation, regridding status, etc.) when regridding to or from Fields on an XGrid. Prior to this release, regridding to or from Fields on an XGrid was only supported when going from one of the grids used to originally create the XGrid. Also, only conservative methods were supported.
• A change in the 3D spherical bilinear weight calculation to handle more complex cells lead to a decrease in performance in releases 8.0.0, 8.1.0, and 8.1.1. The current release restores the performance to the level of ESMF 7.1.0r, and better, while retaining support for the complex cells. (Note that this change has the potential to introduce round off level changes in weights calculated for the 3D spherical bilinear method compared to previous ESMF releases. However, bit-for-bit testing with the Intel compiler using "-O2 -fp-model precise" did not detect any changes.)
• A number of issues that were found with ESMF-managed threading under real application usage, as released with ESMF 8.1.0, have been addressed: (1) PETs that execute a threaded component are no longer instantiated as Pthreads by default but instead execute under the original MPI process. This resolves the issue of not being able to set an unlimited stack size. (2) Issues within the automatic garbage collection of ESMF objects have been resolved, which lead to memory corruption issues during ESMF_Finalize() when Grids or Meshes were transferred between threaded components. (3) Thread affinities and number of OpenMP threads are reset when exiting from a threaded component method, and global resource control can be turned on/off via the optional argument globalResourceControl during ESMF_Initialize().
• It is now possible to override the defaults of a number of global ESMF settings by specifying an ESMF_Config file during ESMF_Initialize(). This is particularly useful for adjusting log specific settings, or to turn on/off resource control on the global VM.
• A new section was added to the ESMF Reference Manual that discusses use of VMEpoch for asynchronous RouteHandle communications.
• The VMEpoch feature allows sending PETs to fill the message queue up to the limit set by the MPI implementation. For message sizes where an MPI implementation chooses to use the EAGER protocol, this can lead to memory exhaustion on the receiving PETs. To prevent this issue, VMEpoch now limits the number of outstanding send cycles to ten by default. This default can be overridden by the user through the optional argument throttle to ESMF_VMEpochEnter().
• The process of replacing the native ESMF mesh implementation with the MOAB library is continuing. The MOAB mesh backend is now available to ESMPy by calling Manager.set_moab(). This allows the user to test ESMPy regridding features with the new MOAB backend in preparation for MOAB becoming the default. Manager.moab returns a boolean value to indicate if the MOAB backend is currently in use. The default is to use the native ESMF mesh backend.

• The ESMF_XGrid construction can lead to degenerate cells in cases where the source and destination grids have edges that are almost the same. Often these cells don't produce weights and are benign, but when weights are produced, they can lead to low accuracy results when transferring data to/from the XGrid.
• Attempting to write weight files from the ESMPy Regrid object when using filemode=FileMode.WITHAUX currently crashes.

Platform-specific issues:

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default, ESMF uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• For GNU compilers GCC>=10.x, the default Fortran argument mismatch checking has become stricter. This results in build failures in some of the code that comes with ESMF. Setting environment variable ESMF_F90COMPILEOPTS="-fallow-argument-mismatch -fallow-invalid-boz", during the ESMF build, can be used as a work-around for this issue.
• On Darwin, with the GNU gfortran+gcc combination, when building MPICH3 from source, it is important to specify the "--enable-two-level-namespace" configure option. By default, i.e. without this option, on Darwin, the produced MPICH compiler wrappers include a linker flag (-flat_namespace) that causes issues with C++ exception handling. Building and linking ESMF applications with MPICH compiler wrappers that specify this linker option leads to "mysterious" application aborts during execution.
• On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting the environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
• The ESMF_ArrayIOUTest unit test fails the binary read test on the S4 test system (Linux+Intel+IntelMPI).
• There is an issue with intercepting the MPI calls for profiling on some of the supported platforms. This results in a single FAIL reported for ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Catania: Darwin+GNU+MPICH3
  • Gaea: Unicos+GNU+cray-mpich
• There is an issue with loading the libesmftrace_preload.so library on some of the supported platforms. This results in a reported CRASH for ESMF_TraceIOUTest.F90 and ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Cori: Unicos+Intel+cray-mpich
  • Cori: Unicos+Intel+mpiuni
  • Discover: Linux+GNU+intelmpi
  • Gaea: Unicos+Intel+cray-mpich
  • Gaea: Unicos+Intel+mpiuni
  • Hera: Linux+GNU+intelmpi
  • Orion: Linux+GNU+mpiuni

• This patch release is backward compatible with ESMF 8.1.0.
• No bit-for-bit changes are expected for this release compared to ESMF 8.1.0.
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• This patch release corrects a regression in the handling of user level OpenMP threads. See the Known Bugs column for details.

• Same as ESMF_8_1_0 with the following exceptions:
  • The issue with OpenMP thread count being reset to 1 within all ESMF components has been fixed.
  • The PETs of all ESMF components, and any potentially created OpenMP threads under such PETs, are no longer pinned automatically to any specific PEs.

• The same as ESMF_8_1_0.

• This release is backward compatible with the last major release update, ESMF 8.0.0, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF_8_0_0, including the rationale and impact for each change.
• Some bit-for-bit changes are expected for this release compared to release ESMF 8.0.0 and patch release ESMF 8.0.1. We observe the following impact with Intel compilers using "-O2 -fp-model precise":
  • Fixed a problem that could result in erroneously unmapped destinations when going from a very fine source grid to a coarse destination grid (e.g. 1/20 degree to 10x15 degree). Expected bit-for-bit changes:
    • ESMF_REGRIDMETHOD_CONSERVE_2ND: roundoff level changes in weight values because of a change in the order of calculation.
    • All regrid methods: Missing weights being added for very fine source grid to coarse destination grid regridding cases as this fix comes into play.
  • Fixed a problem where using the bilinear method between two identical grids doesn't result in an identity matrix for the regridding weights. It also improves the efficiency of the code when using bilinear or patch between identical grids. Expected bit-for-bit changes:
    • ESMF_REGRIDMETHOD_BILINEAR: small changes in regridding weights when a destination point exactly matches a source point.
    • ESMF_REGRIDMETHOD_PATCH: small changes in regridding weights when a destination point exactly matches a source point.
  • Fixed a problem where a set of points with latitudes set at exactly -90.0 are not all mapped to the same point. Expected bit-for-bit changes:
    • All regrid methods: Small weight changes when a point in the grid lies at exactly -90.0 latitude.
  • Optimized the creep fill so the memory doesn't increase as quickly for large numbers of creep levels. Expected bit-for-bit changes:
    • ESMF_EXTRAPMETHOD_CREEP: Small weight changes for the extrapolated destination locations.
  • Fixed an issue where in some cases creep fill weights can trigger an assertion in the code that redistributes weights to their final decomposition. Expected bit-for-bit changes:
    • ESMF_EXTRAPMETHOD_CREEP: Small weight changes for the extrapolated destination locations.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• The ESMF_Info class was introduced as a replacement for ESMF_Attribute. ESMF_Info is based on a modern C++ JSON implementation to provide efficient key-value pair storage. Most of the legacy ESMF_Attribute API is preserved for backward compatibility.
• ESMF is in the process of upgrading the internal mesh representation to use the MOAB mesh library developed by the U.S. Department of Energy. In this release, ESMF capabilities using MOAB have been significantly optimized and expanded, allowing for application-level testing of ESMF with MOAB as the underlying mesh representation. MOAB is built into the ESMF library by default, but its use must be enabled at run-time by calling ESMF_MeshSetMOAB(). When MOAB is activated, the following new capabilities are supported in this release:
  • The Mesh creation, conservative regridding, and bilinear regridding algorithms when MOAB is active have been optimized to reduce their memory use and expand the size of Meshes they can be used on.
  • Grids can now be explicitly converted to a Mesh when MOAB is active, using the ESMF_MeshCreate() method.
  • Grids can be used to do first-order conservative regridding using MOAB.
  • Grids can be used for bilinear regridding on cell centers or corners using MOAB.
• The NUOPC Layer contained in this release has been improved in the following specific technical areas:
  • The NUOPC API has been simplified through the introduction of semantic specialization labels. The new approach leads to clearer and more concise NUOPC "cap" implementations that do not require specifying an Initialize Phase Definition (IPD) version or using the IPDvXXpY nomenclature when registering methods in the SetServices() method. Existing caps do not have to be re-written or updated, although updating to the new semantic specialization labels is recommended for existing and new NUOPC caps. The older IPD version based approach is supported for backward compatibility.
  • The NUOPC layer now provides features for resource control and handling of threaded components. This mechanism supports mixing of hybrid MPI+OpenMP components with different threading levels and mixing with standard MPI components on the same processing elements (PEs), i.e. cores. It allows each component to fully utilize HPC resources independently. Coupling between threaded components is supported automatically via the standard NUOPC Connectors.
  • The external NUOPC interface that supports interaction between an entire NUOPC application and a layer outside of NUOPC (e.g. a Data Assimilation system) was further refined. The associated prototype code (ExternalDriverAPIProto) has been updated to reflect the current status.
  • The NUOPC Profiling attribute, available in the Driver Metadata, Model Metadata, Mediator Metadata, and Connector Metadata, has been re-implemented. The NUOPC layer profiling features now integrate with the ESMF profiling infrastructure.
  • The NUOPC transfer protocol for geometry objects (Grid, Mesh, LocStream) has been made more efficient. Geometries used for multiple Fields are only transferred once, reducing the initialization overhead associated with the transfer.
  • Several optimizations were implemented in the NUOPC layer to reduce overhead. All applications using NUOPC benefit from these optimizations without requiring code changes.
  • Added the creep_nrst_d value to the extrapMethod NUOPC connection options. This is equivalent to the ESMF_EXTRAPMETHOD_CREEP_NRST_D option in ESMF_FieldRegridStore() discussed below.
• Added the extrapolation option: CREEP_FILL to ESMPy. This option fills unmapped destination points by repeatedly moving data from mapped locations to neighboring unmapped locations.
• The implementation of the ESMF_StateReconcile() method was redesigned to improve performance, and scalability. Most users do not interact with this method directly, however, the NUOPC initialization time has been reduced as a consequence, which is most pronounced in applications with large Field and PET counts.
• Added a new extrapolation method called "nearest destination" to the regrid weight generation system. This capability fills destination points that were not filled by an initial regridding by using the nearest regridded destination point. The nearest destination method is accessible by specifying the extrapMethod=ESMF_EXTRAPMETHOD_NEAREST_D option in any of the ESMF_*RegridStore() methods or the --extrap_method nearestd option when using the ESMF_RegridWeightGen application).
• Added a new extrapolation method called "creep nearest destination" to the regrid weight generation system. This capability fills destination points that were not filled by an initial regridding by first applying a creep fill extrapolation and then filling the remaining unmapped destination points using nearest destination extrapolation. The creep nearest destination method is accessible by specifying the extrapMethod=ESMF_EXTRAPMETHOD_CREEP_NRST_D option in any of the ESMF_*RegridStore() methods or the --extrap_method creepnrstd option when using the ESMF_RegridWeightGen application).
• The creep fill extrapolation has been optimized so that it uses less memory per level extrapolated. This allows the creep fill method to extrapolate several times further into unmapped parts of the destination Field.
• Added the optional checkFlag argument to ESMF_FieldRegridStore(). The intention behind this flag is to allow the user to turn on more expensive error checking that may not be appropriate for an operational run. Initially this flag turns on a check for grid self-intersection during conservative regridding.
• The ESMF_MeshGet() call has been expanded to allow the user to query a full set of information for most Meshes. The exception is 2D Meshes with cells of more than four sides for which the element information (e.g. element connectivity) is not yet available.
• A new shared memory feature was introduced that allows sharing of decomposition elements (DEs) between PETs that execute on the same single system image (SSI), i.e. node. This feature provides an efficient way to access data by reference between components that are running on the same PEs (cores), but with different threading levels. Both Field-level DE sharing and Array-level DE sharing are supported.
• Key-value pair storage was added to the ESMF_Mesh and ESMF_LocStream classes through the overloaded ESMF_InfoGetFromHost() method.
• A section discussing "Debugging of ESMF User Applications" has been added to the User's Guide. This section is designed to help users interpret error traces and efficiently locate issues in failing ESMF applications.

• FieldBundles don't currently enforce that every contained Field is built on the same Grid, Mesh, LocStream, or XGrid object, although the documentation says that this should be so.
• The packed FieldBundle implementation uses a concatenated string to create a base object. When this string has more than 255 characters, e.g. a large number of Fields with long individual names is packed, the base object is not created correctly resulting in incorrect behavior at the FieldBundle level.
• When the ESMF regrid weight generation methods and applications are used with nearest destination to source interpolation method, the unmapped destination point detection does not work. Even if the option is set to return an error for unmapped destination points (the default) no error will be returned.
• The ESMF regrid weight generation methods and applications do not currently work for source Fields created on Grids which contain a DE of width less than 2 elements. For conservative regridding the destination Field also has this restriction.
• The ESMF regrid weight generation methods and applications do not currently work on Fields created on Grids with arbitrary distribution.
• When using the ESMF_RegridWeightGen application to generate conservative weights for a Mesh with > 16 million cells, the weight file produced has some of its factors scrambled in a subtle way. This can lead to higher than usual conservation error (>1.0E-7).
• When pole extrapolation is used during regridding operations and a quadrilateral cell that degenerates into a triangle is in the top or bottom row of the grid, the error "Condition {etopo->num_nodes == 4} failed..." is incorrectly returned.
• The ESMF_GridCreate() interface that allows the user to create a copy of an existing Grid with a new distribution will give incorrect results when used on a Grid with 3 or more dimensions and whose coordinate arrays are less than the full dimension of the Grid (i.e. it contains factorized coordinates).
• The ESMF_XGrid construction can lead to degenerate cells for cases where the source and destination grids have edges that are almost the same. Often these cells don't produce weights and are benign, but when weights are produced can lead to low accuracy results when transferring data to/from the XGrid.
• When any of the Redist() methods are used inside a VMEpoch, the execution crashes with an MPI error.
• The OpenMP thread count is being reset to 1 within all ESMF components. This affects user code that leverages OpenMP threading inside of components, and uses the OMP_NUM_THREADS environment variable to set the desired number of OpenMP threads. As a consequence the expected speed up from OpenMP threading in user code will not be present.
• The PETs of all ESMF components, and any potentially created OpenMP threads under such PETs, are pinned to the PE on the respective shared memory node, corresponding to the PET number. As a consequence, even if a user overcomes the OpenMP thread count reset to 1 bug, e.g. by using omp_set_num_threads() API directly, the performance of OpenMP threaded user code is far below that of the expected speed up.

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default ESMF now uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• For GNU compilers GCC>=10.x, the default Fortran argument mismatch checking has become stricter. This results in build failures in some of the code that comes with ESMF. Setting environment variable ESMF_F90COMPILEOPTS="-fallow-argument-mismatch -fallow-invalid-boz", during the ESMF build, can be used as a work around for this issue.
• On Darwin, with the GNU gfortran+gcc combination, when building MPICH3 from source, it is important to specify the "--enable-two-level-namespace" configure option. By default, i.e. without this option, on Darwin, the produced MPICH compiler wrappers include a linker flag (-flat_namespace) that causes issues with C++ exception handling. Building and linking ESMF applications with MPICH compiler wrappers that specify this linker option leads to "mysterious" application aborts during execution.
• On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
• Currently the ESMPy interface to retrieve regridding weights from Python is only supported under the GNU compiler. On all other compilers the method will flag an error.
• There is an issue with intercepting the MPI calls for profiling on some of the supported platforms. This results in a single FAIL reported for ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Catania: Darwin+GNU+MPICH3
  • Gaea: Unicos+GNU+cray-mpich
• There is an issue with loading the libesmftrace_preload.so library on some of the supported platforms. This results in a reported CRASH for ESMF_TraceIOUTest.F90 and ESMF_TraceMPIUTest.F90. The affected platforms are:
  • Discover: Linux+GNU+intelmpi
  • Gaea: Unicos+Intel+cray-mpich
  • Gaea: Unicos+Intel+mpiuni
  • Orion: Linux+GNU+mpiuni

• This patch release is backward compatible with ESMF 8.0.0. Two new interfaces were added to the Fortran API: ESMF_VMEpochEnter() and ESMF_VMEpochExit().
• No bit-for-bit changes are expected for this release compared to ESMF 8.0.0. This has been verified for a large number of regridding tests with Intel compilers using flags "-O2 -fp-model precise".
• No changes were made to the ESMF regrid weight generation methods and applications. The ESMF tables summarizing the ESMF regridding status are unchanged.
• The ESMF Virtual Machine (ESMF_VM) now supports message aggregation to improve performance for some very common communication patterns. Two new methods, ESMF_VMEpochEnter() and ESMF_VMEpochExit(), allow explicit use of this feature.
• The generic NUOPC_Connector automatically takes advantage of VMEpoch message aggregation when used between components on disjoint petLists.
• The ability to write/read ESMF RouteHandles to/from file was added to the ESMPy layer. This allows a user to perform the costly RouteHandle generation once, and re-use it in subsequent runs. This provides a more efficient approach to Regridding in the situation where the number of PETs does not change between runs.
• The pole_kind parameter was added to allow specification of pole behavior when creating an ESMPy Grid.

• Same as ESMF_8_0_0 with the following exceptions:
  • The race condition in the ESMF_FileRegrid() method and ESMF_Regrid application has been fixed. Now if the destination grid is a multi-tile grid in GRIDSPEC MOSAIC format and the tile is distributed into multiple PETs, the regridded field is written correctly into the output file.
  • Applying the sparse matrix multiplication to cases where the local data allocation is above the 32-bit limit now works reliably.
  • The ESMF_GridCreate1PeriDim() method can now be used to create a bipole connection on the lower side.
  • The ESMF_ArrayCreate() method now succeeds when called from inside a component, requesting DE sharing.
  • The ESMF_DistGridGet() method now correctly returns "de" and "tile" information for a "localDe".
  • ESMF now correctly links against the Fortran bindings of NetCDF, even when the C and Fortran bindings of NetCDF are provided in different locations.
  • The ESMF install target now reliably works when installing ESMF into a location that is shared with other library installations.

• The same as ESMF_8_0_0 with the following exceptions:
• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default ESMF now uses the C++11 standard and cannot be downgraded. If you run into build issues due to the C++11 dependency, you must make sure a GCC>=4.8 is loaded.
• The Darwin.intelclang.default build configuration now works correctly.

• This release is backward compatible with the last release, ESMF 7.1.0r, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to a few unmarked methods that may require minor modifications to user code that uses these methods. A number of new interfaces were added. The entire list of API changes is summarized in a table showing interface changes since ESMF_7_1_0r, including the rationale and impact for each change.
• Some bit-for-bit changes are expected for this release compared to the last release, ESMF 7.1.0r. We observe the following impact with Intel compilers using "-O2 -fp-model precise":
  • Roundoff level differences in conservative regridding due to an improvement in an area calculation algorithm
  • Roundoff level differences in regridding when used on a Mesh created from a SCRIP format file that contains longitudes <=0 degrees. This change was due to removing a conversion for non-positive longitudes to improve consistency
  • Minor differences in 2nd order conservative regridding for cells that protrude outside their neighbors (e.g. a peninsula made up of a single cell) due to a bug fix in the weight calculation algorithm for that regridding method
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• Added an option to output component timing profiles in text format by setting the ESMF_RUNTIME_PROFILE environment variable. ESMF and NUOPC component phases are automatically instrumented and user-defined timed regions are also supported. Timing profiles can be written to the end of the ESMF PET log files, to separate per-PET text files, and/or to a single timing summary file. The summary file provides timing statistics across all the PETs. This provides a simple way to understand the relative cost (in terms of wall clock time) of each component in a coupled application.
• The NUOPC Layer contained in this release has been improved in the following specific technical areas:
  • The NUOPC Field Dictionary can now be ingested from a community-based YAML file which resides in a dedicated public repository and can evolve independently of the NUOPC Layer. YAML Ain't Markup Language (YAML) is a human friendly, Unicode-based data serialization language for all programming languages.
  • A NUOPC_Driver can now be called from a higher level driver outside of NUOPC, going through an ESMF API. This is useful for systems that come with their own driver layer, but need to drive a NUOPC system (e.g. Data Assimilation).
  • The NUOPC run sequences now supports the "*" wildcard character in the "@" timestep syntax. This allows the timestep length to be set in code via driver specialization for run sequences ingested from a text file.
  • NUOPC now allows switching between different run sequence sections during execution. For example, this capability allows changing which components are active at different stages during a run. To enable this, the NUOPC_DriverIngestRunSequence() method now supports specification of a run duration, run sequence concatenation, and component run sequence elements outside loops.
  • ConnectionOptions is now an official NUOPC level Connector attribute. The attribute is read during RunSequence ingestion (for each Connector line), and appended by default to all of the CplList entries of the Connector.
  • The NUOPC_DriverAddComp() method now supports adding components with a SetVM() method, allowing the component to configure its own VM, e.g. for PET idling for PE reuse under threaded PETs.
  • Fields that are mirrored now arrive on the acceptor side with attributes set to reflect information about the provider side Field (TypeKind, GeomLoc, MinIndex, MaxIndex, ArbDimCount, GridToFieldMap, UngriddedLBound, UngriddedUBound). These attributes can then be used when creating Fields on the acceptor side.
  • The NUOPC Mesh transfer protocol was extended to correctly transfer either the node DistGrid, the element DistGrid, or both if present on the provider Mesh.
  • Completed the implementation of the sharing protocol. Whether to share the GeomObject and/or Field are now independent decisions, and all four possible combinations are supported. Sharing is available through component hierarchies and supports nested states (e.g. to use the Namespace and/or CplSet features of NUOPC).
  • The sharing protocol now checks whether the combination of provider and acceptor VMs allows Fields and/or GeomObjects to be shared between them and properly sets the share status attributes.
  • The generic Connector was optimized to also reuse Redist RouteHandles for Fields built on Meshes and LocStreams. Previously only the Redist for Fields built on Grids was optimized in this manner. Interleaving Redist and Regrid Field pairs is supported as before.
  • Implemented a communication optimized approach to propagating Field timestamps during Connector Run() to remove unnecessary synchronization between PETs. Only real data dependencies remain.
  • The NUOPC timestamp definition has been extended to include the ESMF calendar kind for more accurate timestamp validation between components.
  • Improved, standardized, and documented Verbosity attribute handling across all NUOPC generic component kinds. Preset Verbosity levels are available: "off", "low", "high", and "max".
  • The Diagnostic attribute option has been implemented across all NUOPC generic component kinds. This attribute allows the user to specify when Fields contained in the importState and exportState of a component are dumped to file when entering and exiting component methods (Initialize/Run/Finalize).
  • Several new NUOPC prototype (example) codes have been added:
    • AtmOcnCplSetProto demonstrates the use of the CplSet feature to support coupling multiple independent sets of Fields between components (e.g., multi-domain coupling).
    • AtmOcnLogNoneProto demonstrates the option to turn off ESMF PET log files completely.
    • AtmOcnMirrorFieldsProto demonstrates the NUOPC Field mirroring protocol.
    • CustomFieldDictionaryProto demonstrates the use of an external YAML file to populate the NUOPC Field Dictionary.
    • ExternalDriverAPIProto demonstrates how an external layer can drive a NUOPC driver component going through ESMF.
    • SingleModelOpenMPProto demonstrates the use of OpenMP threading inside a NUOPC model component and resource allocation through the generic SetVM method.
• The following capabilities were added to the ESMF Python interface (ESMPy):
  • Added an in-memory weight generation option to the Regrid class, allowing re-use of weight vectors without writing them to netCDF. The weight arrays can be returned as NumPy objects or Python dictionary of weight vectors. This allows retrieval of the weights by source and destination key/value pairs.
• The ESMF_Regrid application now supports additional options, and one option was removed:
  • Added --srcdatafile, --dstdatafile, and --tilefile_path options to support the GRIDSPEC Mosaic file format.
  • Added the --dst_loc option to support regridding on different stagger locations when the destination grid is in UGRID format and the regridding method is non-conservative.
  • Added the --check option to check the regridding results using a synthetic field, generated by an analytic function.
  • Removed the --user_areas option because none of the currently supported file formats provide user areas.
• Added new features to the ESMF_FileRegrid() method and the ESMF_Regrid application:
  • Regridding of multiple variables
  • Multi-tile GRIDSPEC Mosaic file format with data stored in separate files, one per tile
  • 2nd order conservative regridding
  • Regridding to the corner stagger location if the regridding method is non-conservative and the destination file is in UGRID format
• Added new features to the ESMF_RegridWeightGen application:
  • 1D network topology support in UGRID format
  • 2D Cartesian grid in CF Single Tile file format
  • Creep fill added as an extrapolation method
• Added a new extrapolation method called "creep fill" to the entire regrid weight generation system. This capability allows the user to spread data from mapped destination points to neighboring unmapped destination points. This action can be repeated for a number of levels where at each level the data is spread from filled to neighboring unfilled points. The creep fill method is accessible by specifying the extrapMethod=ESMF_EXTRAPMETHOD_CREEP option in any of the ESMF_*RegridStore() methods or the --extrap_method creep option when using the ESMF_RegridWeightGen application).
• Additional regridding methods are now supported when ESMF_Mesh is switched to be based on DOE's MOAB mesh library. The new supported regrid methods are: bilinear (when the source Field is not built on Grid), and nearest neighbor. All extrapolation methods, except for "creep fill" are supported.
• RouteHandles can now be written to file via ESMF_RouteHandleWrite() and read back into memory (on the same number of PETs) via ESMF_RouteHandleCreate(). This provides an opportunity to lower the initialization cost e.g. for short production runs that repeatedly require regridding between the same grids.
• The ESMF_GridCreateCubedSphere() and ESMF_GridCreateMosaic() methods now support irregular decompositions.
• The ESMF_GridCreateCubedSphere() method can now apply the Schmidt transformation on the coordinates.
• The ESMF_GridCreateMosaic() and the ESMF_GridCreate() method that reads a grid from file now support different coordinate typekinds.
• The ESMF_GridCreate() method now automatically determines the correct file format if it was not explicitly specified as an argument.
• The ESMF_GridCreate() method now supports grid creation from a GRIDSPEC Mosaic supergrid tile file.
• The ESMF_LocStreamCreate() method now supports 1D network topology in UGRID format.
• Added the capability to create an unstructured Mesh object from a structured Grid object. This allows components that internally work with a Mesh (e.g. a generic Mesh based mediator) to construct a Mesh from a transferred Grid.
• The ESMF_MeshGet() method has been extended to allow the user to get mask and area information from an unstructured Mesh object.
• Wrapped the ESMF_XGrid methods with ESMC bindings, to make them easily available to applications implemented in C.
• Added a new ESMF_FieldBundleCreate() method that allows the creation of a packed FieldBundle. This is an initial capability with some limitations that will be addressed in a future release. The method takes a pre-allocated Fortran array pointer containing the memory of a set of interleaved fields. This is often how sets of fields are structured in model code and this capability allows ESMF to reference this memory directly. Interleaving along any dimension is supported. Packed FieldBundles support communication methods including redistribution, sparse matrix multiplication, and regridding. Currently the number of fields on source and destination must be the same, and permutations of fields are not supported. This means that the order of fields on source and destination must agree.
• Implemented the ESMF_DECOMP_SYMMEDGEMAX option for cases where the number of elements is not evenly divisible by the number of decomposition elements. This option assigns the largest number of elements to the two edge DEs. It then progresses by assigning a descending number of elements to DEs as the center of the decomposition is approached from both sides. This produces a decomposition that is identical to the decomposition the FV3 model uses for this situation.
• The ESMF_Array class now allows sharing of DEs between PETs that run on the same single system image (SSI). This feature can be used for shared memory data access between components running on different number of PETs (e.g. for threading) but are located on the same SSI.
• The ESMF_Config class API has been extended. The ESMF_ConfigCreate() method now supports creating a Config object from a subsection of an existing Config object. This feature allows the consolidation of the content of multiple Config objects into a single configuration file.
• The YAML-CPP parser has been included in the ESMF distribution. It is active by default, but can be turned off by setting ESMF_YAMLCPP=OFF.
• Added support for C++11 standard. ESMF builds with the compiler default, but switches to C++11 if ESMF_YAMLCPP is enabled (default). The new ESMF_CXXSTD environment variable can be used to explicitly switch to specific C++ standards.
• Simplified the use of CMake for projects using ESMF by providing a FindESMF.cmake file. This file is located under the new cmake subdirectory. It parses the esmf.mk file of an ESMF installation, exposing ESMF build variables as global variables accessible by other CMake modules.
• The ESMF regression test suite can now be used to validate a pre-installed ESMF installation. This can be done by setting the ESMF_TESTESMFMKFILE environment variable to ON, and pointing ESMFMKFILE to the esmf.mk file of the ESMF installation to be validated.
• Added an external demo ESMF_RegridCheck to test the ESMF_Regrid application with a set of test grid and data sets. In all the test cases, the input variables were constructed using an analytic function and the regridded destination variables were compared with that function to calculate the mean relative errors.

• FieldBundles don't currently enforce that every contained Field is built on the same Grid, Mesh, LocStream, or XGrid object, although the documentation says that this should be so.
• When the ESMF regrid weight generation methods and applications are used with nearest destination to source interpolation method, the unmapped destination point detection does not work. Even if the option is set to return an error for unmapped destination points (the default) no error will be returned.
• The ESMF regrid weight generation methods and applications do not currently work for source Fields created on Grids which contain a DE of width less than 2 elements. For conservative regridding the destination Field also has this restriction.
• The ESMF regrid weight generation methods and applications do not currently work on Fields created on Grids with arbitrary distribution.
• There is a race condition in the ESMF_FileRegrid() method and the ESMF_Regrid application when the destination grid is of GRIDSPEC Mosaic format and >=12 PETs are used. This issue leads to intermittent failures in the external_demos tests for the GRIDSPEC_1x1_time_to_C48_mosaic_bilinear case when run on the 16 PET configuration.
• When regridding from a very fine source grid (e.g. 1/20th degree) to a fairly coarse grid (e.g. 15 degree) using conservative regridding, source and destination fractions can be erroneously less than they should be.
• When using creep fill extrapolation sometimes the extrapolation will not cross from a piece of a Field on one PET to a piece on another. This can lead to places in the Field not being properly extrapolated to.
• The exact identity matrix is not returned when using bilinear regridding from a grid that extends all the way to the poles, to an identical grid.
• The regridding can hang when doing bilinear or patch regridding on the centers of a mesh and the mesh contains a cell whose center is at the exact same position as one of the corners of the cell.
• When using the ESMF_RegridWeightGen application to generate conservative weights for a Mesh with > 16 million cells, the weight file produced has some of its factors scrambled in a subtle way. This can lead to higher than usual conservation error (>1.0E-7).
• When pole extrapolation is used during regridding operations and a quadrilateral cell that degenerates into a triangle is in the top or bottom row of the grid, the error "Condition {etopo->num_nodes == 4} failed..." is incorrectly returned.
• Applying the sparse matrix multiplication to cases where the local data allocation is above the 32-bit limit will fail with a memory allocation error. This affects all Regrid(), Redist(), Halo(), and SMM() calls.
• The ESMF_GridCreate() interface that allows the user to create a copy of an existing Grid with a new distribution will give incorrect results when used on a Grid with 3 or more dimensions and whose coordinate arrays are less than the full dimension of the Grid (i.e. it contains factorized coordinates).
• Using the ESMF_GridCreate1PeriDim() method to create a grid with a bipole connection on the lower side (typically referring to the southern hemisphere) resulted in no connection there.
• The ESMF_XGrid construction can lead to degenerate cells for cases where the source and destination grids have edges that are almost the same. Often these cells don't produce weights and are benign, but when weights are produced can lead to low accuracy results when transferring data to/from the XGrid.
• The ESMF_ArrayCreate() crashes when used with pinflag=ESMF_PIN_DE_TO_SSI or pinflag=ESMF_PIN_DE_TO_SSI_CONTIG from within a component. The crash is from inside MPI with "invalid communicator". The "pinflag" option works correctly from the application level, i.e. in the context of the global VM.
• Querying the ESMF_DistGridGet() method for "de" or "tile" information for a "localDe" will return incorrect results, and/or crash.
• ESMF_AttributeWrite() has only been verified to work for ESMF standard Attribute packages. Non-standard Attribute packages may trigger a crash inside the ESMF_AttributeWrite() implementation.
• For NetCDF installations that have the C and Fortran bindings installed in different locations, a NetCDF enabled build of ESMF does not correctly include the Fortran NetCDF library during linking.
• When installing ESMF into a location that is shared with other libraries, it can happen that executing the ESMF install target fails with a "permission denied" error.

• The GNU and Intel compilers require GCC>=4.8 for C++11 support (Intel uses the GCC headers). By default ESMF now uses the C++11 standard. If you run into build issues due to the C++11 dependency, you can either (1) make sure a GCC>=4.8 is loaded, or (2) set ESMF_YAMLCPP=OFF. In the latter case the YAML-dependent features in ESMF will not be available.
• For GNU compilers GCC>=10.x, the default Fortran argument mismatch checking has become stricter. This will result in build failures. Setting environment variable ESMF_F90COMPILEOPTS="-fallow-argument-mismatch -fallow-invalid-boz", during the ESMF build, can be used as a work around for this issue.
• On some systems with the PGI compiler, there is an issue with shared memory pointers between PETs on the same SSI. We see failures or crashes for Array tests that exercise this feature (ESMF_ArraySharedDeSSISTest.F90, ESMF_ArrayCreateGetUTest.F90) on the following platforms:
  • Hera/PGI-18.10.1
  • Gaea/PGI-16.5.0
  • Electra/PGI-17.1.0
  • Pleiades/PGI-17.1.0
  • Summitdev/PGI-19.7.0
  However, we do not observe these failures or crashes on:
  • Discover/PGI-14.1.0
  • Discover/PGI-17.7.0
• On Summitdev/PGI-19.7.0 we see ESMF_XGridUTest.F90 unit test failures due to erroneously produce weights for source and destination grids that have edges that are almost the same.
• On Discover/PGI-14.1.0 the ESMF_FieldRegridUTest.F90 and ESMF_FieldBundleCrGetUTest.F90 unit tests are failing.
• Currently the ESMPy interface to retrieve regridding weights from Python is only supported under the GNU compiler. On all other compilers the method will flag an error.
• On Darwin, with the Intel Fortran compiler, command line arguments cannot be accessed from ESMF applications when linked against the shared library version of libesmf. There is no issue when linked against the static libesmf.a version. Setting environment variable ESMF_SHARED_LIB_BUILD=OFF, during the ESMF build, can be used as a work around for this issue.
• On some systems with the Cray compiler (CCE version 8.x), the ESMF library fails to build. The error can be prevented by setting the ESMF build environment variable ESMF_MOAB=OFF.
• The Darwin.intelclang.default build configuration is broken.

• This release is backward compatible with ESMF 7.0.0. There are no API changes.
• The ESMF regridding status tables have not changed since 7.0.0.

• The same as ESMF_7_0_1.

• Same as ESMF_7_0_1 with the following exceptions:
  • All platforms using GNU g++: The library build failure with GNU g++ 6.1 (and newer) has been corrected. The default option of ESMF_MOAB=internal can safely be used.
  • Linux with intel (Theia): The problem of applications built with optimization level -O2 or higher, and linked against ESMF using the linker flags provided by esmf.mk, failing after the October system upgrade has been mitigated. The linker flags provided by esmf.mk no longer explicitly contain the -lsvml option.

• This release is backward compatible with ESMF 7.0.0. There are no API changes.
• The ESMF regridding status tables have not changed since 7.0.0.

• Same as ESMF_7_0_0 with the following exceptions:
  • The overestimation of the internal buffer requirement in ESMF_StateReconcile() was fixed. It is now possible to run beyond the 10,000 core limit that previously caused ESMF_StateReconcile() to run out of memory on typical configurations.

• The same as ESMF_7_0_0.

• This release is backward compatible with the last public release, ESMF 6.3.0rp1, for all the interfaces that are marked as backward compatible in the Reference Manual. There were API changes to some of the unmarked methods that may require modifications to user code that uses these methods. The entire list of API changes is summarized in a table showing interface changes since ESMF_6_3_0rp1, including the rationale and impact for each change.
• Tables summarizing the ESMF regridding status have been updated to include the new capability to perform regridding to/from a set of disconnected points (i.e. an ESMF LocStream) and to perform bilinear regridding on 3D spherical grids. These tables include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• The NUOPC Layer contained in this release is a refinement of the previously released reference API. One of the design goals was to provide a more object oriented interface that clearly reflects the NUOPC concepts of Driver, Model, Connector, Mediator. The following specific technical areas were also improved:
  • The NUOPC Driver now allows child components to be dynamically added without prior setting of the total number of components in the driver. This flexibility simplifies coding of highly configurable coupled systems.
  • The consistent use of labels to reference components and their initialize/run/finalize phases simplifies interaction with the NUOPC Layer and makes application code more readable and less error prone.
  • The introduction of namespaces provides more control over the automatic field pairing implemented by the NUOPC Connector.
  • The NUOPC Compliance Checker has become more context aware and as a consequence produces more usable output.
  • The transfer protocol of Geom objects (e.g. ESMF Grids and Meshes) between components during initialization has been extended to cover ESMF LocStreams (a geometric type that can be used to represent a set of disconnected points).
• The following capabilities were added to the ESMF Python interface (ESMPy):
  • Added ability to represent tripole Grids.
  • Added ability to create Fields on ungridded dimensions of a Grid or Mesh.
  • Added Field.read() which allows a Field to be filled with data from a CF-compliant NetCDF file.
  • Added LocStream class to represent observational data.
  • Added support for numpy array slicing for Grid, Mesh, LocStream, and Field (extraction of certain array elements into an alias of the original array).
  • Added support for Mesh cells with more than 4 sides.
  • Added public <object>.destroy() functions to provide explicit control over memory usage of persistent Python objects.
  • Added norm_type flag to allow the destination fraction to be included when calculating conservative regridding weights.
  • Added ignore_degenerate flag to allow users to ignore degenerate cells in Grids and Meshes.
  • Added coord_sys flag to Mesh.create() to allow switching between Cartesian and spherical coordinate systems.
  • Added read-only properties for all object attributes.
  • Added copy() for Grid, Mesh, LocStream, and Field to produce a shallow copy of the object.
  • Added ability to return Mesh element coordinates.
  • Added Anaconda packages for easy installation of ESMPy on Linux and Darwin. Specific package availability is recorded in a table of NESII Anaconda packages.
• Added a new application ESMF_Regrid to support file-based regridding. This application regrids a data field defined in a grid file to another field defined in a different grid file. The regridded field will be written into the destination grid file. The grid files have to be in either UGRID or GRIDSPEC format and are limited to 2D grids.
• ESMF LocStreams can now be used in regridding. This is useful for operations involving observational data, for example in data assimilation.
• ESMF now offers the option to build the unstructured Mesh class on DOE's MOAB mesh library as an alternative to its native implementation. Meshes built on MOAB support a limited number of operations. They can be created, destroyed, queried for information, and used in the creation of an ESMF Field. The standard ESMF Field operations are expected to work on a Field build on a MOAB Mesh, however testing of these has been limited. In the case of ESMF regridding only the first order conservative regridding method is supported.
• Added the ability for an element in an ESMF unstructured Mesh to consist of more than one polygon. This allows the representation of geographically disconnected areas (e.g. Hawaiian islands) as a single element. Meshes containing elements of this type will also work in the ESMF regrid weight generation methods and applications.
• Added a new algorithm to compute bilinear interpolation weights in 3D. To use this algorithm set lineType=ESMF_LINETYPE_GREAT_CIRCLE when calling into the ESMF regrid weight generation methods and applications. Weights are then calculated along lines that follow the curve of a sphere instead of straight lines. This significantly reduces the error in 3D spherical grids with highly attenuated cells.
• Expanded ESMF's first-order conservative interpolation algorithm to work on quadrilateral concave cells. Previously an error was returned under this condition.
• Removed a restriction in the unstructured Mesh class so that now element coordinates can be set and retrieved for elements with more than four sides. Previously an error was returned under this condition.
• Removed a restriction in the ESMF regrid weight generation methods and applications so that now non-conservative regridding can be done on Mesh elements with more than four sides. Previously an error was returned under this condition.
• Extended the ESMF_RegridWeightGen application and the ESMF_MeshCreate() method to support 3D unstructured grid definitions provided in the UGRID file format.
• The ESMF_RegridWeightGen application now supports additional options:
  • Added --src_loc and --dst_loc options to allow non-conservative regridding on the cell center locations of an unstructured grid defined in the UGRID or the ESMF unstructured file format.
  • Added --ignore_degenerate to ignore degenerate cells in either the source or the destination grid. This can be useful when cells that collapse into a point or line aren't a concern (e.g. when they are located in a region not used in regridding).
  • Added --line_type or -l to allow users to choose between great circle or Cartesian paths for the line between two points on a sphere during the calculation of bilinear interpolation weights.
• Added two new shortcut Grid create methods: ESMF_GridCreate1PeriDimUfrm() and ESMF_GridCreateNoPeriDimUfrm(). Each of these methods creates a Grid and fills the coordinate arrays with uniformly spaced coordinate values.
• Added a new interface to create an ESMF LocStream from a grid file. Currently, SCRIP, UGRID and ESMF unstructured file formats are supported. For a grid in either UGRID or ESMF format, the center coordinates or the corner coordinates can be chosen to construct the LocStream. For a grid in SCRIP format, the center coordinates will be used to construct the LocStream.
• Expanded the ESMF_MeshCreate() method to support creation of a dual Mesh using the center coordinates defined in a UGRID or ESMF unstructured grid file.
• The ESMF I/O methods that depend on the Parallel I/O (PIO) library are now turned on by default on platforms where certain Fortran 2003 features are supported (compilers not supported are Absoft and Lahey). The affected methods are ESMF_FieldBundleRead(), ESMF_FieldBundleWrite(), ESMF_FieldRead(), ESMF_FieldWrite(), ESMF_ArrayBundleRead(), ESMF_ArrayBundleWrite(), and ESMF_ArrayRead(), ESMF_ArrayWrite(). On platforms that do not fulfill the above system requirements, the PIO dependent methods will return an error when invoked.
• Extended FieldBundle, Field, ArrayBundle, and Array I/O to support multiple DEs per PET and undistributed dimensions.
• Extended the Log class to allow closing of a Log object (including the default Log) and reopening it under different name.
• All of the distributed data classes (FieldBundle, Field, ArrayBundle, Array, Grid, DistGrid, DELayout) now offer the "noGarbage" option in their Destroy() method. This allows the user to deallocate all memory associated with an object, even the reference held by the ESMF garbage collection. Use of this new option has shown to be necessary to prevent out-of-memory conditions in applications that use many temporary objects (e.g. create new temporary objects each time step).
• The ESMF VM class is now accelerator aware. The user can query the VM for information about available accelerator hardware. This feature is implemented on top of four different lower level software stacks: OpenCL, OpenACC, Intel MIC, and OpenMP (version 4). The appropriate software stack must be chosen when building the ESMF library by setting the ESMF_ACC_SOFTWARE_STACK environment variable in order to activate this new feature. See the User's Guide for details.
• Added functionality to retrieve names of all ESMF deep objects using ESMF_AttributeGet(). This is the next step in implementing full access to object information through the ESMF Attribute interface.
• Added the attnestflag argument to ESMF_AttributeGet() to allow users to specify whether they want to search through the entire Attribute hierarchy when retrieving an Attribute.

• When transferring Mesh objects between components using the NUOPC_Connector, the ungridded dimensions of the Field object are ignored. This leads to errors in the NUOPC_Connector phase that precomputes the data transfer for the Field.
• Reference sharing requests are ignored by the NUOPC_Connector for Fields with ungridded dimensions. The result is the use of the default copy behavior instead.
• When a Mesh is transferred between components, e.g. via the NUOPC_Connector, where the destination component's PET list is not simply a truncated version of the source component's PET list, then the resulting Mesh will have internal inconsistencies. A Regrid or a second Mesh transfer operation involving this Mesh will result in an error.
• The ESMF_StateReconcile() call overestimates the size requirement of an internal temporary buffer. The estimation error increases with core count and causes ESMF_StateReconcile() to run out of memory for large core counts (~10,000) on typical configurations.
• FieldBundles don't currently enforce that every contained Field is built on the same Grid, Mesh, LocStream, or XGrid object, although the documentation says that this should be so.
• The ESMF_FieldCompleteEmpty() function does not error out when trying to construct a Field with incorrect ungridded dimensions. Instead ESMF_SUCCESS, and an internally inconsistent Field object is returned.
• When the ESMF regrid weight generation methods and applications are used with nearest destination to source interpolation method, the unmapped destination point detection does not work. Even if the option is set to return an error for unmapped destination points (the default) no error will be returned.
• The ESMF regrid weight generation methods and applications do not currently work for source Fields created on Grids which contain a DE of width less than 2 elements. For conservative regridding the destination Field also has this restriction.
• The ESMF regrid weight generation methods and applications do not currently work on Fields created on Grids with arbitrary distribution.
• When the ESMF regrid weight generation methods and applications are used to do conservative interpolation from a concave quadrilateral source cell to a destination cell which only partially overlaps the source cell, a weight of infinity can sometimes be generated.
• When the ESMF regrid weight generation methods and applications are used for bilinear regridding with an extremely coarse spherical source grid (e.g. 5 cells to cover the entire Earth), then destination points can be mapped to the wrong side of the sphere leading to incorrect weights.
• ESMF_FieldRegridStore() that takes XGrid as an argument does not support regridding between Fields that are constructed on Grids for which ESMF_GridMatch() locally returns a value of ESMF_GRIDMATCH_EXACT or greater. This may result in unexpected local errors.
• Using first-order conservative regridding through the XGrid does not ensure bit-for-bit results when the PET count changes.
• The ESMF_Regrid application and the ESMF_FileRegrid() method abort with an error when the destination file is in NetCDF4 format and the destination variable does not exist.
• The destination field written by the ESMF_Regrid application and the ESMF_FileRegrid() method is incorrect when the field contains one or more undistributed dimension.
• The ESMF_Regrid application and the ESMF_FileRegrid() method does not set the destination field's initial value to the _FillValue of the source field when the destination field does not exist in the destination grid file. The unmapped points in the destination field are filled with zeros instead of the correct _FillValue.
• The ESMF_GridCreate() interface which allows the user to create a copy of an existing Grid with a new distribution will give incorrect results when used on a Grid with 3 or more dimensions and whose coordinate arrays are less than the full dimension of the Grid (i.e. it contains factorized coordinates).
• The ESMF_Grid creation interfaces that take the petMap argument (e.g. ESMF_GridCreate1PeriDim()) mistakenly return an error for inputs that put more than one DE on a PET, if the DEs are not mapped onto the PETs in sequential order.
• The ESMF_GridCreate() from a UGRID file crashes when the node index is 0-based and the number of elements in a PET is less than four.
• When the gridAlign argument is used during Grid creation to set the index space alignment to 1 without explicitly using the gridEdgeLWidth argument to set the index space lower edge width, then an incorrect lower edge width of 0 is used. This results in the Grid not having the correct index space padding for non-corner staggers.
• The ESMF_XGridCreate() fails with error when created on Meshes or Grids where a source cell shares a single vertex with a destination cell.
• The ESMF_<class>Read() and ESMF_<class>Write() methods for Field, FieldBundle, Array, and ArrayBundle are known to not work correctly when ungridded/undistributed dimensions are present. The results are situation dependent and range from errors returned by the method, incorrectly written or read data, and crashes.
• ESMF_AttributeWrite() has only been verified to work for ESMF standard Attribute packages. Non standard Attribute packages may trigger a crash inside the ESMF_AttributeWrite() implementation.
• ESMF does not link correctly with the Fortran libraries of the latest MPICH 3 series due to a separation of the libraries holding C and Fortran symbols.

• All platforms using GNU g++: The library build fails with GNU g++ 6.1 (and newer) by default. The offending source code is in a section of code that can be disabled by setting ESMF_MOAB=OFF.
• Linux with g95 (Fusion): Fails the ESMF_ArrayIOUTest.F90 unit test.
• Linux with gfortran (Discover, Endeavour, Pleiades, Pluto): Fails the ESMF_ArrayIOUTest.F90 unit test.
• Linux with gfortran (Pluto): Fails the ESMF_ArrayBundleIOUTest.F90 and the ESMF_ArrayIOUTest.F90 unit tests.
• Linux with intel (Theia): After a system upgrade on Theia in October 2017, applications built with optimization level -O2 or higher, and linked against ESMF using the linker flags provided by esmf.mk, started failing. The symptoms were similar to memory corruption issues. It was determined that the problem could be triggered for a very simple Fortran program (not depending on ESMF), using ifort -O2 (or higher optimization), and explicitly linking against Intel's small vector math library via -lsvml. This finding was reported to Cray and Intel and a reproducer was made available to the vendor.
• Linux with intel (Pleiades): Fails the ESMF_ArrayIOUTest.F90 unit test.
• Linux with intel (Yellowstone): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently.
• Linux with PGI (Discover, Fusion): Fails the ESMF_DELayoutWorkQueue.F90 unit test intermittently with ESMF_COMM=mvapich2.
• Linux with PGI (Discover): Fails the ESMF_FieldBundleCrGetUTest.F90 unit test with ESMF_COMM=mvapich2 and ESMF_BOPT=O.
• Linux with PGI (Discover): Fails the ESMF_ArrayIOUTest.F90 unit test with ESMF_COMM=openmpi and ESMF_BOPT=O.
• Unicos with PGI (Titan): Fails the ESMF_ArrayIOUTest.F90 unit test with ESMF_BOPT=O.
• Windows/Cygwin with g95 (Nix): Fails the ESMF_UtilUTest.F90 and TestHarness unit tests.
• Windows/Cygwin with g95 (Nix): The command line tools ESMF_RegridWeightGen and ESMF_Regrid do not work.

• This release is backward compatible with ESMF 6.1.1 for all the interfaces that were previously marked as backward compatible. There were API changes to unmarked methods, and new methods were added. All of these changes are backward compatible with ESMF 6.1.1. The entire list of API changes is summarized in a table showing interface changes since ESMF_6_1_1, including the rationale and impact for each change.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• This release contains a reference version of the NUOPC Layer software. The NUOPC Layer provides generic components that represent Drivers, Models, Mediators, and Connectors. These pre-fabricated, customizable components allow users to easily build interoperable ESMF applications. New web pages, a Reference Manual, and many examples are available to help users learn about the software.
• Specific technical improvements to the NUOPC Layer in this release include:
  • The default entries in the Field Dictionary were reduced to those entries that are compliant with the Climate and Forecast (CF) Convention. It is the intention to grow the number of default entries over time, based on a clear relationship with CF.
  • Support for dynamic resolution of data-dependencies between components during initialization was added.
  • Support for compatibility checking of components that interact on multiple timescales was added.
• The capability of retrieving coordinates from an unstructured Mesh object created from a NetCDF file in SCRIP or UGRID format was added to the ESMF Python interface. More information about the Python interface is available at the ESMP web page.
• Support to write out weight files in the NetCDF4 format was added to the ESMF_RegridWeightGen application and the ESMF_RegridWeightGen() interface.
• The ESMF_RegridWeightGen application and the ESMF_MeshCreate() interface have been extended to support 3D unstructured grids provided in the UGRID convention.
• The ESMF_FieldRegridStore() call has been enhanced so that now regridding takes into account Mesh node masking. This will allow the user to mask out portions of an unstructured Mesh.
• The user can now get the coordinate system that a Grid was created on from the ESMF_GridGet() interface.
• A new ESMF_MeshCreate() interface was added. This interface allows a user to create a copy of an existing Mesh, but with a different distribution of the nodes and elements from that of the original Mesh.
• The ESMF_MeshCreate() and ESMF_MeshAddNodes() methods have been extended to allow the user to optionally set node masks.
• The ESMF_UtilIOMkDir() interface was added to allow creation of a directory in the file system. The complementary ESMF_UtilIORmDir() interface was also added that allows removal of an empty directory.
• A new demonstration program, ESMF_Regrid, was added to the ESMF external_demos repository. This program demonstrates regridding between Fields built on Grids created from GRIDSPEC formatted files.

• Same as ESMF_6_1_1 with the following exceptions:
  • The ESMF_RegridWeightGen application now correctly outputs the grid mask array in the output weight file when either input grid is supplied following the UGRID convention.
  • The ESMF regrid weight generation methods and applications now consistently give the correct answer when a destination point falls very close to the edge of a source cell when using the bilinear method on 3D Grids or 3D Meshes.
  • The ESMF regrid weight generation methods and applications no longer suffer from performance issues when going from a very high resolution grid to a coarse grid (e.g. a 2km global grid to a 7.5x1.875 degree global grid).
  • The ESMF_RegridWeightGen application and interface no longer fail if an input grid contains a polygon with more than 20 sides. The allocation is now done dynamically.
  • The ESMF_GridCreate() interface which allows the user to create a copy of an existing Grid with a new distribution now gives correct results if the Grid contains items (e.g. areas or mask values). The new Grid contains correctly redistributed versions of the original Grid items.
  • The ESMF_GridCreate() interface which allows the user to create a copy of an existing Grid with a new distribution now correctly works if the Grid contains stagger locations other than ESMF_STAGGERLOC_CENTER, ESMF_STAGGERLOC_EDGE1, and ESMF_STAGGERLOC_EDGE2. The new Grid will contain correctly redistributed versions of the original Grid stagger locations.
  • ESMF_AttributeUpdate() now works correctly even if the only changes have been Attribute removals.
• The ESMF_GridCreate() interface which allows the user to create a copy of an existing Grid with a new distribution will give incorrect results when used on a Grid with 3 or more dimensions and whose coordinate arrays are less than the full dimension of the Grid (i.e. it contains factorized coordinates).
• The code that allows the compliance checker to report memory usage on Linux is not thread-safe. The ESMF_ComplianceCheckerSTest.F90 system test fails or hangs on Linux systems when ESMF_TESTWITHTHREADS is set to ON.
• Also see SourceForge bugs list.

• Linux with PGI (Discover, Fusion, Jet): Systems tests hang intermittently with ESMF_COMM=mvapich2 due to a system level issue with the mvapich2/PGI combination.
• Linux/SGI Altix ICE with gfortran (Pleiades): Fails the ESMF_field_1_NP4UTest test harness test with ESMF_COMM=mvapich2.
• Darwin with g95 (Haumea): Fails the ESMF_LocStreamEx.F90 example with ESMF_BOPT=g for ESMF_COMM=mpiuni.
• Darwin with gfortran (Haumea): Fails the ESMF_LocStreamUTest.F90 and ESMF_MeshUTest.F90 unit tests and the ESMF_LocStreamEx.F90 example with ESMF_BOPT=O for ESMF_COMM=mpich2 and ESMF_COMM=mvapich2.
• Linux with PGI (Fusion): Fails the ESMF_FieldRegridUTest.F90 unit test with ESMF_COMM=mvapich2.
• Windows/Cygwin with g95: Fails the ESMF_UtilUTest.F90 unit test when linked against the shared library version of ESMF.
• Windows/MinGW with intel: The ESMF_CompTunnelUTest.F90 unit test fails.

• This release is backward compatible with ESMF 6.1.0. There are no API changes.
• The regridding capabilities in this release are identical to ESMF 6.1.0. Tables summarizing the ESMF regridding status are available. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.

• Same as ESMF_6_1_0 with the following exceptions:
  • The bug in the NUOPC Layer that lead to an "Input argument out of range" error during Component initialization has been fixed. Full functionality of the NUOPC Layer has been restored.
• Also see SourceForge bugs list.

• This release is backward compatible with ESMF 5.3.0 for all the interfaces that were previously marked to remain backward compatible in future ESMF releases. However, there were API changes to unmarked methods that may require user code modifications. In FieldBundle, Field, ArrayBundle, and Array, changes were made to the Read/Write methods as a consequence of reworking the I/O capabilities. In Grid, an unmarked method was removed as obsolete. In XGrid, several unmarked methods were modified to provide a cleaner, more general interface. In Attribute the convention and purpose of a few Attribute packages were updated. In the NUOPC Layer, there were changes in the initialize phase and run sequence definitions. The entire list of API changes is summarized in a table showing interface changes since ESMF_5_3_0, including the rationale for each change.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• The capability of creating an unstructured Mesh object from a NetCDF file in SCRIP or UGRID format was added to the ESMF Python interface. More information about the Python interface is available at the ESMP web page.
• The NUOPC_Model component now supports nesting. Parent and nested domains are separate instances of the same model component, running as siblings under the same NUOPC_Driver component.
• A Science Component class, named ESMF_SciComp, has been implemented. The ESMF_SciComp is intended to represent scientific domains, or realms, in an Earth Science Model. Its primary purpose is to provide a means for representing Component metadata within a hierarchy of Components by functioning as a container for Attributes as well as other Components.
• The implementation of the ESMF_StateReconcile() method was redesigned to improve performance, and scalability. There were no changes to the user interface.
• A new option was introduced to the container Get() methods to support the retrieval of container elements in the order in which they were added, instead of returning them in the default alphabetical order. All container classes (ESMF_State, ESMF_FieldBundle, and ESMF_ArrayBundle) support this new functionality.
• Added two new arguments to the ESMF_RegridWeightGen application: --src_coordinates and --dst_coordinates to allow users to choose the coordinate variables if multiple sets of coordinates are defined in a GRIDSPEC file.
• The ESMF_FieldRegridStore() method, and the offline regrid weight generation application (ESMF_RegridWeightGen), have been extended to support two new nearest neighbor regridding methods:
  • The nearest source to destination method allows the user to map each destination point to the closest source point.
  • The nearest destination to source method allows the user to map each source point to the closest destination point.
• The ESMF_FieldRegridStore() call has been extended to allow the user to obtain a list of unmapped destination points. This list is available via an optional argument passed into the method.
• An option was added to the ESMF_FieldRegridStore() call to allow users to ignore degenerate cells when doing first-order conservative regridding. Cells are considered degenerate if enough corners are closer than 1E-15 degrees, and practically collapse into a line or point. Previously these cells caused the method to return with an error.
• A new method was added to the library that supports regrid weight generation from grid files. The new interface is called ESMF_RegridWeightGen().
• The capability to support bit-for-bit reproducibility was added to the sparse matrix multiply methods. This capability is also available for the internal sparse matrix multiply part of the regrid method, although the regrid weight generation part is not yet guaranteed to be bit-for-bit. Three levels of bit-for-bit reproducibility are supported: strict reproducibility across different number of PETs, relaxed reproducibility when the number of PETs does not change, and no guarantee of bit-for-bit reproducibility when performance is most important.
• The interfaces and the implementation of the ESMF I/O methods for ESMF_Field, ESMF_FieldBundle, ESMF_Array, and ESMF_ArrayBundle were redesigned to be more consistent and complete. None of these interfaces have been marked as backward compatible yet. The interface changes are detailed in the table showing interface changes since ESMF_5_3_0.
• The ESMF_GridCreate() method that creates Grids from SCRIP or GRIDSPEC files has been parallelized to improve performance.
• XGrid creation has been extended to support flexible ways of supplying Grids and Meshes on either side of the XGrid. Weights generated by regridding through XGrid can be renormalized by user supplied source or destination area. The user can retrieve the area and centroid information from the XGrid.
• An Attribute package implementing the METAFOR Common Information Model (CIM) of version 1.5.1 was added to allow a CIM 1.5.1 compliant XML file to be written from the internal information stored in an ESMF Grid.
• The ESMF_AttributeWrite() method was modified and is now a collective call. This means that all PETs that are part of the current VM must issue this call.
• The ESMF_UtilSort() utility was added for sorting arrays of data. Currently supported data types include: character string, integer, long integer, real, and double precision.

• Same as ESMF_5_3_0 with the following exceptions:
  • Added a small tolerance (1E-10 degrees) when determining whether two corner coordinates are identical in a SCRIP unstructured grid file. This allows the ESMF_RegridWeightGen application to correctly handle SCRIP unstructured grid files that have small round off errors in the corner coordinate arrays.
  • The ESMF_RegridWeightGen application now correctly handles bilinear and patch regridding on SCRIP unstructured grid files containing concave cells.
  • The instability that led to undefined behavior (e.g. segmentation fault) for all the sparse matrix store calls (i.e., ESMF_FieldBundleSMMStore(), ESMF_FieldSMMStore(), ESMF_ArrayBundleSMMStore(), and ESMF_ArraySMMStore()) for very large problem sizes (~15 million factors in the sparse matrix) has been fixed.
  • The decomposition scheme of the original DistGrid (indicated by the ESMF_Decomp_Flag) is now correctly duplicated by the ESMF_DistGridCreate() methods that take a DistGrid input argument.
• A bug in the NUOPC Layer leads to an "Input argument out of range" error during Component initialization. This issue renders the NUOPC Layer practically nonfunctional in this ESMF release.
• When the ESMF_FieldRegridStore() call or the ESMF_RegridWeightGen application is used with nearest destination to source method, the unmapped destination point detection does not work. Even if the option is set to return an error for unmapped destination points (the default) no error will be returned.
• Also see SourceForge bugs list.

• AIX/IBM (Bluefire): Fails the ESMF_ArrayRedistOpenMPSTest.F90 system test intermittently.
• Linux with PGI (Discover, Fusion, Jet): Systems tests hang intermittently with ESMF_COMM=mvapich2 due to a system level issue with the mvapich2/PGI combination.
• Linux/SGI Altix ICE with gfortran (Pleiades): Fails the ESMF_field_1_NP4UTest test harness test with ESMF_COMM=mvapich2.
• Linux with PGI (Jet): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently with ESMF_COMM=mvapich2.
• Linux with PGI (Pleiades): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently with ESMF_COMM=mpi.
• Darwin with g95 (Haumea): Fails the ESMF_LocStreamEx.F90 example with ESMF_BOPT=g for ESMF_COMM=mpiuni.
• Darwin with gfortran (Haumea): Fails the ESMF_LocStreamUTest.F90 and ESMF_MeshUTest.F90 unit tests and the ESMF_LocStreamEx.F90 example with ESMF_BOPT=O for ESMF_COMM=mpich2 and ESMF_COMM=mvapich2.
• Linux with PGI (Fusion): Fails the ESMF_FieldRegridUTest.F90 unit test with ESMF_COMM=mvapich2.
• Windows/Cygwin with g95: Fails the ESMF_UtilUTest.F90 unit test when linked against the shared library version of ESMF.
• Windows/MinGW with intel: The ESMF_AttributeWriteInternalUTest.F90 unit test fails.
• Windows/MinGW with intel: The ESMF_CompTunnelUTest.F90 unit test and the ESMF_CompTunnelEx.F90 example fail.
• Windows/MinGW with intel: The ESMF_FieldBundleSMMSTest.F90 and ESMF_FieldSparseMatMulSTest.F90 system tests fail during VM shutdown.
• Windows/MinGW with intel: Several ESMF_Attribute unit tests (ABundle, Array, CplComp, FBundle, Field, Grid, Sci, State) fail due to file comparison differences between Windows and Linux line terminators.

• This release is backward compatible with ESMF 5.2.0rp2 for all the interfaces that were previously marked to remain backward compatible in future ESMF releases. However, there were API changes to unmarked methods in Mesh and Attachable Methods. These changes require user code modifications unless argument keyword syntax was used. The entire list of API changes is summarized in a table showing interface changes since ESMF_5_2_0rp2, including the rationale for each change.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• A Python interface to ESMF, currently called ESMP, has been implemented. It is available for download at the ESMP web page. ESMP provides support for Grids, Meshes, Fields, and Field regridding. The regridding capability includes bilinear, higher-order patch recovery and first-order conservative regridding methods and the choice to ignore unmapped points. There is support for applying masking or user areas to the Grid or Mesh, and retrieving areas or fractions from a Field. There is also limited access to the ESMF VM and Log capability through ESMP.
• A fault-tolerant extension to the ESMF_GridComp and ESMF_CplComp interfaces has been implemented. This extension allows ESMF Components, running in separate executables, to be linked via a socket based connection. A user controlled "timeout" mechanism allows either side of this link to continue with normal execution even after the other side of the connection has become unresponsive. The practical use cases of this new feature are large ensemble configurations where individual ensemble members may crash during the course of a simulation run. The fault-tolerant Component mechanism allows the ensemble driver, under these circumstances, to complete the run in a controlled manner with the remaining ensemble members. This mechanism was implemented by adding a "timeout" argument to ESMF_GridCompDestroy(), ESMF_GridCompFinalize(), ESMF_GridCompInitialize(), ESMF_GridCompReadRestart(), ESMF_GridCompRun(), ESMF_GridCompSetServices(), ESMF_GridCompWait(), and ESMF_GridCompWriteRestart(), as well as the equivalent ESMF_CplComp methods. New methods ESMF_GridCompServiceLoop(), and ESMF_CplCompServiceLoop() were added with the "timeout" argument.
• The generic NUOPC_Driver component now supports petLists for all model and connector components. This enables these components to be run concurrently if desired.
• The generic NUOPC_Driver and NUOPC_Model components now support simple explicit, semi-implicit, as well as complex implicit component run sequences.
• The ESMF_StateReconcile() method can now be called repeatedly on the same State during the course of execution. This can be used to re-reconcile the State after adding to or removing objects from it.
• Added an optional "index" argument to all of the ESMF_MethodAdd(), ESMF_MethodExecute(), and ESMF_MethodRemove() calls to support multiple methods under the same "label". The NUOPC layer utilizes this feature to implement phase-specific Component method specializations.
• An ESMF_FieldCopy() method was added. This method copies data values from a source Field to a destination Field. The underlying data arrays in the source and destination Fields must have the same shape and distribution (i.e., their DistGrids must be identical).
• The ESMF_FieldRegridStore() call has been extended to support first order conservative regridding involving Fields built on 3D Grids or 3D Meshes. The 3D Meshes this capability will work on are ones composed of hexahedrons and tetrahedrons. (In contrast to this, bilinear interpolation is supported on 3D Meshes composed just of hexahedrons and higher-order patch interpolation isn't supported in 3D.)
• The ESMF_FieldRegridStore() call has been enhanced so that now conservative regridding takes into account Mesh cell masking. This will allow the user to mask out portions of an unstructured Mesh.
• The ESMF_FieldRegridStore() call has been extended so that now conservative regridding is adjusted to use the cell areas set in the Grid or Mesh object upon which the source and destination Fields are built. If these areas are set, the conservation will now hold for these areas. If not set, the conservation will hold for the cell areas as calculated by ESMF.
• The offline regrid weight generation application (ESMF_RegridWeightGen) now has the capability to generate conservative weights for 3D unstructured Meshes specified in the ESMF unstructured file format.
• The ESMF_RegridWeightGen application supports two new grid file formats: Climate and Forecast (CF) UGRID convention for 2D unstructured grids and CF GRIDSPEC convention for single-tile logically rectangular 2D grids.
• The ESMF_RegridWeightGen application allows users to construct a grid mask by using the missing values of a data variable defined in the UGRID or the GRIDSPEC file.
• A new command line option (--users_area) was added to the ESMF_RegridWeightGen application. For conservative regridding this option causes the weight generation to use the grid areas provided in the grid files instead of calculating the areas internally. If these areas are read in, the conservation will now hold for these areas. If the option is not set, the conservation will hold for the cell areas as calculated by ESMF.
• An ESMF_ArrayCopy() method was added. This method takes two existing Array objects and copies the data from one Array into the other, after verifying that both have the same shape and distribution (i.e., the DistGrids must be identical).
• The ESMF_GridCreate() method that supports creation of a Grid from a SCRIP format NetCDF file was extended to support files using the CF GRIDSPEC convention. Options for user area and masking were also added to the interface.
• The ESMF_MeshCreate() and ESMF_MeshAddElements() methods have been extended to allow the user to optionally set cell areas and cell masks.
• The ESMF_MeshCreate() method that supports creation of a Mesh from a SCRIP format or ESMF unstructured Mesh format NetCDF file was extended to support files using the CF UGRID convention. Options for user area and masking were also added to the interface.
• The ESMF_MeshCreate() method that supports creation of a Mesh from an ESMF unstructured Mesh format NetCDF file was extended to support 3D unstructured Meshes.
• The ESMF_XGridCreate() method was extended to support lists of Grids on either side of the XGrid (exchange grid). The flux exchange calculated through the XGrid is first order conservative. Masking and merging when creating an XGrid are not supported in this release.
• The ability to access Grid object information through the Attribute class with string valued input parameters was added to ESMF_AttributeGet(). This is an initial example of how information can be drawn directly from data objects, and does not need to be entered by the user.
• Implemented the custom calendar kind, ESMF_CALKIND_CUSTOM, in Time Manager. This can be used to define calendars for other planets, such as Mars, as well as custom Earth calendars.
• Added support for Fortran logical data type in ESMF_VMGather() and ESMF_VMAllToAllV().
• Added support for Fortran character array in ESMF_VMAllGatherV() and ESMF_VMAllToAllV().
• Implemented ESMF_VMAllToAll() for real and integer data types.
• Support for the Grid, Fields built on Grids and Field regridding was added to the C interface. There is support for applying masking or user areas to the Grid or Mesh, and retrieving areas or fractions from a Field.
• By default, LAPACK dependencies are now satisfied using internal code. Linking against an external LAPACK library is no longer required to use the higher order patch interpolation.

• The ESMF_StateReconcile() call overestimates the size requirement of an internal temporary buffer. The estimation error increases with core count and causes ESMF_StateReconcile() to run out of memory for large core counts (~10,000) on typical configurations.
• FieldBundles don't currently enforce that every contained Field is built on the same Grid, Mesh, LocStream, or XGrid object, although the documentation says that this should be so.
• ESMF_FieldRegridStore() doesn't currently work on Fields created on Grids which contain a DE of width less than 2 elements.
• ESMF_FieldRegridStore() doesn't currently work on Fields created on Grids of arbitrary distribution.
• ESMF_FieldRegridStore() that takes XGrid as an argument does not support regridding between Fields that are constructed on Grids for which ESMF_GridMatch() locally returns a value of ESMF_GRIDMATCH_EXACT or greater. This may result in unexpected local errors.
• A larger than expected difference in integrals (8.7E-10) is occurring when using the conservative regridding between a 1 deg lat lon grid and a 2.5 deg lat lon grid. This problem appears isolated to a subset of the cases when grid cell edges align, but not exactly due to round off. It is recommended that users test the integration of their conservative interpolation under these circumstances.
• The ESMF_Grid creation interfaces that take the petMap argument (e.g. ESMF_GridCreate1PeriDim()) mistakenly return an error for inputs that put more than one DE on a PET, if the DEs are not mapped onto the PETs in sequential order.
• The ESMF_RegridWeightGen application does not correctly handle SCRIP unstructured grid files that have small round off errors in the corner coordinate arrays. This problem leads to an abort with error about degenerated cells.
• The ESMF_RegridWeightGen application does not correctly handle bilinear and patch regridding on SCRIP unstructured grid files containing concave cells. This problem leads to an abort with error about degenerated cells.
• The ESMF_RegridWeightGen application does not output the grid mask array in the output weight file when either input grid is supplied following the UGRID convention. As a result, the output weight file produced under this condition does not work with the scrip_test utility program provided by the SCRIP package.
• The ESMF regrid weight generation methods and applications can give wrong answers for the bilinear method on 3D Grids or 3D Meshes when a point in the destination falls very close to the edge of a source cell. In this case the destination point can sometimes be mapped to a wrong cell.
• The ESMF regrid weight generation methods and applications take noticeably longer to generate regridding weights when going from a very high resolution grid to a coarse grid (e.g. a 2km global grid to a 7.5x1.875 degree global grid) than when going to another high resolution grid.
• The ESMF_RegridWeightGen application and interface will fail with an error if an input grid contains a polygon with more than 20 sides.
• All of the sparse matrix store calls (i.e., ESMF_FieldBundleSMMStore(), ESMF_FieldSMMStore(), ESMF_ArrayBundleSMMStore(), and ESMF_ArraySMMStore()) suffer from an instability that can lead to undefined behavior (e.g. segmentation fault) for very large problem sizes (~15 million factors in the sparse matrix).
• The ESMF_GridCreate() interface which allows the user to create a copy of an existing Grid with a new distribution will give incorrect results if the Grid contains items (e.g. areas or mask values). The new Grid is created without items.
• The ESMF_GridCreate() interface which allows the user to create a copy of an existing Grid with a new distribution will give incorrect results if the Grid contains stagger locations other than ESMF_STAGGERLOC_CENTER, ESMF_STAGGERLOC_EDGE1, and ESMF_STAGGERLOC_EDGE2. The new Grid will not contain any stagger locations other than the above.
• The decomposition scheme of the original DistGrid (indicated by the ESMF_Decomp_Flag) is not duplicated by the ESMF_DistGridCreate() methods that take a DistGrid input argument. Instead, the duplicate DistGrid is always constructed using ESMF_DECOMP_BALANCED. This issue is most noticeable when building Grids on DistGrids with decomposition schemes other than the default ESMF_DECOMP_BALANCED. A Field built on such a Grid will incorrectly use a balanced decomposition.
• ESMF_AttributeWrite() only works for ESMF standard Attribute packages.
• ESMF_AttributeUpdate() will not work correctly if the only changes have been Attribute removals. In this situation the call will return success but no updating will have occurred. This situation can occur during the removal of Attribute packages as well as single Attributes.
• Also see SourceForge bugs list.

• AIX/IBM (Bluefire): Fails the ESMF_ArrayRedistOpenMPSTest.F90 system test intermittently.
• Darwin with gfortran (Haumea): Fails the ESMF_CompTunnelUTest.F90 unit test with ESMF_COMM=mpich2 and ESMF_COMM=mvapich2.
• Linux with Intel (Columbia): Fails the ESMF_FieldRegridUTest.F90 unit test with ESMF_COMM=mpi and mpiuni + ESMF_BOPT=O. It also fails the ESMF_FieldSphereRegridEx.F90 example with ESMF_COMM=mpi + ESMF_BOPT=O.
• Linux with Intel (Discover): Fails the ESMF_CompTunnelUTest.F90 unit test with ESMF_COMM=mvapich2 + ESMF_BOPT=O. It also fails the ESMF_CompTunnelEx.F90 example intermittently with ESMF_COMM=mvapich2 + ESMF_BOPT=O.
• Linux with g95 (Pluto): Fails the ESMF_CompTunnelUTest.F90 unit test intermittently with ESMF_COMM=mvapich2 and mpich2. It also fails the ESMF_CompTunnelEx.F90 example intermittently with ESMF_COMM=mvapich2 + ESMF_BOPT=O.
• Linux with gfortran (Pluto): Fails the ESMF_CompTunnelUTest.F90 unit test intermittently with ESMF_COMM=mpich2 and openmpi. It also fails the ESMF_CompTunnelEx.F90 example intermittently with ESMF_COMM=mpich2 + ESMF_BOPT=O.
• Linux with Intel (Pluto): Fails the ESMF_CompTunnelUTest.F90 unit test with ESMF_COMM=mvapich2 + ESMF_BOPT=g.
• Linux with NAG (Pluto): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently with ESMF_COMM=openmpi.
• Linux with NAG (Pluto): Fails the ESMF_CompTunnelUTest.F90 unit test with ESMF_COMM=mpich2 + ESMF_BOPT=O and ESMF_COMM=mvapich2 + ESMF_BOPT=g.
• Linux with PGI (Discover, Fusion, Jet): Systems tests hang intermittently with ESMF_COMM=mvapich2 due to a system level issue with the mvapich2/PGI combination.
• Linux/SGI Altix ICE with Intel (Pleiades): Fails the ESMF_field_1_NP4UTest test harness test with ESMF_COMM=openmpi.
• Linux/SGI Altix ICE with Intel (Pleiades): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently with ESMF_COMM=intelmpi.
• Linux/SGI Altix ICE with Intel (Pleiades): Fails the ESMF_CompTunnelEx.F90 example intermittently with ESMF_COMM=mvapich2 + ESMF_BOPT=g.
• Linux/SGI Altix ICE with gfortran (Pleiades): Fails the ESMF_field_1_NP4UTest test harness test with ESMF_COMM=mvapich2.
• Linux/Sun Fire X4640 with PGI (Euclid): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently with ESMF_COMM=openmpi.
• Linux with PGI (Jet): Fails the ESMF_DELayoutWorkQueueUTest.F90 unit test intermittently with ESMF_COMM=mvapich2.
• Windows/Cygwin with g95: Fails the ESMF_UtilUTest.F90 unit test when linked against the shared library version of ESMF.
• Windows/MinGW with intel and intelcl: The ESMF_CompTunnelUTest fails and the ESMF_CompTunnelEx hangs.
• Windows/MinGW with intel: The ESMF_FieldBundleSMMSTest fails during VM shutdown.
• Windows/MinGW with intelcl: The ESMF_FieldSparseMatMulSTest fails during VM shutdown.

• Added support for the PGI 11.x compiler version on Linux/PC and the Cray XT platform.
• The ESMF regridding status have not changed since 5.2.0.

• Same as ESMF_5_2_0.
• Also see SourceForge bugs list.

• An intensive effort to make interfaces consistent in preparation for future backward compatibility resulted in significant interface changes for this release. These are summarized in a table showing interface changes since ESMF_5_1_0, including the rationale for and the impact of each change.
• Tables summarizing the ESMF regridding status have been updated. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• Added ESMF_StateReplace() method, which replaces an existing State item with a new item of the same name.
• Added ESMF_StateRemove() method, which removes an item from a State.
• Masking is now supported for conservative interpolation, for logically rectangular grids, for both the offline regrid weight generation application and the ESMF_FieldRegridStore() method.
• Support was added for regridding a regional Grid on a sphere. This allows regridding between two regional grids on a sphere, between a regional Grid on a sphere and spherical global Grid and between a spherical global Grid and a regional Grid on a sphere. This capability is available in both the offline regrid weight generation application and ESMF_FieldRegridStore().
• Enabled a Grid constructed using DistGrid connections to work with ESMF_FieldRegridStore(). For example, this means that a user constructed tripole Grid can now participate in a regrid operation.
• Optimized ESMF_FieldBundleRegridStore() to be faster when Fields are built on the same Grid. When computing the sparse matrices for the regridding, if the Grids in the source and destination Fields match the Grids in the previous set of source and destination Fields in their respective FieldBundle lists, then that same sparse matrix is used instead of recomputing a new one. In particular, this means that if all the Fields in the source FieldBundle are built on the same Grid and all the Fields in the destination FieldBundle are built on the same Grid, then the sparse matrix between those Grids only needs to be computed once. This can save a substantial amount of time for FieldBundles containing many Fields.
• Support for options from ESMF_FieldRegridStore() were added to ESMF_FieldBundleRegridStore(). Masking, pole options and the ability to chose what happens when destination points can't be mapped were added to the ESMF_FieldBundleRegridStore() interface.
• Added ability to create multiple instances of the Attribute packages ISO 191115/Responsible Party Description and/or ISO 191115/Citation Description, nested within a standard CIM 1.0/Model Component Simulation Description package for Components.
• The Attribute values within the ESG/General and CF/Extended standard packages, nested within the standard CIM 1.0/Inputs Description package for Fields, now appear in the Component Properties section of the CIM XML output file. This allows input and output Fields to be described for components other than the top level component in CIM output files.
• The ESMF Coupled Flow demonstration program has been updated to work with current interfaces. It is now part of the external demos module.
• Added ESMF_UtilGetArgC() subroutine, which returns the number of command line arguments.
• Added ESMF_UtilGetArg() subroutine, which returns a given command line argument.
• Added ESMF_UtilGetArgIndex() subroutine, which searches for a specified command line argument (e.g., '-filename') and, if found, returns its index.
• An online tutorial based on the Coupled Flow demonstration program was added to the ESMF website.
• The prototype Exchange Grid implementation has been extended to work with multiple PETS.

• Same as ESMF_5_1_0 with the following exceptions:
  • The argument attPackInstanceName, introduced in ESMF_5_1_0, in ESMF_AttributeGet(), ESMF_AttributeSet(), and ESMF_AttributeRemove() is now implemented, supporting multiple instances of Attribute packages as described in the release notes.
  • The offline regrid weight generation application, ESMF_RegridWeightGen, is not limited to 2D global grids.
  • The offline regrid weight generation application, ESMF_RegridWeightGen, now supports masking.
  • The coordinate unit attribute is now read correctly on all machines, so grid files with coordinates in radians will be interpreted correctly.
  • Masking is now supported for conservative regridding.
  • The Attribute Table in Section 33.2.6 of the pdf version of the Fortran Reference Manual is not complete, please refer to HTML version of the document for a complete table. This table was in Section 34.2.6 in release ESMF_5_1_0.
• A larger than expected difference in integrals (8.7E-10) is occurring when using the conservative regridding between a 1 deg lat lon grid and a 2.5 deg lat lon grid. This problem appears isolated to a subset of the cases when grid cell edges align, but not exactly due to round off. It is recommended that users test the integration of their conservative interpolation under these circumstances.
• The ESMF_FieldRegridStore() call or the ESMF_RegridWeigthGen application may erroneously report destination points as unmapped when running on multiple processors.
• The matrix coming out of ESMF_FieldRegridStore() needs to be transposed before it can be used in ESMF_FieldSMMStore().
• The weight file produced by the ESMF_RegridWeightGen application may contain multiple entries for the same source and destination pair. This does not affect the accuracy of the regridding, but it does increase the file size.
• Also see SourceForge bugs list.

• IBM Bluefire AIX: Fails the ESMF_ArrayRedistOpenMPSTest.F90 system test with ESMF_BOPT=O.
• Linux Pluto with g95: Fails the ESMF_FieldRegridXGUTest.F90 unit test with ESMF_COMM=mvapich2.
• PC Xeon(64) Cluster Jet with PGI: Fails the ESMF_GridCreateUTest.F90 and ESMF_UtilUTest.F90 unit tests.
• Cray XT5 Jaguarpf with PGI: Fails the ESMF_GridCreateUTest.F90 and ESMF_UtilUTest.F90 unit tests.
• Windows/Cygwin g95: Fails the ESMF_LogErrUTest.F90 and ESMF_UtilUTest.F90 unit tests.

• List of ESMF interface changes since ESMF_5_0_0.
• An integrated ESMF compliance checker has been implemented. It can be activated by setting the runtime environment variable ESMF_RUNTIME_COMPLIANCECHECK=ON. The compliance checker is currently geared toward the NUOPC requirements. The current version is described in more detail in the "NUOPC Layer: ESMF Compliance Checker - ESMF v5.1.0" document.
• Tables summarizing the ESMF regridding status have been compiled. These include supported grids and capabilities of the offline and integrated regridding, and numerical results of some specific test cases.
• ESMF_StateGet() has been enhanced to allow additional nested State capabilities. These include:
  • Items can now be searched by name using the itemSearch argument.
  • The nestedFlag argument controls whether the search traverses the entire State hierarchy.
  • State items can be accessed directly by using a path style naming convention.
• The methods ESMF_StateSetNeeded(), ESMF_StateIsNeeded(), and ESMF_StateGetNeeded() that were removed in release ESMF_5_0_0 have been reinstated. This was necessary to reenable the ESMF COUPLED_FLOW demo.
• The first-order conservative regridding algorithm has been reworked. This new version is based on the overlap of source cells with destination cells, and is less prone to interpolation artifacts and diffusion than the previous version. It is supported for regridding between any combination of Fields built on the center-stagger of 2D logically rectangular Grids and Fields built on the element location of 2D unstructured Meshes. Grids and Meshes can be either regional or mapped to the sphere (using the ESMF_REGRID_SCHEME_FULL3D option to ESMF_FieldRegridStore()). This functionality is available either offline (via the ESMF_RegridWeightGen application) or through the library (via ESMF_FieldRegridStore()).
• The ESMF_FieldRegridStore(), ESMF_FieldRegrid() and ESMF_FieldRegridRelease() methods have been overloaded to support Fields created on XGrids.
• The two offline weight generation applications ESMC_RegridWgtGenEx and ESMF_CubedSphereRegridEx have been merged into a single application named ESMF_RegridWeightGen. The application is documented under the "Applications" section of the ESMF Reference Manual. This application generates interpolation weights using either bilinear, finite element patch recovery or first order conservation methods. The input grids can be either 2D global logically rectangular or 2D global unstructured.
• The new location argument in ESMF_FieldCreate() allows Fields to be created on Mesh elements or Mesh nodes. Conservative regridding is only supported between Fields created on Mesh elements.
• Added ESMF_FieldCreate() method that supports creation of a Field on an XGrid.
• ESMF_FieldCreate() and ESMF_ArrayCreate() have been overloaded to support typekind and rank arguments in addition to the ArraySpec versions.
• Implemented halo support for arbitrarily decomposed FIeld and Array objects.
• This release includes new parallel I/O capabilities introduced by using the Parallel I/O (PIO) library within ESMF. PIO development was led by John Dennis of NCAR and it has evolved under a NCAR/DOE laboratory partnership. ESMF data classes Field, FieldBundle, Array and ArrayBundle can now read and write NetCDF files in parallel using methods such as ESMF_FieldRead(), and ESMF_FieldWrite(). Please see the "IO Capabilties" section in the ESMF Reference Manual for how to enable PIO in the ESMF build.
• Added ESMF_GridCreate() method that supports creation of a logically rectangular 2D Grid from a SCRIP format NetCDF file.
• Added ESMF_MeshCreate() method that supports creation of an unstructured 2D Mesh from a SCRIP format or ESMF unstructured Mesh format NetCDF file.
• A first implementation of the Exchange Grid (XGrid) class has been added. An XGrid object can be created from user supplied lists of Grids and sparse matrix parameters. The functionality of ths initial implementation is currently limited to single PET execution. This is not a limitation of the XGrid design but reflects the current implementation status.
• Added standard ESMF-supplied Attribute packages for Components and Fields, whose XML output conforms to the Climate and Forecast Conventions (CF) and METAFOR's Common Information Model (CIM) schema. The output is ingestable into the Earth System Grid (ESG) web portal for display. For details and examples please see the Attribute section of the ESMF Reference Manual.
• Added ability for Attribute packages to contain multiple nested Attribute packages of different types.
• The new msgAllow argument in ESMF_LogSet() permits users to specify the type of Log messages allowed for output. Other Log messages are ignored.
• The ESMF COUPLED_FLOW demo application has been restored but is currently limited to single PET execution. The documentation in the User's Guide has been updated.
• Starting with this release a collection of external demonstration programs is available in form of a separate CVS module. These external demos have been written to serve as good examples to show how users would interact with an ESMF installation. A description of the available demos and download instructions are available on the ESMF website under Users->Code Examples->External Demos.
• Added ESMF_ComplianceCheckerSTest.F90 system test which demonstrates the operation and output of the compliance checker within a multi-Component, multi-PET application.
• Added ESMF_AttributeCIMSTest.F90 system test which demonstrates the use of the new standard ESMF-supplied, METAFOR CIM Attribute packages for Components and Fields, within a multi-Component, multi-PET application.

• ESMF now offers a Reference Manual for the ESMF C API. This document is a work in progress.
• The available C interfaces are a subset of the Fortran API. This subset was constructed to fullfil the need of a specific coupling project.
• Changes were made to improve interface consistency.

• Same as ESMF_5_0_0 with the following exceptions:
  • The Field description in the reference manual now correctly states that the halo update operation is supported.
  • Conservative regridding has been reworked so that its accuracy is now within the expected range (15 decimal places).
  • Resolution differences between source and destination grids no longer cause problems in conservative regridding.
  • Suspicious interpolation weights (large, outside [0,1]) that occurred for cubed sphere regridding no longer occur for bilinear and conservative interpolation.
• The offline regrid weight generation application, ESMF_RegridWeightGen, only works for 2D global grids.
• The offline regrid weight generation application, ESMF_RegridWeightGen, does not support masking, and thus any mask information in the grid files will be ignored.
• On some machines (e.g. bluefire), the coordinate unit attribute is not read correctly by the ESMF_RegridWeightGen application, so grid files with coordinates in radians won't be interpreted correctly and will produce incorrect weights or errors.
• Masking is currently not supported for conservative regridding.
• Masking is currently not supported for unstructured grids (Mesh) for any regridding method.
• ESMF_ArrayWrite() and ESMF_FieldWrite() will fail with an internal PIO error if a NetCDF output file already exists from a previous run.
• XGrid does not support regridding between Fields that are constructed on Grids for which ESMF_GridMatch() locally returns a value of true. This may result in unexpected local errors.
• Fields built on XGrids cannot be added to FieldBundles.
• Only single-PET XGrids are supported. Creating and using multi-PET XGrids will result in undefined behavior.
• ESMF_AttributeUpdate() will not work correctly if the only changes have been Attribute removals. In this situation the call will return success but no updating will have occurred. This situation can occur during the removal of Attribute packages as well as single Attributes.
• The new argument attPackInstanceName in ESMF_AttributeGet(), ESMF_AttributeSet(), and ESMF_AttributeRemove() is not fully implemented. Consequently multiple instances of an Attribute package of the same type (same convention and purpose) within a nested Attribute package are not supported.
• The Attribute Table in Section 34.2.6 of the pdf version of the Fortran Reference Manual is not complete, please refer to HTML version of the document for a complete table.

• Discover with intel: Fails ESMF_CompCreateSTest.F90 with ESMF_COMM=mvapich2 and ESMF_BOPT=O.
• Discover with pgi: the ESMF_DELayoutWorkQueueUTest.F90 fails.
• IBM Frost Bluegene: Fails ESMC_ConfigUTest.C unit test with ESMF_BOPT=g.
• IBM Bluefire AIX: Fails ESMC_ConfigUTest.C unit test with ESMF_BOPT=g and ESMF_ArrayRedistOpenMPSTest.F90 with ESMF_ABI=64.
• Pluto with nag.: Fails system test ESMF_FieldBundleSMMSTest.F90 with ESMF_COMM=mvapich2 and mpich2 intermittently.
• Windows/MinGW with intel and intelcl: the ESMF_ArrayRedistOpenMPSTest.F90, the ESMF_FieldRegridUTest.F90, and ESMF_FieldArbGridUTest.F90 fail.
• Windows/Cygwin g95: the ESMF_MeshUTest.F90 fails.
• Windows/Cygwin g95: ESMF_LogErrUTest.F90 fails intermittently.

• List of ESMF interface changes since ESMF_4_0_0rp2.
• Tables summarizing the ESMF regridding status have been compiled. These include supported grids and capabilities in the offline and online regridding and numerical results of some specific test cases.
• ESMF_StatePrint() has a new nestedFlag argument for printing nested States. The options argument now supports 'short' and 'long' printouts. (Default is short.)
• ESMF_StateValidate() has a new nestedFlag argument for validating nested States.
• ESMF_StateGet() has a new nestedFlag argument for inquiry requests, which allows returning lists of nested objects and their types.
• The following methods were removed: ESMF_StateSetNeeded(), ESMF_StateIsNeeded(), and ESMF_StateGetNeeded().
  The reason for removal is that the semantics surrounding these routines were not well-defined, and they were rarely used. These methods may be revisited as part of an ongoing investigation into broader interoperability issues.
• Implemented halo communication operation for Field, FieldBundle, Array and ArrayBundle classes. Variable and asymmetric halo depths are supported.
• Added regridding support for FieldBundle objects.
• First order conservative regridding has been added as a modification to the existing Field regridding methods in both the offline and online and regridding. This modification is based on a finite element L2 projection method. In this initial version quantities are conserved globally. Further work is required for satisfactory local conservation.
• Improved the algorithm in ESMF_FieldRegridStore() to fix problems with interpolation on a sphere. The change fixes incidents of destination points that are lying close to the boundary between two source elements from being mapped into the wrong element. This led to errors in the interpolation weights.
• The offline weight generation application for structured grids has been more accurately renamed to ESMC_RegridWgtGenEx. The bilinear interpolation option, as well as a conservative option was added. For details please see the "Structured Grid to Structured Grid" subsection under the "File Based Regrid Weight Applications" section in the "Infrastructure: Fields and Grids" part of the ESMF Reference Manual.
• Added a new offline weight generation application called ESMF_CubedSphereRegridEx. This application supports regridding between a cubed sphere and a 2D logically rectangular grid mapped to a sphere. The user may use higher-order patch recovery interpolation or bilinear interpolation between the grids. For details please see the "Cubed Sphere to Structured Grid" subsection under the "File Based Regrid Weight Applications" section in the "Infrastructure: Fields and Grids" part of the ESMF Reference Manual.
• Support the reuse of RouteHandles for Fields and Arrays where the size of the undistributed dimensions that come before the first distributed dimension is different from the size used during the precompute.
• RouteHandle-based Array communication calls now provide a non-blocking option. This option allows user code to overlap computation with communication.
• Improved the implementation of ESMF_GridMatch(). Previously the method just checked to see if the Grids had identical pointers, now it checks to see if the two Grids' internal data is the same (dimensions, coordinates, items, etc.). This means that two Grids constructed independently of each other are checked more reliably.
• Added an option to ESMF_MeshGet() to allow users to retrieve coordinates for just the locally owned Mesh nodes. These coordinates will correspond exactly with the data elements stored in a Field created on the Mesh. This is useful for setting data in the Field or performing calculations involving coordinates on the data in a Field.
• Added the ability to validate Attributes read from an XML file by specifying a corresponding XSD file in method ESMF_AttributeRead(), via new optional argument schemaFileName. For standard ESMF-supplied Component, Field, and Grid Attribute packages validation is performed automatically, without the need to specify schemaFileName. For examples see the Attribute section of the ESMF Reference Manual.
• XML Attribute file writing is now done with the open source third party Xerces C++ library, replacing the previous internally hard-coded fprintf() technique. For reading and/or writing XML Attribute files, ESMF must be built against the Xerces library, which requires the setting of additional ESMF environment variables. See the "Third Party Libraries" section in the ESMF User's Guide for details.
• A new system test, ESMF_FieldRegridMeshToMesh, was added to demonstrate regridding between Fields built on unstructured Meshes.

• FieldBundles don't currently enforce that every contained Field is built on the same Grid, although the documentation says that this should be so.
• The Field description in the reference manual incorrectly states that the halo update operation is not supported. Field halo is available in this release.
• ESMF_FieldRegridStore() doesn't currently work on Fields created on Meshes which contain tetrahedra.
• ESMF_FieldRegridStore() doesn't currently work on Fields created on Grids which contain a DE of width less than 2 elements.
• ESMF_FieldRegridStore() doesn't currently work on Fields created on Grids of arbitrary distribution.
• ESMF_FieldRegridStore() only works on Fields created on Grids with indexflag equal to ESMF_INDEX_GLOBAL. This is not the default.
• Grid edge connections don't work in this release and will have no effect on the actions of ESMF_FieldRegridStore().
• he ESMF_Grid creation interfaces that take the petMap argument (e.g. ESMF_GridCreateShapeTile()) mistakenly return an error for inputs that put more than one DE on a PET, if the DEs are not mapped onto the PETs in sequential order.
• Testing on a constant field has shown that the integration weights that are generated for the conservative regridding are accurate to only 9 decimal places, rather than the expected 15 decimal places. This limits the precision of the conservative regridding method.
• The conservative regridding may generate errors when the destination grid is of higher resolution than the source grid, specifically:
  • Values of destination points will not be set when the destination cell lies completely within a single source cell.
  • There can be regions of high interpolation error where shifting grid alignment causes different numbers of source points to contribute to the destination point.
  These drawbacks are due to the fact that this conservative method does not have a constraint on local conservation.
• The ESMF_CubedSphereRegridEx produces some suspicious interpolation weights (large, outside [0,1]). These should be further investigated to see if they are within the bounds of expected results for non-monotonic remapping methods or represent true errors.
• ESMF_AttributeWrite() only works for ESMF standard Attribute packages.
• Also see SourceForge bugs list.

• Discover: ESMF_ArrayRedistOpenMPSTest.F90 fails with intelmpi when run by an automatic test script, but runs correctly when executed manually.
• SunOS test platform: the C interfaces do not work (seg fault).
• SunOS test platform: the ESMF_AttributeSTest.F90 and ESMF_AttributeUpdateEx.F90 fail.
• SunOS test platform: the ESMF_FieldRegridDisjointSTest.F90 system test fails.
• SunOS test platform: the Field test harness results are ignored.
• Windows/MinGW with intel and intelcl: the ESMF_ArrayRedistOpenMPSTest.F90, and ESMF_FieldArbGridUTest.F90 fail.
• Windows/Cygwin g95: the ESMF_MeshUTest.F90 fails.
• Windows/Cygwin g95: ESMF_LogErrUTest.F90 fails intermittently.

• List of ESMF interface changes since ESMF_3_1_1.
• Added public C interfaces for select methods of the Mesh class. The prefix for these methods is ESMC. We do not yet have a Reference Manual that describes these interfaces.
• Modifications were made to selected GridComp and CplComp methods that enable use of keyword syntax for procedure arguments. This was not permitted before because they did not have explicit interfaces. Using keyword syntax ensures that the compiler will check argument types. We strongly recommend that users take advantage of this feature. All of the ESMF calls that take dummy procedure arguments, or call into previously registered user procedures, now provide explicit interfaces: ESMF_GridCompSetVM(), ESMF_GridCompSetServices(), ESMF_GridCompSetEntryPoint(), ESMF_GridCompInitialize(), ESMF_GridCompRun(), ESMF_GridCompFinalize(), ESMF_GridCompReadRestart(), ESMF_GridCompWriteRestart(), ESMF_CplCompSetVM(), ESMF_CplCompSetServices(), ESMF_CplCompSetEntryPoint(), ESMF_CplCompInitialize(), ESMF_CplCompRun(), ESMF_CplCompFinalize(), ESMF_CplCompReadRestart(), ESMF_CplCompWriteRestart().
• Those methods that call into a user supplied routine provide two separate (optional) return codes - the standard "rc" return code for the ESMF library, and the "userRc" return code set by the user routine. If keyword syntax is not used, the two return codes can get confused, with dire results. Specifically, using the old interface argument order, without keyword syntax, would lead to incorrect rc association. Users are strongly encouraged to make appropriate changes to their codes. See the API changes page for details.
• ESMF_GridCompSetServices() and ESMF_CplCompSetServices() now support runtime loading of ESMF components via shared objects. This allows users to maintain separate build systems for individual components in the same application. The newly added ESMF_ArrayRedistSharedObj system test demonstrates this feature.
• Changed the behavior of ESMF_GridCompSetServices() and ESMF_CplCompSetServices() to support the MAPL/GEOS-5 Component hierarchy paradigm. For most user code the SetServices change is expected to be completely transparent. All previous versions of ESMF executed the specified SetServices routine in the context of the parent VM. Now, the specified public child routine is executed within the context of the child VM. This means that calls that would affect the child VM must be issued before the child SetServices routine is called. Two new interfaces, ESMF_GridCompSetVM() and ESMF_CplCompSetVM(), have been added that allow child component code to be called and executed from within the context of the parent VM. The benefit of the SetServices change is that the child SetServices routine may now contain calls that require execution from within the context of the child VM, e.g. creation of sub-components.
• Added standard ReadRestart and WriteRestart methods to the ESMF_GridComp and ESMF_CplComp interfaces.
• The ESMF_GridCompGet() and ESMF_CplCompGet() interfaces now offer access to the current method (for example, initialize, run, ...) and current phase of a Component.
• Added "attachable methods" to ESMF States. User supplied methods can be attached, executed and removed. Currently attached user methods are ignored during ESMF_StateReconcile(), and attached methods cannot be copied or moved between States. For details see the new "Attachable Methods" section in the "Infrastructure: Utilities" part of the ESMF Reference Manual.
• The InternalState documentation in the Reference Manual has been reworked to be more applicable to real world situations.
• Added support in ESMF_FieldCreate() methods for Fields on arbitrarily distributed Grids. ESMF_FieldRegrid() does not yet work for arbitrarily distributed Grids. Overall, communications using Fields on arbitrarily distributed Grids have not been well tested.
• The ESMF_FieldRegridStore() call now supports source and destination masking. The user can use this masking to tell the regrid to ignore source and/or destination points.
• The ESMF_FieldRegridStore() call now has an unmappedDstAction flag. This allows the user to choose if they want unmapped destination points to cause an error or to be ignored.
• A number of bug fixes and tweaks have improved the accuracy and stability of ESMF_FieldRegridStore(). This is in particular true of the higher-order interpolation weight generation.
• Added a new public method ESMF_FieldGetBounds() to allow a user to query localDe specific data bounds.
• Grids now have the capability to store other types of data besides coordinates. This data is referred to as Grid items. A set of new interfaces which mirror the ESMF_GridCoord interfaces has been added to Grid to support this functionality. The Grid currently supports four items and a set of new flags (ESMF_GRIDITEM_MASK, ESMF_GRIDITEM_AREA, ESMF_GRIDITEM_AREAM, ESMF_GRIDITEM_FRAC) has been added to be used in the interface to indicate the item being accessed. The ESMF_GRIDITEM_MASK item is used to store mask information for use by ESMF_FieldRegridStore().
• Added support to create arbitrarily distributed Grids using ESMF_GridCreateShapeTile(), ESMF_GridCreate(), and ESMF_GridSetCommitShapeTile(). Other Grid interfaces have been modified as needed. Grid items do not work for arbitrarily distributed Grids.
• Changed ESMF_GridGet() interface to get the information defined for the arbitrarily distributed Grid, including arbDim (the dimension in the DistGrid that represents the arbitrarily distributed dimension(s)), localCount (the number of PET-local arbitrarily distributed index locations), and localIndices (a list of PET-local arbitrarily distributed index locations). The names of these arguments are likely to change in future releases.
• Added ability to read Attributes from an XML file via new method ESMF_AttributeRead(), which then attaches the Attributes to a Component, Field, or Grid.
• XML Attribute file reading is done with the open source third party Xerces C++ library. ESMF must be built against the Xerces library, which requires the setting of additional ESMF environment variables. See the "Third Party Libraries" section in the ESMF User's Guide for details.
• Implemented ability to specify time units as 8-byte floating point values in Time Manager methods.
• Implemented ESMF standard Attribute packages for Components, State, Field, and Grid.
• Added Attributes to ArrayBundle and DistGrid.
• Allow Field Attribute hierarchy to connect to Grid Attribute hierarchy.
• Allow ESMF_AttributeWrite() to be called from Components, State, FieldBundle, and Field.
• Allow ESMF_AttributeCopy() to locally copy Attribute values between Components.
• Added ESMF_AttributeUpdate() routine to build consistent Attribute structure and values for objects defined across a VM. One use of this feature is to use it in conjunction with ESMF_AttributeCopy() during the CplComp run method in order to transfer Attributes between GridComps defined on mutually exclusive sets of PETs.
• Overloaded ESMF_AttributeRemove() to remove entire Attribute packages, Attributes in an Attribute package, or just single Attributes.
• Added the ability to nest Attribute packages inside of each other.
• ESMF_StateReconcile() now has a flag to allow optional Attribute reconciliation. The default is currently that Attribute reconciliation is off, due to concerns about performance and robustness.
• Added a set of overloaded ESMF_AttributeLinkRemove() interfaces. These detach an object's Attribute hierarchy from that of another object.
• Modified the ESMF_Attribute system test to use ESMF_AttributeUpdate(), ESMF_AttributeCopy(), and ESMF_StateReconcile() on both States and Components where appropriate.
• The range of Fortran unit numbers used within ESMF can be overridden via the ESMF_Initialize() call by using the new IOUnitLower and IOUnitUpper optional arguments. A new ESMF_IOUnitGet() method to find an unused unit number within the ESMF unit number range is provided as a service to application developers.
• ESMF_IOUnitFlush() is provided as a common method to flush Fortran unit numbers. This is intended for use as a service to application developers.
• Added ESMF_ConcurrentEnsemble system test demonstrating an ensemble in which different ensemble members, which are distinctly different components, run concurrently on distinct sets of PETs.
• Added ESMF_InternalStateEnsemble system test demonstrating how different ensemble members representing the same Component can be written using a single Component object which indexes different elements of an ESMF internal State.
• Activated TestHarness for Array and Field testing.
• Added support for Windows MinGW/MSYS, using the Intel Fortran and C++ compiler and Microsoft MPI. This was developed and tested under Windows Compute Cluster Server (CCS) 2003.
• Standardized 3rd party library support (NETCDF, PNETCDF, LAPACK, XERCES).
• An offline file-to-file regridding application has been developed as part of ESMF. This application takes two netcdf grid files and outputs interpolation weights generated using ESMF's higher-order interpolation. This application supports a number of options for handling the pole: no pole, treating the pole as the average of the values surrounding the pole, and an n-point stencil average of the points surrounding the pole. The format of the files makes this application a suitable replacement for SCRIP. For details see the new "File Based Regrid Weight Application" section in the "Infrastructure: Fields and Grids" part of the ESMF Reference Manual.
• Furthermore, all the features listed in the release notes for ESMF_3_1_0rp2 apply to this release.

• The ESMF_StateReconcile() call, generally used by couplers dealing with Components running on exclusive petLists, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile()
• The ESMF_StateReconcile() call fails for nested States.
• ESMF_LOG_SINGLE is known to be broken. The symptoms are platform dependent and may appear non-deterministic.
• Fields built on LocStreams and Meshes will not work with ESMF_StateReconcile() and therefore cannot be transfered between Components running concurrently.
• ESMF_FieldRegrid() is only implemented for Fields created on Grids with indexflag equal to ESMF_INDEX_GLOBAL. This is not the default.
• Proxy Field objects, internally created during ESMF_StateReconcile(), return the wrong dimCount.
• The ESMF_Grid creation interfaces that take the petMap argument (e.g. ESMF_GridCreateShapeTile()) mistakenly return an error for inputs that put more than one DE on a PET, if the DEs are not mapped onto the PETs in sequential order.
• Grid connections don't work in this release and will have no effect on the actions of ESMF_FieldRegridStore().
• Repeating sticky Alarms whose ringInterval is less than or equal to the Clock's timeStep do not ring properly.
• Repeating sticky Alarms that are not turned off during a waiting period of a Clock run do not ring properly after the waiting period is over, Alarms are cleared, or turning off of subsequent Alarms resumes.
• Repeating non-sticky Alarms do not ring properly for a Clock that goes into ESMF_MODE_REVERSE at the Clock's stopTime, while an Alarm is ringing.
• Negative whole numbers specified as 8-byte floating point values for *_r8 time units in ESMF_TimeSet() and ESMF_TimeIntervalSet( ) methods incorrectly lose their sign, becoming positive.
• Array and ArrayBundle Attribute hierarchies are not implemented yet.
• ESMF_AttributeWrite() only works for ESMF standard Attribute packages.
• In the Reference Manual, "Figure 29: The structure of the Attribute class" shows an attrCount variable which is no longer present, and does not show two vectors of Attribute pointers called packList and linkList.
• In the API description of ESMF_AttributeLink() and ESMF_AttributeLinkRemove(), in the ESMF Reference Manual, the argument keyword should be "comp1" instead of "comp".
• Also see SourceForge bugs list.

• The Mesh and ESMF_FieldRegrid() code is unstable when compiled with PGI.
• ESMF_DistGridCreateGetUTest.F90 fails on IBM Lightning with PGI.
• ESMF_ClockUTest.F90 fails on IBM/AIX in ESMF_BOPT=O mode.
• ESMF_ClockUTest.F90 fails on Linux with NAG.
• All the ArrayHarness unit tests fail on IBM Bluefire with ESMF_ABI=32 and ESMF_BOPT=g.
• ESMC_ConfigUTest.C fails on IBM Bluefire with ESMF_ABI=64 and ESMF_BOPT=O.
• ESMF_GridCoordUTest.F90 fails on Darwin with Absoft and with ESMF_BOPT=O.
• The Attribute unit tests fail with Lahey and ESMF_BOPT=O.
• ESMF_ConfigOverviewEx.F90 and ESMF_ConfigUTest.F90 fail on ia32 with PGI.
• Examples ESMF_LogErrEx.F90, ESMF_AttReadGridCompEx.F90, ESMF_AttReadGridEx.F90, and ESMF_AttributePackageEx.F90 fail on Linux with the g95 v0.5 compiler.
• The C unit tests and examples fail to link on wJet with the Intel compiler.

• List of ESMF interface changes since ESMF_3_1_0rp1.
• Added public C interfaces for select methods of the following classes:
  • GridComp, CplComp
  • State
  • Clock, TimeInterval, Time, Calendar
  • Config
  • Array
  • ArraySpec
  • DistGrid
  • VM
  The prefix for these methods is ESMC. We do not yet have a Reference Manual that describes these interfaces.
• Restored most Field and FieldBundle communication methods except the halo update methods.
• Added a method to perform a bilinear or higher order regridding between two Fields constructed on Grids. The Fields must be based on Grids with indexflag equal to ESMF_INDEX_GLOBAL (not the default) and must have data at cell centers (the default). This method will not work for Fields built on LocStreams or Meshes. Interpolation weights are computed in parallel.
• Added a LocStream (Location Stream) class to represent and manipulate observational data streams. Fields can be created using a LocStream instead of a Grid.
• Added ability to represent an unstructured Mesh in ESMF. Fields can be created using a Mesh instead of a Grid.
• The DistGrid interface changes in this release reflect modifications that are necessary to support factorized sequence indices in the future. This feature is required for the ongoing arbitrary Grid development and is not fully supported yet.
• Added Attributes to the CplComp and GridComp classes.
• Added the ability to build Attribute hierarchies.
• Added Attribute packages to CplComp, GridComp, and Field. Either standard or custom Attribute packages can be created.
• Added a method to locally copy Attributes between States called ESMF_AttributeCopy().
• Overloaded ESMF_AttributeGet() and ESMF_AttributeSet() to allow the manipulation of Attributes in Attribute packages.
• Added an optional default value to ESMF_AttributeGet().
• Added an interface to write out an Attribute hierarchy in either tab delimited or XML format called ESMF_AttributeWrite().
• Added character lists as an option for Attribute type.
• Standardized the use of count and itemCount in Attribute calls and made them optional in most cases.
• Added support for the Modified Julian Day calendar, ESMF_CAL_MODJULIANDAY.
• Alarms now ring properly for a clock in ESMF_MODE_REVERSE.
• Argument lists for Fortran-callable entry points which contain logical data have been regularized to always use Fortran LOGICAL. Previously, some routines used the internal ESMF_Logical derived type.
• Parallel build of the ESMF library supported via GNU make -j option.
• Added ESMF_CompFortranAndC system test that tests and demonstrates passing of State objects between Fortran and C components.
• Added ESMF_Attribute system test demonstrating Attribute packages and hierarchies.
• Added ESMF_FieldBundleSMM system test demonstrating the sparse matrix multiply using FieldBundles.
• Added ESMF_ArrayRedist3D system test demonstrating a three dimensional redistribution operation for Arrays.
• Added ESMF_SeqEnsemEx system test demonstrating an ensemble in which different ensemble members, which are distinctly different components, run on the same set of PETs.

• The same as ESMF_3_1_0rp1 with the following exception:
  • The problem with the ESMC Component interfaces on XLF and PGI compilers has been fixed.
• For higher-order interpolation, ESMF_FieldRegrid() can give wrong answers for multiple processors. We recommend only using this on a single processor.
• In this release the accuracy of the higher-order interpolation has been degraded by a bug. For better results we recommend using ESMF v4.0.0. or later.
• ESMF_FieldRegrid() has problems with the pole. When using ESMF_REGRID_SCHEME_FULL3D, we recommend not using any destination points which either lie in a cell next to the pole (for both bilinear and higher-order) or two cells away from the pole (for just higher-order).
• Fields built on LocStreams and Meshes cannot be transferred between Components running concurrently.
• ESMF_FieldRegrid() is only implemented for Fields with data located at cell centers.
• ESMF_FieldRegrid() is only implemented for Fields created on Grids with indexflag equal to ESMF_INDEX_GLOBAL. This is not the default.
• The Grid connections don't work in this release and will have no effect on the actions of ESMF_FieldRegrid().
• New Attribute methods are unreliable across a range of platforms. They should be considered an early prototype and used with caution.
• On IBM Bluegene the ESMF_FieldRegridEx.F90 and ESMF_FieldSphereRegridEx.F90 fail with ESMF_BOPT=O.
• A problem in the Config C interface leads to failures in the ESMC_ConfigUTest.C on the XLF compiler.
• ESMF_AttributeWrite() does not write out the values in the header of the .xml file generated when calling with ESMF_AttWriteFlag=ESMF_ATTWRITE_XML.
• The Use Test Cases were not updated or tested for this release.
• The ESMF_ArrayHarness2UTest is incorrectly reported as failing when executed through any of ESMF's batch mode mpirun scripts.
• Also see SourceForge bugs list.

• List of ESMF interface changes since ESMF_3_0_3.
• Field, Bundle, State, and Component classes are now built on new Grid and Array classes. Field and Bundle method arguments have significantly changed so that they more closely resemble those of Grid and Array.
• The following classes were removed:
  • IGrid (in the InternGrid directory, which was also removed)
  • FieldDataMap
  • BundleDataMap
• The InternArray class is no longer visible at public interfaces. It will be removed in a future release along with InternDG.
• Added system test FieldSparseMatMul. This creates Fields and gets Arrays from them in order to perform an ESMF_ArraySparseMatMul() operation.
• The following were disabled:
  • Field and Bundle communication methods and corresponding unit tests and examples
  • Many Field and Bundle get and set methods and corresponding unit tests and examples
  • System tests *except* for CompCreate, DistDir, ArraySparseMatMul, FieldSparseMatMul, ArrayRedist, and ArrayScatterGather. Other system tests have not yet been modified to work with updated Field and Bundle classes, and will be restored in a future release.
• Further work on the new Grid class:
  • Grids can be created or set with a regular distribution.
  • Coordinate arrays can be less than the Grid's rank (e.g. rectilinear coordinates)
  • Grids can be serialized and deserialized (e.g. as part of a state reconcile operation)
  • Grids can be queried for information about a stagger location including local DE exclusive and computational bounds, computationalEdgeWidths, and undistributed bounds.
  • Grids can be queried for element counts in addition to lower and upper bound information.
• The Grids have been altered to live entirely within the Array exclusive region to allow for smoother interaction with Array methods.
• Stagger locations have also been changed to allow:
  • Comparisons between staggerlocs
  • The output of a string describing the staggerloc
• Implemented ESMF_ArrayGather()
• Implemented ESMF_ArrayRedist()
• Implemented ESMF_ArraySet()
• Support the use of precomputed ArraySparseMatMul RouteHandles with different actual Arrays.
• Optimized performance of ESMF_ArraySparseMatMulStore().
• Optimized performance of ESMF_ArraySparseMatMul().
• Implemented Array replication along specified DistGrid dimensions.
• Generalized ArraySparseMatMul to support general Array - DistGrid dimension mapping.
• Generalized ArraySparseMatMul to support tensor term mixing.
• Implemented a more flexible Array computational bounds scheme that allows a computational region that is smaller than the exclusive region.
• Changed Array total widths to be relative to the Array computational region.
• Added computationalEdgeLWidth and computationalEdgeUWidth arguments to ESMF_ArrayCreate() interfaces to allow special treatment of the edge.
• Changed DistGrid basic elements from cells to elements.
• Changed Array's dimmap and inverseDimmap arguments to distgridToArrayMap and arrayToDistGridMap, respectively.
• Added LocalArray section to ESMF Reference Manual.
• Added support for Intel compiler version 10.
• Increased robustness of g95 and gfortran configurations on Linux and Darwin.
• Added support for Cray XT3/4 systems with CNL kernel.

• The ESMF_Grid creation interfaces that take the petMap argument (e.g. ESMF_GridCreateShapeTile()) mistakenly return an error for inputs that put more than one DE on a PET, if the DEs are not mapped onto the PETs in sequential order.
• The only Use Test Case that currently builds is ESMF_SparseMatMul. Due to failures of the other UTCs it must be built and run separately, not from the collective UTC makefile.
• The zeroflag=ESMF_TRUE option in the ESMF_ArraySparseMatMul() call is broken for Arrays that have total regions that are larger than the exclusive region. The symptom of this bug is that not all destination elements are reset to zero before adding the sparse matrix multiplication terms.
• The ESMF_StateReconcile() call, generally used by couplers dealing with Components running on exclusive petLists, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile() .
• C/C++ Component examples fail with the XLF and PGI compilers as well as for the Linux/absoftintel combination.
• The documentation for ESMF_GetAttribute() and ESMF_SetAttribute() calls for multi-valued list Attributes does not show a required integer input "count" argument after the "name" argument. If the "count" argument, which represents the number of Attribute values, is inserted the call will work.
• ESMF_LOG_SINGLE is known to be broken. The symptoms are platform dependent and may appear non-deterministic.
• On IRIX64 the exhaustive unit tests compiled against MPI will intermittently hang when run in uni-PET mode if the ESMF library was built with ESMF_PTHREADS=ON. This issue does not show up when running the MPI version in multi-PET mode or when using the MPIUNI version in uni-PET runs. The issue disappears when the ESMF library is compiled with ESMF_PTHREADS=OFF (which is the default for IRIX64).
• ESMF fails to build on Cray XT3/XT4 with PGI 6.2.5 with an internal compiler error.
• On IBM AIX systems in 32-bit mode StateLimitUTest fails in uni-PET mode.
• On x86_64 systems using the medium-memory model with PGI 6.2-3 or PathScale 2.4 ESMF_ArrayDataUTest.F90 will report two failures. These failures indicate a problem with the F90 dope vector and ESMF must be assumed unreliable under these conditions.
• On x86_64 systems using the medium-memory model with PGI 6.2-3 the ESMF_ArrayLarrayEx.F90 fails.
• On Cray X1 the ESMF_DistDir system test fails.
• On IBM BlueGene/L ESMF_VMSendVMRecvUTest.F90 fails in uni-PET mode.
• On Linux Lahey, a test in ESMF_ArrayDataUTest.F90 fails with ESMF_BOPT=g.
• All NAG compiler combinations we tested (on Linux as well as on Darwin) show random run-time failures when ESMF is compiled in optimized mode and also when user code that links with ESMF is compiled in optimized mode. Consequently ESMF sets ESMF_OPTLEVEL to 0 (zero) by default when compiling ESMF with NAG and we advise NAG users to not use optimization on user code that links against the ESMF library.
• On Linux intel 9.1 with mpich2, unit test ESMF_StringUTest.F90 fails with ESMF_BOPT=O.
• On x86_64 systems with PGI 6.2-3 or PathScale 2.4, unit test ESMF_VMBarrierUTest.F90 fails intermittently.
• Also see SourceForge bugs list.

• List of ESMF interface changes since ESMF_3_0_2.
• Renamed all references to Grids that existed in previous versions of the framework to IGrid and moved the class to an InternGrid directory. The IGrid class is deprecated and will be removed in a future release.
• Created a new Grid class, which is still at an early stage of development. The new Grids can:
  • be 1D, 2D, 3D and higher dimensions
  • have multiple undistributed dimensions
  • hold curvilinear coordinates at multiple stagger locations.
  The user can now:
  • create a new Grid either with shortcut methods or incrementally
  • get parameters from the new Grid
  • set and get Grid coordinates from an ESMF Array
  • get a pointer to the memory holding the coordinates in the Grid
  • get array bound information for the coordinate arrays
  • destroy a new Grid.
• The ESMF_ArraySparseMatMulStore() and ESMF_ArraySparseMatMul() methods were rewritten and improved performance and reduced memory usage are expected. The methods have the following new features:
• support for distributed definition of factors
• support for user-supplied arbitrary sequence indices via 1D DistGrid
• support for ESMF_TYPEKIND_I4/I8/R4/R8
• default initialization of destination Array to zero (can be overridden)
• A collective ESMF_DistGridCreate() call was introduced that takes local lists of arbitrary sequence indices and returns a matching 1D DistGrid. This can be used in combination with the Array sparse matrix multiply to regrid unstructured grids.
• The ESMF_DistGridGet() interface was massively updated to provide complete access to DistGrid information. The names of many of the arguments have changed.
• The ESMF_ArrayCreate() calls now provide the "name" argument.
• Many unimplemented methods will return the failure code ESMF_RC_NOT_IMPL. Validate methods are considered implemented if at least initialization checks are present.
• Cygwin.g95.default and Cygwin.gfortran.default configurations were added.
• Starting with this version of ESMF, support for PGI compilers has been limited to PGI versions >= 6.2. We will stop reporting ESMF known bugs for older PGI versions.

• C/C++ Component examples fail with the XLF and PGI compilers as well as for the Linux/absoftintel combination.
• The ESMF_StateReconcile() call, generally used by couplers dealing with Components running on exclusive petLists, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile() .
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_ArrayGather() are only expected to work on 2D grids with 2D data.
• When ESMF_InternArray objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
• Overall, we have found that regridding using ESMF_InternArrayRegrid(), FieldRegrid(), and BundleRegrid() is not robust. We encourage customers interested in ESMF regridding to move to the ESMF_ArraySparseMatMul() approach while regrid methods are reworked.
• The conservative regrid algorithm for full spheres, which includes special treatment of the poles, is largely untested and expected to produce errors.
• The conservative regrid method for both regional and global grids produces incorrect results for PGI in MPIUNI mode.
• The ESMF_Grid creation interfaces that take the petMap argument (e.g. ESMF_GridCreateShape()) mistakenly return an error for inputs that put more than one DE on a PET, if the DEs are not mapped onto the PETs in sequential order.
• The documentation for ESMF_GetAttribute() and ESMF_SetAttribute() calls for multi-valued list Attributes does not show a required integer input "count" argument after the "name" argument. If the "count" argument, which represents the number of Attribute values, is inserted the call will work.
• ESMF_LOG_SINGLE is known to be broken. The symptoms are platform dependent and may appear non-deterministic.
• On IRIX64 the exhaustive unit tests compiled against MPI will intermittently hang when run in uni-PET mode if the ESMF library was built with ESMF_PTHREADS=ON. This issue does not show up when running the MPI version in multi-PET mode or when using the MPIUNI version in uni-PET runs. The issue disappears when the ESMF library is compiled with ESMF_PTHREADS=OFF (which is the default for IRIX64).
• On Cray X1, the conservative regrid method does not work when the grid is defined as a uniform horizontal LatLon grid with periodic longitude coordinate and the source and destination grid are of the same size.
• ESMF fails to build on Cray XT3/XT4 with PGI 6.2.5 with an internal compiler error.
• On IBM AIX systems in 32-bit mode StateLimitUTest fails in uni-PET mode.
• On x86_64 systems using the medium-memory model with PGI 6.2-3 or PathScale 2.4 ESMF_ArrayDataUTest.F90 will report two failures. These failures indicate a problem with the F90 dope vector and ESMF must be assumed unreliable under these conditions.
• On x86_64 systems using the medium-memory model with PGI 6.2-3 the ESMF_ArrayLarrayEx.F90 fails.
• On Cray X1 the ESMF_DistDir system test fails.
• On IBM BlueGene/L ESMF_VMSendVMRecvUTest.F90 fails in uni-PET mode.
• On Linux Lahey, a test in ESMF_ArrayDataUTest.F90 fails with ESMF_BOPT=g.
• All NAG compiler combinations we tested (on Linux as well as on Darwin) show random run-time failures when ESMF is compiled in optimized mode and also when user code that links with ESMF is compiled in optimized mode. Consequently ESMF sets ESMF_OPTLEVEL to 0 (zero) by default when compiling ESMF with NAG and we advise NAG users to not use optimization on user code that links against the ESMF library.
• The documentation of the ESMF_ArraySparseMatMulStore() API incorrectly states that the returned "routehandle" argument can be used to perform ESMF_ArraySparseMatMul() on any Array pair that is DistGrid-conforming to the Array pair used during the ESMF_ArraySparseMatMulStore(). This feature is _not_ yet supported, i.e. currently the "routehandle" can only be used to perform ESMF_ArraySparseMatMul() between the Array pair for which it was precomputed.
• Also see SourceForge bugs list.

• List of ESMF interface changes since ESMF_3_0_1.
• The ESMF_ArrayCreate(), ESMF_ArrayGet() and ESMF_ArrayScatter() interfaces were type/kind/rank overloaded.
• ESMF_VMScatterV() and ESMF_VMGatherV() calls were added.
• Methods across the framework had their return codes initialized to not implemented (ESMF_RC_NOT_IMPL). This is a step towards ensuring that unimplemented code returns appropriate errors when called.
• LogErr calls were modified so that they need not set return codes prematurely to ESMF_SUCCESS. This is also a step towards ensuring that unimplemented code returns appropriate errors when called. This new LogErr behavior is not yet turned on, as there are still calls that rely on the former behavior of LogErr.
• The declaration and usage of real and integer datatypes across the framework was standardized. The most significant change was the combination of type and kind into a single ESMF_TypeKind parameter. The new datatype conventions are included in the Developer's Guide.
• A policy for the auto-promotion of user data via compile options was developed and is included in the Developer's Guide.
• An ESMF_ConfigSetAttribute() call was implemented for 4-byte integers.
• Conventions were established for a public ESMC optional argument API based on the variable-length argument list functionality provided by <stdarg.h>.
• Added C interfaces for Config class (ESMC_Config). The ESMC_ConfigSetAttribute method is present, but not implemented.
• An advanced stub for a new testing control system for running exhaustive tests on regrid and redist, called the test_harness, has been added. It runs and parses trivial bundle and field redistribution test configurations.
• Fixed a bug that required Log files to be explicitly closed by the user before calling ESMF_Finalize.
• Users may now open, write and close multiple Log files.

• C/C++ Component examples fail with the XLF and PGI compilers as well as for the Linux/absoftintel combination.
• The ESMF_StateReconcile() call, generally used by couplers dealing with Components running on exclusive petLists, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile() .
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_ArrayGather() are only expected to work on 2D grids with 2D data.
• When ESMF_InternArray objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
• Overall, we have found that regridding using ESMF_InternArrayRegrid(), FieldRegrid(), and BundleRegrid() is not robust. We encourage customers interested in ESMF regridding to move to the ESMF_ArraySparseMatMul() approach while regrid methods are reworked.
• The conservative regrid algorithm for full spheres, which includes special treatment of the poles, is largely untested and expected to produce errors.
• The conservative regrid method for both regional and global grids produces incorrect results for PGI in MPIUNI mode.
• The documentation for ESMF_GetAttribute() and ESMF_SetAttribute() calls for multi-valued list Attributes does not show a required integer input "count" argument after the "name" argument. If the "count" argument, which represents the number of Attribute values, is inserted the call will work.
• ESMF_LOG_SINGLE is known to be broken. The symptoms are platform dependent and may appear non-deterministic.
• On IRIX64 the exhaustive unit tests compiled against MPI will intermittently hang when run in uni-PET mode if the ESMF library was built with ESMF_PTHREADS=ON. This issue does not show up when running the MPI version in multi-PET mode or when using the MPIUNI version in uni-PET runs. The issue disappears when the ESMF library is compiled with ESMF_PTHREADS=OFF (which is the default for IRIX64).
• On Cray X1, the conservative regrid method does not work when the grid is defined as a uniform horizontal LatLon grid with periodic longitude coordinate and the source and destination grid are of the same size.
• On Cray XT3/XT4, the ESMF_VMBarrierUTest is incorrectly reported as failing. The origin of this problem seems to be a bug in Cray's implementation of MPI_Wtime(). This problem will also affect the validity of ESMF_VMWtime().
• ESMF fails to build on Cray XT3/XT4 with PGI 6.2.5 with an internal compiler error.
• On IBM AIX systems in 32-bit mode StateLimitUTest fails in uni-PET mode.
• On x86_64 systems using the medium-memory model with PGI 6.2-3 or PathScale 2.4 ESMF_ArrayDataUTest.F90 will report two failures. These failures indicate a problem with the F90 dope vector and ESMF must be assumed unreliable under these conditions.
• On IBM BlueGene/L, tests in ESMF_ClockUTest.F90 fail for ESMF_BOPT=O/g. Fails ESMF_VMSendVMRecvUTest.F90 in uni-PET mode.
• On Linux Lahey, a test in ESMF_ArrayDataUTest.F90 fails with ESMF_BOPT=g.
• All NAG compiler combinations we tested (on Linux as well as on Darwin) show random run-time failures when ESMF is compiled in optimized mode and also when user code that links with ESMF is compiled in optimized mode. Consequently ESMF sets ESMF_OPTLEVEL to 0 (zero) by default when compiling ESMF with NAG and we advise NAG users to not use optimization on user code that links against the ESMF library.
• On the PGI 5.0-2 compiler with mpiuni, five regriding unit tests fail.
• On the PGI 6.1-1 compiler with mpiuni, multiple unit tests, examples and 2 system tests fail.
• On the PGI 5.2-4 compiler with mpich, multiple unit tests fail with ESMF_BOPT=g.
• Also see SourceForge bugs list.

• List of ESMF interface changes since ESMF_3_0_0.
• Standardized the checking of initialization in ESMF. This introduces two new error return codes. The first, ESMF_RC_OBJ_NOT_CREATED, is returned if the user attempts to use a deep class variable without first creating it. The second ESMF_RC_OBJ_DELETED is returned if the user attempts to use a deep class variable after its deletion. Also, before this change, shallow class variables were not initialized on platforms with compilers that didn't support type component initialization. Now this initialization is handled by ESMF. Implementing this change necessitated the change of the intents of shallow class input parameters throughout ESMF from strictly in to inout.
• Added an ESMF_LOG_NONE option to the "defaultLogType" argument of ESMF_Initialize() to turn off ESMF's default error logging.
• Added "mpiCommunicator" argument to ESMF_Initialize() to allow ESMF to be initialized on a subset of MPI processes. The added VM example ESMF_VMUserMpiCommEx.F90 tests this new feature.
• Added an ESMF_KEEPMPI option to the "terminationflag" argument of ESMF_Finalize() to allow ESMF finalization without MPI shutdown.
• Removed deprecated GridCompCreate() and CplCompCreate() interfaces.
• Optimized ESMF_BundleRedistStore(), ESMF_BundleRedist(), ESMF_FieldRedistStore(), and ESMF_FieldRedist() for arbitrarily distributed grids. A new algorithm was implemented i n the ESMF Route code to initialize the arbitrary to arbitrary grid redistribution. The redist run routine has been optimized specifically for the Cray X1 machine.
• Added ESMF_VMAllGatherV() and VMAllToAllV() methods.
• Added "errorMask" argument to ESMF_LogSet() to allow masking of error codes in LogErr handling.
• Reworked the build system to be more robust and portable. Some of the previously used environment variables have become obsolete or were replaced by new variables. A full list of the environment variables as well as build system details can be found in the User's Guide. The main README contains enough information to get started quickly.
• Introduced enhanced support for compiling and linking user applications against ESMF. See the User's Guide or main README for details.
• Added a section to the User's Guide on how to port ESMF to other platforms.
• Ported ESMF to the Cray XT3/XT4 and IBM BlueGene/L.
• Added the following system tests to test the Bundle and Field redistribution methods for both arbitrarily distributed grids and block distributed grids:
• ESMF_BundleRedistArb2ArbSTest.F90
• ESMF_BundleRedistBlk2ArbSTest.F90
• ESMF_BundleRedistBlk2BlkSTest.F90
• ESMF_FieldRedistArb2ArbSTest.F90
• ESMF_FieldRedistBlk2ArbSTest.F90
• ESMF_FieldRedistBlk2BlkSTest.F90
• Augmented the exhaustive regrid unit test ESMF_RegridToolUTest.F90 to include regrid scenarios with different resolution source grid and destination grid.
• Fixed ESMF_InternArrayUTest unit test failures on the Linux PGI compiler.
• The problem with PGI 6.1.1 on Linux was resolved as a consequence of the reworked build system.
• The following issues appear to be resolved due to compiler upgrades or indirectly as a result of other development:
• Problem building with XL Fortran compiler version 8.1.1.6, when compiling with the combination ESMF_PREC=64 and ESMF_BOPT=g.
• Issues with configuration of MPICH with Intel 8.1.
• The rank check error that incorrectly occurred during redistribution of Fields or Bundles that are transferred from block to arbitrary distribution (and vice versa) has been corrected.
• The ESMF distribution available for download on the ESMF website has been pruned of many extraneous files, and is no longer bundled with documentation files. The distribution has been reduced in size from about 127MB to 12MB. Users can get documentation files by downloading from SourceForge.
• This ESMF release is under the University of Illinois-NCSA Open Source License. Licenses are still a subject of discussion.

• C/C++ Component examples fail with the XLF and PGI compilers as well as for the Linux/absoftintel combination.
• The ESMF_StateReconcile() call, generally used by couplers dealing with Components running on exclusive petLists, has an internal size limit (which will be addressed in a su bsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile().
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_ArrayGather() are only expected to work on 2D grids with 2D data.
• When ESMF_InternArray objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
• Overall, we have found that regridding using ESMF_InternArrayRegrid(), FieldRegrid(), and BundleRegrid() is not robust. We encourage customers interested in ESMF regridding to move to the ESMF_ArraySparseMatMul() approach while regrid methods are reworked.
• On IRIX64 the exhaustive unit tests compiled against MPI will intermittently hang when run in uni-PET mode if the ESMF library was built with ESMF_PTHREADS=ON. This issue does not show up when running the MPI version in multi-PET mode or when using the MPIUNI version in uni-PET runs. The issue disappears when the ESMF library is compiled with ESMF_PTHREADS=OFF (which is the default for IRIX64).
• The conservative regrid algorithm for full spheres, which includes special treatment of the poles, is largely untested and expected to produce errors.
• The conservative regrid method for both regional and global grids produces incorrect results for PGI in MPIUNI mode.
• The documentation for ESMF_GetAttribute() and ESMF_SetAttribute() calls for multi-valued list Attributes does not show a required integer input "count" argument after the "name" argument. If the "count" argument, which represents the number of Attribute values, is inserted the call will work.
• ESMF_LOG_SINGLE is known to be broken. The symptoms are platform dependent and may appear non-deterministic.
• On Cray X1, the conservative regrid method does not work when the grid is defined as a uniform horizontal LatLon grid with periodic longitude coordinate and the source and destination grid are of the same size.
• On Cray X1, there are unit test failures across a wide range of classes, including Time classes, Field, State and Array, and we do not consider the release to be supported on this platform.
• On Cray XT3/XT4, the ESMF_VMBarrierUTest is incorrectly reported as failing. The origin of this problem seems to be a bug in Cray's implementation of MPI_Wtime(). This problem will also affect the validity of ESMF_VMWtime().
• On Cray XT3/XT4, the system test ESMF_ArraySparseMatMulSTest() and example ESMC_AppMainEx.C fail for ESMF_BOPT=O/g. System test ESMF_FieldHaloSTest() fails with ESMF_BOPT=O.
• On IBM BlueGene/L, tests in ESMF_ArrayScatterUTest.F90 fail for ESMF_BOPT=g. Tests in ESMF_ClockUTest.F90 fail for ESMF_BOPT=O/g.
• All NAG compiler combinations we tested (on Linux as well as on Darwin) show random run-time failures when ESMF is compiled in optimized mode and also when user code that links with ESMF is compiled in optimized mode. We advise NAG users to set environment variable ESMF_OPTLEVEL to 0 (zero) when compiling ESMF and to not use optimization on user code that links against the ESMF library.
• Also see SourceForge bugs list.

• List of ESMF interface changes since ESMF_2_2_2.
• The following classes were modified to implement an index space layer according to JST design discussions:
• Array
• DELayout
• DistGrid
• Adjusted library sources, examples and tests to DELayout changes.
• Added support for arbitary to arbitrary redistribution of Fields and Bundles.
• Renamed previous Array class to InternArray. This class is expected to become an internal class in future releases.
• Adjusted library sources, examples and tests to InternArray. Removed as many InternArray dependencies as possible.
• Fixed VMBarrierUTest for PGI (Argonne Jazz cluster). This was a known bug in the previous release.
• Fixed the C++ Component interface for most platforms. PGI and XLF still do not work.
• Re-enable the C++ Component examples, which were disabled for the previous release.
• Modified VMKernel and makefiles to allow compilation on NEC SX.
• Added support for the LSF batch queueing system on the AIX platform. (For NCAR computer users, this supports 'bluevista'.)

• The ESMF_StateReconcile() call, generally used by couplers dealing with concurrently running Components, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile().
• Redistribution of Fields or Bundles that are transferred from block to arbitrary distribution (and vice versa) fails with a message about mismatched ranks.
• The C/C++ Component examples fail with the XLF and PGI compilers.
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_InternArrayGather() are only expected to work on 2D grids with 2D data.
• The conservative regrid algorithm for full spheres, which includes special treatment of the poles, is largely untested and expected to produce errors.
• When ESMF_InternArray objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
• On AIX systems the ESMF fails to build with the XL Fortran compiler version 8.1.1.6, when compiling with the combination ESMF_PREC=64 and ESMF_BOPT=g. (Compiler version 8.1.1.4 works fine with all compile options, and ESMF_PREC=32 or ESMF_BOPT=O also work with either compiler version.)
• When compiling with Intel 8.1 and using MPICH, make sure the Intel/MPICH libraries are configured with: #define MPI2CPP_HAVE_BOOL 1 which may not be the default output from configure.
• On the Linux PGI compiler unit test ESMF_InternArrayUTest.F90 fails with both ESMF_BOPT=O/g
• Build fails on Longs Linux PGI 6.1.1 compiler with ESMF_BOPT=g.
• On the Cray X1, several system tests, examples and unit tests fail.
• All NAG compiler combinations we tested (on Linux as well as on Darwin) show random run-time failures when ESMF is compiled in optimized mode and also when user code that links with ESMF is compiled in optimized mode. We advise NAG users to set environment variable ESMF_OPTLEVEL to 0 (zero) when compiling ESMF and to not use optimization on user code that links against the ESMF library.
• Also see SourceForge bugs list.

• The ESMF_FieldExclSTest system test has been modified to demonstrate a simplified run loop for the concurrent case. This simplification was made possible by the modified blocking behavior of Components that was introduced with ESMF release 2.2.1.
• Fixed a copy in/out problem on the F90/C VM interface. On Linux using the Lahey f95 compiler this problem had caused the non-blocking VM communication calls to send/receive from/into invalid data buffers, resulting in corrupt communication results.
• The top level README has been updated to include more information about linking applications and porting ESMF to other platforms.
• Added an optional environment variable, ESMF_ARRAY_LITE, to restrict the number of array-level interfaces automatically generated. This is required on the NEC build since single byte integer arrays are not supported on that platform, but is available as a general feature to reduce the number of overloaded interfaces and code size.
• Added additional environment variables, ESMF_CXX_LIBRARY_PATH and ESMF_F90_LIBRARY_PATH, to support explicit library search paths. If defined, these will be used at link time to locate the correct F90 and C++ libraries. Otherwise, the standard LD_LIBRARY_PATH will be used.
• Added "Appendix B: ESMF Error Return Codes" to the ESMF Reference Manual.

• List of ESMF interface changes since ESMF_2_2_0rp1
• Changed the following in the ESMF_GridCompCreate() and ESMF_CplCompCreate() interfaces:
• Added an interface with all optional arguments. In subsequent releases this may become the only supported interface.
• We recommend that users call ESMF_xxxCompCreate() without specifying the parent VM. All tests, examples and demo codes have been modified to match the new convention.
• If the parent VM is not specified the current VM is used as the parent VM (previously the global VM was used in this case).
• Added support for high level MPMD. This makes xxxCompCreate() collective across all parent PETs and xxxCompSetServices() collective only across participating parent PETs.
• Added a new blockingFlag for xxxCompInitialze(), xxxCompRun() and xxxCompFinalize() that eliminates some unnecessary synchronization points. This flag (ESMF_VASBLOCKING) is now the default.
• The ESMF_StateReconcile() method no longer dumps core when the buffer size is exceeded. It returns an error message instead. (Please see the first item in the Known Bugs column.)
• DELayout now issues warning if non 1-to-1 DELayouts or non logically rectangular DELayouts are created since most of the rest of the system does not handle these yet.
• The Field and Bundle level redistribution communication calls have been rewritten. They now include 3+D data support and optimized communication calls for Bundles which contain Fields where the data arrays are congruent (same data type, same rank, etc).
• Validate routines have been added for the Field and Bundle levels of the regrid, halo, and redistribution routines. If a hang or core dump appears to be associated with a call to the run routine for these operations, the route handle can be passed to a Validate routine which looks for inconsistencies and other problems.
• Added full Fortran interface access to the non-blocking VM communications calls:
• ESMF_VMSend()
• ESMF_VMRecv()
• ESMF_VMSendRecv()
• ESMF_VMScatter()
• ESMF_VMGather()
• ESMF_VMReduce()
• ESMF_VMAllGather()
• ESMF_VMBroadcast()
• Configuration attributes can be validated.
• Configuration attributes can be checked for uniqueness. The "unique" argument in ESMF_ConfigLoadFile() now works.
• Configuration files may contain logical attribute values.
• LogErr no longer opens/closes logfile upon each write, enhancing performance.
• The Log message length increased to 256 characters.
• All Log messages (originating in either C++ or Fortran) go through Fortran for consistent, ordered output.
• Time Manager defaults for any year, month, day calendar:
• year = 0
• month = 1
• day =1
• Improved regrid unit test structure to facilitate increasing test coverage.
• Added system tests for MPMD (these are not automatically run by test scripts).
• Added a unit test VMUserMpiInitUTest which checks that the user can explicitly call MPI_Init() before ESMF_Initialize().
• Added unit tests for non-blocking VM communications.
• Added support for the LSF batch queueing system on the AIX platform. (For NCAR computer users, this supports 'bluevista'.)
• The files 'esmfuser.mk' and 'makefile.sample' in $ESMF_DIR/build replace the outdated 'link_rules.mk' files, showing how to link a user application with the ESMF library for any platform.
• Added MPICH2 and IntelMPI support for Linux.intel.
• The default linker front end on IRIX64 has been changed to CC, i.e. MIPSPro C++, for all ESMF applications. CC is now being used during link time regardless of the language in which the main program was written (Fortran90 or C++). In previous ESMF releases the f90 linker front end had been used to link applications on IRIX64 that provided the main program in Fortran. However, this led to problems during link time for applications that also contained C++ user code.
• More documentation was added to the implementation notes part of the communication routines section.
• Documented non-blocking features and limitations.
• Added extensive new documentation on Grids and regridding.
• Added reference documentation explaining that ESMF_CAL_NOCALENDAR does not track days, since the unit of a day is planet-specific.

• The ESMF_StateReconcile() call, generally used by couplers dealing with concurrently running Components, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 20, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile().
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_ArrayGather() are only expected to work on 2D grids with 2D data.
• The conservative regrid algorithm for full spheres, which includes special treatment of the poles, is largely untested and expected to produce errors.
• When ESMF_Array objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
• The C/C++ Component interface is broken, so the C++ examples have been removed from the makefiles for this release.
• On AIX systems there is a problem in the XL Fortran compiler version 8.1.1.6, when compiling with the combination ESMF_PREC=64 and ESMF_BOPT=g. (Compiler version 8.1.1.4 works fine with all compile options, and ESMF_PREC=32 or ESMF_BOPT=O also work with either compiler version.)
• On AIX the "ESMF_FlowWithCoupling" system test and the "ESMF_COUPLED_FLOW" demo fail intermittently.
• On the Linux PGI 5.0-2 compiler the "ESMF_FlowWithCoupling" system test and the "ESMF_COUPLED_FLOW" demo fail to compile with -g, resulting in an internal compiler error. (PGI 5.2-4 works fine)
  
• When compiling with Intel 8.1 and using MPICH, make sure the Intel/MPICH libraries are configured with: #define MPI2CPP_HAVE_BOOL 1 which may not be the default output from configure.
• Main programs in C/C++ do not work with the PGI compiler.
• On the Linux PGI compiler the followingunit tests fail:
• ESMF_ArrayUTest.F90
• ESMF_VMBarrierUTest.F90
• On the Linux Lahey compiler the ESMF_VMSendNbVMRecvNbUTest.F90 unit test fails.
• Support for the LSF batch queueing system on the AIX platform needs script not included in the release.
• All NAG compiler combinations we tested (on Linux as well as on Darwin) show random run-time failures when ESMF is compiled in optimized mode and also whe n user code that links with ESMF is compiled in optimized mode. We advise NAG users to set environment variable ESMF_OPTLEVEL to 0 (zero) when compiling ESMF a nd to not use optimization on user code that links against the ESMF library.
• Also see SourceForge bugs list.

• List of ESMF interface changes since ESMF_2_1_0rp3
• More error checking added to the State routines.
• Added the Abort option on ESMF_Finalize().
• Fixed two regrid bugs for globally-indexed Fields:
• Appeared in the bilinear regridding around domain edges on any processor except PE 0.
• Appeared for 3D fields where the non-grid related index did not have a lower bound of one, for any regridding algorithm.
• Additional regrid unit tests added.
• ESMF_BundleRedistStore(), ESMF_FieldRedistStore() and ESMF_ArrayRedistStore() routines take a VMinstead of a DELayout.
• Optional argument to select internal route data-packing and asychronous settings added to the ESMF_BundleRedistStore(), ESMF_FieldRedistStore() and ESMF_ArrayRedistStore() routines.
• Halowidth removed from the ESMF_FieldCreate() calls where an already-created array is specified, which already contains halo information.
• ESMF_VMAllGather() routine added.
• Added option to create child Components which share the parent's VM.
• Julian calendar added to Time Manager.
• Added ESMF_CalendarIsLeapYear() and ESMF_TimeIsLeapYear() methods.
• Additional internal packing options added to support different memory/ communication overhead ratios on different hardware platforms.
• Extensive makefile and build system updates, including 64-bit PGI support, user-contributed g95 support, support for mpich on OSF (HP), updated make targets, and updated README's and User's Guide content.
• It is now possible to compile ESMF withoutPthread dependency. If the environment variable ESMF_PTHREADS is set to OFF all Pthread dependent features of ESMF will be disabled. This can be used to simplify debugging and on systems that don't provide a functioning Pthread library. (This is different from the pthread-less default VM that was introduced in 2.1.0rp2!)
• Compile-time options in system tests to run with threading enabled.
• Output of unit and systems tests now go through the ESMF_Log facility, allowing multiprocessor test output to be cleanly collated.
• Updated output scripts for results of unit and system tests.
• More complete compiler and flag information printed for the output of 'gmake info'.
• Site files need now only contain differences, not entire copies of the default makefiles.
• Makefile target changes:
• The previous default target was named 'all' but it only built libesmf. The current default target is named 'lib' and does the same thing. The new target 'all' builds the library and all the examples, tests, and demos.
• The previous targets which were just 'tests' (e.g. build_tests) are now more specific: build_unit_tests, etc. There are new targets which are 'all_tests' to build everything.
• Complete makefile target documentation may be found in the ESMF top level README file and in the User's Guide (subsection 6.2.3).
• Added reference documentation about negative timesteps (as currently implemented, which is different from the upcoming reverse mode).
• Expanded and updated the reference document bibliography, with citations.

• The ESMF_StateReconcile() call, generally used by couplers dealing with concurrently running Components, has an internal size limit (which will be addressed in a subsequent release). There is no hard-coded limit to how many elements, i.e. States, Bundles, Fields, Arrays can be added to the State. However, there is a limit, roughly 2, on the maximum number of sub-elements contained by any element in the State passed into ESMF_StateReconcile(). ESMF_StateReconcile() causes a core dump if the buffer size is exceeded.
• Calling ESMF_StateReconcile() for objects with attributes causes memory corruption.
• Calling the redistribution and halo functions from a parent, coupler, or higher level Component fails on ESMF_Fields created in a child Component if that child is running on only a subset of the caller Component's PETs. This is the case, for example, when child Components are running concurrently. ESMF_FieldRegrid() does work correctly in this situation. Redistribution and halo functions work correctly when called from a child Component on ESMF_Fields created in that child Component.
• The Log function "Halt on Error" has not been implemented, so we expect an error from the unit test ESMF_LogErrHaltUTest.F90 on all platforms. Log message is: FAILED, ESMF_LogErrHaltUTest.F90, line 96. The Halt on Error test failed.
• The conservative regrid algorithm for full spheres, which includes special treatment of the poles, is largely untested and expected to produce errors. There is also some logic that is not correctly handling the situation where the physical domains of two Grids are slightly offset.  It does not result in errors or incorrect results but can be very inefficient.
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_ArrayGather() are only expected to work on 2D grids with 2D data.
• When ESMF_Array objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
• The following Time Manager calendar conversions do not work: Gregorian <--> NoLeap <--> 360  Day.
• On several platforms examples with a main in C/C++ do not link.These include Linux/Lahey, Linux/Absoft, Linux/PGI with mpiuni, and OSF1 (halem).
• The operating system and all compilers on the Cray X1 at Ames were updated this week. There are problems running with the updated system; to run on the X1 follow the instructions for reverting back to the original environment, or contact us for help in running with the latest system.
• The default compiler on the Compaq (halem), Fortran  551J, is compatible with this release. Fortran 551L core dumps when running system tests.
• On AIX systems there is a problem in the XL Fortran compiler version 8.1.1.6, when compiling with the combination ESMF_PREC=64 and ESMF_BOPT=g. (Compiler version 8.1.1.4 works fine with all compile options, and ESMF_PREC=32 or ESMF_BOPT=O also work with either compiler version.)
• When compiling with Intel 8.1 and using MPICH, make sure the Intel/MPICH libraries are configured with:
  #define MPI2CPP_HAVE_BOOL 1
  which may not be the default output from configure.
• On the Cray X1, the __LINE__ macro does not work in F90; "line 8" is always printed from methods such as ESMF_Test().
• The ESMF_VMAllFullReduceEx.F90 example fails on the Cray X1.
• On the SGI IRIX64 with ESMF_PREC=32, all system tests pass but hang up with ESMF_PREC=64.
• The ESMF_LogErrUTest.F90 unit tests fail on IRIX64 due to a difference in log file format as described in Sourceforge bug 1172026.
• The ESMF_VMBarrierUTest.F90 unit test fails on Linux/NAG, Linux/Intel, Linux/PGI and Linux/Lahey platforms.
• Also see SourceForge bugs list.

• ESMF_ArrayGather() now works even when the individual Arrays aren't identical sizes. This also fixes problems with ESMF_ArrayWrite()/ESMF_FieldWrite().
• Array object now sets halo width correctly (bug 995309).
• ESMF_FieldSetDataPointer() routine fixed to correctly create an Array.
• ESMF_FieldWrite() supports R8 data.
• Bundle level communication routines added (loops over Fields).
• Bundles now validate that the Grids match, and contain more error checking.
• The src/Infrastructure/IO/src files no longer require preprocessing.
• Regrid supports R4 data.
• Regrid code now correctly handles Fields with non-zero halo widths.
• LogError now buffers messages internally before flushing them to a file, and has updated documentation.
• Config no longer triggers errors from Fortran OPEN() when the file has read-only file permissions.
• Time manager bugs fixed:
  
• Bug 1014766 fixed; "dayOfYear always returns 1."  (also fixed similar bug in dayOfWeek).
• The maximum number of alarms per clock has been increased to 200, from 100 previously.
• Bug 996229 fixed; "Non-sticky alarms not functioning properly".
• Build and test fixes:
• ESMF_BUILD fixed for building files in a different directory from the source.
• Automated nightly unit tests now run multiprocessor on some platforms.
• Documents now build without having to compile the library first.
• More documentation added for Grid and Regrid functions.

• The default compiler on the Compaq (halem), Fortran  551J, is compatible with this release. Fortran 551L core dumps when running system tests.
  
• ESMF_BundleGather(), ESMF_FieldGather() and ESMF_ArrayGather() are only expected to work on 2D grids with 2D data.
• When ESMF_Array objects are created with a non-zero halo width, that width is added to all dimensions of the array, not only ones which correspond to the decomposed grid axes.
  
• This release returns an internal compile error with the PGI/Linux compiler and ESMF_BOPT=g. It builds successfully with ESMF_BOPT=O.
• There are known problems running with -O on Mac OS X compiling with absoft and running with lam mpi.  The workaround is to compile with ESMF_BOPT=g.
  
• On AIX systems there is a problem in the XL Fortran compiler version 8.1.1.6, when compiling with the exact combination ESMF_PREC=64 and ESMF_BOPT=g. (Compiler version 8.1.1.4 works fine with all compile options, and ESMF_PREC=32 or ESMF_BOPT=O also work with either compiler version.)
•  When compiling with Intel 8.1 and using MPICH, make sure the Intel/MPICH libraries are configured with:
  #define MPI2CPP_HAVE_BOOL 1
  which may not be the default output from configure.
• Main programs in C/C++ do not work with the PGI compiler.
• Routes will only work on PET counts that are powers of 2.
• Some regrid results may show "banding" when regridding between decompositions where processor domains overlap only slightly.
• Halo does not work correctly for periodic boundaries on non-decomposed axes.
  
• Also see SourceForge bugs list.